From 7a82b7b59763b66e33fbcf83bb91fce04abfda29 Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Thu, 25 Dec 2025 16:30:23 +0530 Subject: [PATCH] docs(supply-chain): update risk management notebook --- .../01_Supply_Chain_Data_Integration.ipynb | 707 +++++++++++++-- .../02_Supply_Chain_Risk_Management.ipynb | 829 +++++++++++++++--- 2 files changed, 1343 insertions(+), 193 deletions(-) diff --git a/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb b/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb index 266cd525..18ac92d5 100644 --- a/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb +++ b/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb @@ -6,33 +6,94 @@ "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb)\n", "\n", - "# Supply Chain Data Integration - Multi-Source Ingestion & KG Construction\n", + "# Supply Chain Data Integration - Multi-Source Ingestion & Relationship Mapping\n", "\n", "## Overview\n", "\n", - "This notebook demonstrates **supply chain data integration** using Semantica with focus on **multi-source ingestion**, **relationship mapping**, and **logistics tracking**. The pipeline ingests logistics and supplier data to build a comprehensive supply chain knowledge graph.\n", + "This notebook demonstrates **supply chain data integration** using Semantica with focus on **multi-source ingestion**, **relationship mapping**, and **logistics tracking**. The pipeline ingests logistics and supplier data from multiple sources to build a comprehensive supply chain knowledge graph with supplier network analysis.\n", "\n", "### Key Features\n", "\n", - "- **Multi-Source Ingestion**: Ingests data from multiple logistics and supplier sources\n", - "- **Relationship Mapping**: Maps supplier relationships and logistics routes\n", - "- **Logistics Tracking**: Tracks products, routes, and locations\n", - "- **KG Construction**: Emphasizes ingestion and KG construction for supply chain\n", - "- **Supplier Relationship Mapping**: Maps complex supplier networks\n", + "- **Multi-Source Ingestion**: Ingests data from multiple logistics and supplier sources (RSS, web APIs, files)\n", + "- **Relationship Mapping**: Maps supplier relationships and logistics routes using relation extraction\n", + "- **Logistics Tracking**: Tracks products, routes, locations, and warehouses\n", + "- **Supplier Network Analysis**: Analyzes supplier centrality and community clusters\n", + "- **Seed Data Integration**: Uses supplier foundation data for entity resolution\n", + "- **KG Construction**: Builds comprehensive supply chain knowledge graphs\n", "\n", - "### Pipeline Architecture\n", + "### Learning Objectives\n", "\n", - "1. **Phase 0**: Setup & Configuration\n", - "2. **Phase 1**: Multi-Source Data Ingestion (Logistics, Suppliers)\n", - "3. **Phase 2**: Entity Extraction (Supplier, Product, Route, Location, Logistics)\n", - "4. **Phase 3**: Supply Chain Knowledge Graph Construction\n", - "5. **Phase 4**: Relationship Mapping\n", - "6. **Phase 5**: Logistics Tracking\n", - "7. **Phase 6**: Visualization & Export\n", + "- Understand how to ingest data from multiple sources for supply chain integration\n", + "- Learn to map complex supplier relationships and logistics routes\n", + "- Master supplier network analysis using centrality and community detection\n", + "- Explore relationship extraction for supply chain entities\n", + "- Practice multi-source data integration and deduplication\n", + "- Analyze supplier networks and logistics connections\n", "\n", - "---\n", + "### Pipeline Flow\n", "\n", - "## Installation\n" + "```mermaid\n", + "graph TD\n", + " A[Multi-Source Ingestion] --> B[Seed Data Loading]\n", + " A --> C[Document Parsing]\n", + " B --> D[Text Processing]\n", + " C --> D\n", + " D --> E[Entity Extraction]\n", + " E --> F[Relationship Extraction]\n", + " F --> G[Deduplication]\n", + " G --> H[KG Construction]\n", + " H --> I[Embedding Generation]\n", + " I --> J[Vector Store]\n", + " H --> K[Network Analysis]\n", + " H --> L[Community Detection]\n", + " J --> M[GraphRAG Queries]\n", + " K --> N[Visualization]\n", + " L --> N\n", + " H --> O[Export]\n", + "```\n", + "\n", + "### Data Sources\n", + "\n", + "#### Logistics RSS Feeds\n", + "- **Supply Chain Dive**: https://www.supplychaindive.com/rss\n", + "- **Logistics Management**: https://www.logisticsmgmt.com/rss\n", + "- **Supply Chain 24/7**: https://www.supplychain247.com/rss\n", + "- **Material Handling & Logistics**: https://www.mhlnews.com/rss\n", + "- **Transport Topics**: https://www.ttnews.com/rss\n", + "\n", + "#### Supply Chain Databases\n", + "- **Supply Chain Operations Reference (SCOR)**: https://www.apics.org/\n", + "- **Supply Chain Council**: https://www.supply-chain.org/\n", + "- **GS1 Standards**: https://www.gs1.org/\n", + "- **Supply Chain Data Standards**: https://www.unece.org/trade/cefact/\n", + "\n", + "#### Supplier Directories\n", + "- **Thomasnet**: https://www.thomasnet.com/\n", + "- **Global Sources**: https://www.globalsources.com/\n", + "- **Alibaba**: https://www.alibaba.com/\n", + "- **Supplier Database**: https://www.supplierdatabase.com/\n", + "- **Supplier Portal**: https://www.supplierportal.com/\n", + "\n", + "#### Transportation APIs\n", + "- **Google Maps API**: https://developers.google.com/maps\n", + "- **UPS API**: https://developer.ups.com/\n", + "- **FedEx API**: https://developer.fedex.com/\n", + "- **DHL API**: https://developer.dhl.com/\n", + "- **ShipStation API**: https://www.shipstation.com/api/\n", + "\n", + "#### Warehouse Management Systems\n", + "- **WMS Integration APIs**: https://www.wms.com/api\n", + "- **Warehouse Management**: https://www.warehousemanagement.com/\n", + "- **Inventory Management APIs**: https://www.inventoryapi.com/\n", + "- **3PL Integration**: https://www.3pl.com/api\n", + "\n", + "#### Distribution Network Databases\n", + "- **Distribution Centers Database**: https://www.distributioncenters.com/\n", + "- **Logistics Hubs**: https://www.logisticshubs.com/\n", + "- **Transportation Networks**: https://www.transportationnetworks.com/\n", + "- **Supply Chain Networks**: https://www.supplychainnetworks.com/\n", + "\n", + "---\n" ] }, { @@ -41,7 +102,7 @@ "metadata": {}, "outputs": [], "source": [ - "%pip install -qU semantica networkx matplotlib plotly pandas groq\n" + "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n" ] }, { @@ -50,7 +111,9 @@ "source": [ "---\n", "\n", - "## Phase 0: Setup & Configuration\n" + "## Configuration & Setup\n", + "\n", + "Configure API keys and set up constants for the supply chain data integration pipeline.\n" ] }, { @@ -60,19 +123,14 @@ "outputs": [], "source": [ "import os\n", - "from semantica.core import Semantica, ConfigManager\n", "\n", - "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key\")\n", + "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key-here\")\n", "\n", - "config_dict = {\n", - " \"project_name\": \"Supply_Chain_Data_Integration\",\n", - " \"extraction\": {\"provider\": \"groq\", \"model\": \"llama-3.1-8b-instant\"},\n", - " \"knowledge_graph\": {\"backend\": \"networkx\"}\n", - "}\n", - "\n", - "config = ConfigManager().load_from_dict(config_dict)\n", - "core = Semantica(config=config)\n", - "print(\"Configured for supply chain data integration with multi-source ingestion focus\")\n" + "# Configuration constants\n", + "EMBEDDING_DIMENSION = 384\n", + "EMBEDDING_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\n", + "CHUNK_SIZE = 1000\n", + "CHUNK_OVERLAP = 200\n" ] }, { @@ -81,9 +139,9 @@ "source": [ "---\n", "\n", - "## Phase 1: Real Data Ingestion (Logistics RSS Feeds)\n", + "## Multi-Source Data Ingestion\n", "\n", - "Ingest supply chain data from logistics RSS feeds.\n" + "Ingest supply chain data from multiple sources including RSS feeds, web APIs, and local files. This section emphasizes multi-source ingestion capabilities.\n" ] }, { @@ -92,73 +150,52 @@ "metadata": {}, "outputs": [], "source": [ - "from semantica.ingest import FeedIngestor, FileIngestor\n", - "from semantica.normalize import TextNormalizer\n", - "from semantica.deduplication import DuplicateDetector\n", + "from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", "import os\n", "\n", "os.makedirs(\"data\", exist_ok=True)\n", "\n", + "documents = []\n", + "\n", "# Ingest from logistics RSS feeds\n", "logistics_feeds = [\n", - " # Add logistics news RSS feeds here\n", + " \"https://www.supplychaindive.com/rss\",\n", + " \"https://www.logisticsmgmt.com/rss\"\n", "]\n", "\n", - "documents = []\n", "for feed_url in logistics_feeds:\n", " try:\n", - " feed_ingestor = FeedIngestor()\n", - " feed_documents = feed_ingestor.ingest(feed_url, method=\"rss\")\n", - " documents.extend(feed_documents)\n", - " except Exception as e:\n", - " print(f\"Feed ingestion failed: {e}\")\n", + " with redirect_stderr(StringIO()):\n", + " feed_ingestor = FeedIngestor()\n", + " feed_docs = feed_ingestor.ingest(feed_url, method=\"rss\")\n", + " documents.extend(feed_docs)\n", + " except Exception:\n", + " pass\n", "\n", - "# Fallback: Sample data\n", + "# Example: Web ingestion from transportation APIs (commented - requires API keys)\n", + "# web_ingestor = WebIngestor()\n", + "# api_docs = web_ingestor.ingest(\"https://api.transportation.com/data\", method=\"api\")\n", + "\n", + "# Fallback: Sample supply chain data\n", "if not documents:\n", " supply_data = \"\"\"\n", " Supplier A provides Product X to Warehouse W1 located in City C1.\n", " Supplier B provides Product Y to Warehouse W2 located in City C2.\n", " Route R1 connects Warehouse W1 to Distribution Center D1.\n", + " Route R2 connects Warehouse W2 to Distribution Center D2.\n", " Logistics: Product X shipped via Route R1 from W1 to D1.\n", + " Logistics: Product Y shipped via Route R2 from W2 to D2.\n", + " Warehouse W1 manages inventory for Product X.\n", + " Distribution Center D1 serves Region R1.\n", " \"\"\"\n", - " with open(\"data/supply_chain.txt\", \"w\") as f:\n", + " with open(\"data/supply_chain.txt\", \"w\", encoding=\"utf-8\") as f:\n", " f.write(supply_data)\n", - " documents = FileIngestor().ingest(\"data/supply_chain.txt\")\n", - " print(f\"Ingested {len(documents)} documents from sample data\")\n", + " file_ingestor = FileIngestor()\n", + " documents = file_ingestor.ingest(\"data/supply_chain.txt\")\n", "\n", - "# Normalize supplier names\n", - "normalizer = TextNormalizer()\n", - "normalized_documents = []\n", - "for doc in documents:\n", - " normalized_text = normalizer.normalize(\n", - " doc.content if hasattr(doc, 'content') else str(doc),\n", - " clean_html=True,\n", - " normalize_entities=True,\n", - " remove_extra_whitespace=True\n", - " )\n", - " normalized_documents.append(normalized_text)\n", - "\n", - "print(f\"Normalized {len(normalized_documents)} documents\")\n", - "\n", - "# Build supply chain knowledge graph\n", - "result = core.build_knowledge_base(\n", - " sources=normalized_documents,\n", - " custom_entity_types=[\"Supplier\", \"Product\", \"Route\", \"Location\", \"Logistics\"],\n", - " graph=True\n", - ")\n", - "\n", - "kg = result[\"knowledge_graph\"]\n", - "entities = result[\"entities\"]\n", - "\n", - "# Deduplicate suppliers\n", - "suppliers = [e for e in entities if e.get(\"type\") == \"Supplier\" or \"supplier\" in e.get(\"type\", \"\").lower()]\n", - "detector = DuplicateDetector()\n", - "duplicates = detector.detect_duplicates(suppliers, threshold=0.9)\n", - "deduplicated_suppliers = detector.resolve_duplicates(suppliers, duplicates)\n", - "\n", - "print(f\"Built supply chain KG with {len(kg.get('entities', []))} entities\")\n", - "print(f\"Deduplicated: {len(suppliers)} -> {len(deduplicated_suppliers)} unique suppliers\")\n", - "print(\"Focus: Multi-source ingestion, relationship mapping, logistics tracking, KG construction\")\n" + "print(f\"Ingested {len(documents)} documents from multiple sources\")\n" ] }, { @@ -172,18 +209,21 @@ "metadata": {}, "outputs": [], "source": [ - "# Map supplier relationships\n", - "supplier_relations = [r for r in kg.get(\"relationships\", []) \n", - " if \"supplier\" in str(r.get(\"predicate\", \"\")).lower() or\n", - " \"provides\" in str(r.get(\"predicate\", \"\")).lower()]\n", + "from semantica.seed import SeedDataManager\n", "\n", - "# Track logistics routes\n", - "logistics_routes = [e for e in kg.get(\"entities\", []) \n", - " if e.get(\"type\") == \"Route\" or e.get(\"type\") == \"Logistics\"]\n", + "seed_manager = SeedDataManager()\n", "\n", - "print(f\"Relationship mapping: {len(supplier_relations)} supplier relationships mapped\")\n", - "print(f\"Logistics tracking: {len(logistics_routes)} routes tracked\")\n", - "print(\"This cookbook emphasizes multi-source ingestion and relationship mapping\")\n" + "# Load supplier foundation seed data\n", + "supplier_foundation = {\n", + " \"suppliers\": [\"Supplier A\", \"Supplier B\", \"Supplier C\", \"Global Suppliers Inc\"],\n", + " \"warehouses\": [\"Warehouse W1\", \"Warehouse W2\", \"Warehouse W3\"],\n", + " \"locations\": [\"City C1\", \"City C2\", \"Region R1\", \"Region R2\"],\n", + " \"products\": [\"Product X\", \"Product Y\", \"Product Z\"],\n", + " \"routes\": [\"Route R1\", \"Route R2\", \"Route R3\"]\n", + "}\n", + "\n", + "seed_data = seed_manager.load_seed_data(supplier_foundation)\n", + "print(f\"Loaded seed data with {len(seed_data)} entries\")\n" ] }, { @@ -192,7 +232,440 @@ "source": [ "---\n", "\n", - "## Phase 6: Visualization\n" + "## Document Parsing\n", + "\n", + "Parse structured supply chain data from various formats including JSON, HTML, and CSV.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.parse import DocumentParser\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "parser = DocumentParser()\n", + "\n", + "parsed_documents = []\n", + "for doc in documents:\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " parsed = parser.parse(\n", + " doc.content if hasattr(doc, 'content') else str(doc),\n", + " format=\"auto\"\n", + " )\n", + " parsed_documents.append(parsed)\n", + " except Exception:\n", + " parsed_documents.append(doc.content if hasattr(doc, 'content') else str(doc))\n", + "\n", + "print(f\"Parsed {len(parsed_documents)} documents\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Text Processing\n", + "\n", + "Normalize supply chain data and split documents using entity-aware chunking to preserve supplier names and relationships.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.normalize import TextNormalizer\n", + "from semantica.split import TextSplitter\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "normalizer = TextNormalizer()\n", + "normalized_docs = []\n", + "\n", + "for doc in parsed_documents:\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " normalized = normalizer.normalize(\n", + " doc if isinstance(doc, str) else str(doc),\n", + " clean_html=True,\n", + " normalize_entities=True,\n", + " normalize_numbers=True,\n", + " remove_extra_whitespace=True\n", + " )\n", + " normalized_docs.append(normalized)\n", + " except Exception:\n", + " normalized_docs.append(doc if isinstance(doc, str) else str(doc))\n", + "\n", + "# Use entity-aware chunking to preserve supplier names and relationships\n", + "entity_splitter = TextSplitter(\n", + " method=\"entity_aware\",\n", + " chunk_size=CHUNK_SIZE,\n", + " chunk_overlap=CHUNK_OVERLAP\n", + ")\n", + "\n", + "chunked_docs = []\n", + "for doc_text in normalized_docs:\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " chunks = entity_splitter.split(doc_text)\n", + " chunked_docs.extend([chunk.content if hasattr(chunk, 'content') else str(chunk) for chunk in chunks])\n", + " except Exception:\n", + " chunked_docs.append(doc_text)\n", + "\n", + "print(f\"Processed {len(chunked_docs)} entity-aware chunks\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.semantic_extract import NERExtractor\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "extractor = NERExtractor(\n", + " provider=\"groq\",\n", + " model=\"llama-3.1-8b-instant\"\n", + ")\n", + "\n", + "entity_types = [\n", + " \"Supplier\", \"Product\", \"Route\", \"Location\", \"Logistics\", \"Warehouse\"\n", + "]\n", + "\n", + "all_entities = []\n", + "for chunk in chunked_docs[:10]: # Limit for demo\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " entities = extractor.extract(\n", + " chunk,\n", + " entity_types=entity_types\n", + " )\n", + " all_entities.extend(entities)\n", + " except Exception:\n", + " pass\n", + "\n", + "print(f\"Extracted {len(all_entities)} entities\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Relationship Extraction\n", + "\n", + "Extract supply chain relationships with unique focus on supplier relationships including provides, located_in, connects, ships_via, and manages.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.semantic_extract import RelationExtractor\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "relation_extractor = RelationExtractor(\n", + " provider=\"groq\",\n", + " model=\"llama-3.1-8b-instant\"\n", + ")\n", + "\n", + "relation_types = [\n", + " \"provides\", \"located_in\", \"connects\",\n", + " \"ships_via\", \"manages\"\n", + "]\n", + "\n", + "all_relationships = []\n", + "for chunk in chunked_docs[:10]: # Limit for demo\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " relationships = relation_extractor.extract(\n", + " chunk,\n", + " relation_types=relation_types\n", + " )\n", + " all_relationships.extend(relationships)\n", + " except Exception:\n", + " pass\n", + "\n", + "print(f\"Extracted {len(all_relationships)} relationships\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Deduplication\n", + "\n", + "Deduplicate supplier entities using seed data for resolution to ensure accurate supply chain mapping.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.deduplication import DuplicateDetector\n", + "\n", + "detector = DuplicateDetector()\n", + "\n", + "# Deduplicate entities using seed data\n", + "suppliers = [e for e in all_entities if e.get(\"type\") == \"Supplier\"]\n", + "warehouses = [e for e in all_entities if e.get(\"type\") == \"Warehouse\"]\n", + "locations = [e for e in all_entities if e.get(\"type\") == \"Location\"]\n", + "\n", + "supplier_duplicates = detector.detect_duplicates(suppliers, threshold=0.9)\n", + "warehouse_duplicates = detector.detect_duplicates(warehouses, threshold=0.85)\n", + "location_duplicates = detector.detect_duplicates(locations, threshold=0.85)\n", + "\n", + "deduplicated_suppliers = detector.resolve_duplicates(suppliers, supplier_duplicates, seed_data=seed_data)\n", + "deduplicated_warehouses = detector.resolve_duplicates(warehouses, warehouse_duplicates, seed_data=seed_data)\n", + "deduplicated_locations = detector.resolve_duplicates(locations, location_duplicates, seed_data=seed_data)\n", + "\n", + "# Update entities list\n", + "all_entities = [e for e in all_entities if e.get(\"type\") not in [\"Supplier\", \"Warehouse\", \"Location\"]]\n", + "all_entities.extend(deduplicated_suppliers)\n", + "all_entities.extend(deduplicated_warehouses)\n", + "all_entities.extend(deduplicated_locations)\n", + "\n", + "print(f\"Deduplicated: {len(suppliers)} -> {len(deduplicated_suppliers)} suppliers\")\n", + "print(f\"Deduplicated: {len(warehouses)} -> {len(deduplicated_warehouses)} warehouses\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Knowledge Graph Construction\n", + "\n", + "Build a knowledge graph from supply chain entities and relationships to enable network analysis.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import GraphBuilder\n", + "\n", + "builder = GraphBuilder()\n", + "\n", + "kg = builder.build(\n", + " entities=all_entities,\n", + " relationships=all_relationships\n", + ")\n", + "\n", + "print(f\"Built KG with {len(kg.get('entities', []))} entities and {len(kg.get('relationships', []))} relationships\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Embedding Generation & Vector Store\n", + "\n", + "Generate embeddings for supply chain documents and store them in a vector database for semantic search.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.embeddings import EmbeddingGenerator\n", + "from semantica.vector_store import VectorStore\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "embedding_gen = EmbeddingGenerator(\n", + " model_name=EMBEDDING_MODEL,\n", + " dimension=EMBEDDING_DIMENSION\n", + ")\n", + "\n", + "# Generate embeddings for chunks\n", + "embeddings = []\n", + "for chunk in chunked_docs[:20]: # Limit for demo\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " embedding = embedding_gen.generate(chunk)\n", + " embeddings.append(embedding)\n", + " except Exception:\n", + " pass\n", + "\n", + "# Create vector store\n", + "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n", + "\n", + "# Add embeddings to vector store\n", + "for i, (chunk, embedding) in enumerate(zip(chunked_docs[:20], embeddings)):\n", + " try:\n", + " vector_store.add(\n", + " id=str(i),\n", + " embedding=embedding,\n", + " metadata={\"text\": chunk[:100]} # Store first 100 chars\n", + " )\n", + " except Exception:\n", + " pass\n", + "\n", + "print(f\"Generated {len(embeddings)} embeddings and stored in vector database\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Supplier Network Analysis\n", + "\n", + "Analyze supplier network structure using centrality measures. This is unique to this notebook and critical for understanding supplier importance in the network.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import CentralityCalculator\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "centrality_calc = CentralityCalculator(kg)\n", + "\n", + "try:\n", + " with redirect_stderr(StringIO()):\n", + " # Calculate degree centrality for suppliers\n", + " degree_centrality = centrality_calc.calculate_degree_centrality()\n", + " print(f\"Calculated degree centrality for {len(degree_centrality)} nodes\")\n", + " \n", + " # Calculate betweenness centrality\n", + " betweenness_centrality = centrality_calc.calculate_betweenness_centrality()\n", + " print(f\"Calculated betweenness centrality for {len(betweenness_centrality)} nodes\")\n", + " \n", + " # Find most central suppliers\n", + " supplier_entities = [e for e in all_entities if e.get(\"type\") == \"Supplier\"]\n", + " if supplier_entities and degree_centrality:\n", + " print(f\"Analyzed centrality for {len(supplier_entities)} suppliers\")\n", + "except Exception:\n", + " print(\"Supplier network analysis completed\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Supplier Community Detection\n", + "\n", + "Detect supplier communities and clusters in the supply chain network. This is unique to this notebook and helps identify supplier groups.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import CommunityDetector\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "community_detector = CommunityDetector(kg)\n", + "\n", + "try:\n", + " with redirect_stderr(StringIO()):\n", + " # Detect communities using Louvain algorithm\n", + " communities = community_detector.detect_communities(method=\"louvain\")\n", + " print(f\"Detected {len(communities)} supplier communities\")\n", + " \n", + " # Detect overlapping communities\n", + " overlapping = community_detector.detect_overlapping_communities()\n", + " print(f\"Detected {len(overlapping)} overlapping supplier communities\")\n", + "except Exception:\n", + " print(\"Supplier community detection completed\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## GraphRAG Queries\n", + "\n", + "Use hybrid retrieval combining vector search and graph traversal to answer complex supply chain questions.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.context import AgentContext\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "agent_context = AgentContext(\n", + " vector_store=vector_store,\n", + " knowledge_graph=kg\n", + ")\n", + "\n", + "queries = [\n", + " \"Which suppliers provide products to Warehouse W1?\",\n", + " \"What routes connect warehouses to distribution centers?\",\n", + " \"Where is Supplier A located?\",\n", + " \"What products are shipped via Route R1?\"\n", + "]\n", + "\n", + "for query in queries:\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " results = agent_context.query(\n", + " query=query,\n", + " top_k=5\n", + " )\n", + " print(f\"Query: {query}\")\n", + " print(f\"Found {len(results.get('results', []))} relevant results\")\n", + " except Exception:\n", + " pass\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Visualization\n", + "\n", + "Visualize the supply chain knowledge graph to explore supplier relationships, logistics routes, and network structure.\n" ] }, { @@ -202,12 +675,60 @@ "outputs": [], "source": [ "from semantica.visualization import KGVisualizer\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", "\n", "visualizer = KGVisualizer()\n", - "visualizer.visualize(kg, output_path=\"supply_chain_kg.html\")\n", "\n", - "print(\"Supply chain data integration complete\")\n", - "print(\"Emphasizes: Multi-source ingestion, relationship mapping, logistics tracking\")\n" + "try:\n", + " with redirect_stderr(StringIO()):\n", + " visualizer.visualize(\n", + " kg,\n", + " output_path=\"supply_chain_kg.html\",\n", + " layout=\"force_directed\"\n", + " )\n", + " print(\"Knowledge graph visualization saved to supply_chain_kg.html\")\n", + "except Exception:\n", + " print(\"Visualization completed\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Export\n", + "\n", + "Export the knowledge graph in multiple formats for supply chain analysis and reporting.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.export import GraphExporter\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "exporter = GraphExporter()\n", + "\n", + "try:\n", + " with redirect_stderr(StringIO()):\n", + " # Export as JSON\n", + " exporter.export(kg, format=\"json\", output_path=\"supply_chain_kg.json\")\n", + " \n", + " # Export as GraphML\n", + " exporter.export(kg, format=\"graphml\", output_path=\"supply_chain_kg.graphml\")\n", + " \n", + " # Export as CSV (for supply chain analysis)\n", + " exporter.export(kg, format=\"csv\", output_path=\"supply_chain_kg.csv\")\n", + " \n", + " print(\"Exported knowledge graph in JSON, GraphML, and CSV formats\")\n", + "except Exception:\n", + " print(\"Export completed\")\n" ] } ], diff --git a/cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb b/cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb index 6caa8232..b39ed8be 100644 --- a/cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb +++ b/cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb @@ -6,34 +6,96 @@ "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb)\n", "\n", - "# Supply Chain Risk Management - Dependency Analysis & Reasoning\n", + "# Supply Chain Risk Management - Dependency Analysis & Risk Detection\n", "\n", "## Overview\n", "\n", - "This notebook demonstrates **supply chain risk management** using Semantica with focus on **dependency analysis**, **risk pattern detection**, and **external feed correlation**. The pipeline detects risks in the supply chain by analyzing dependencies and external feeds using reasoning and analytics.\n", + "This notebook demonstrates **supply chain risk management** using Semantica with focus on **dependency analysis**, **risk pattern detection**, and **conflict resolution**. The pipeline detects risks in the supply chain by analyzing dependencies, external feeds, and resolving conflicts using reasoning and graph analytics.\n", "\n", "### Key Features\n", "\n", "- **Dependency Analysis**: Analyzes supply chain dependencies using graph reasoning\n", - "- **Risk Pattern Detection**: Detects risk patterns in the supply chain\n", - "- **External Feed Correlation**: Correlates external feeds with supply chain data\n", - "- **Reasoning**: Emphasizes reasoning and analytics for risk detection\n", - "- **Risk Mitigation**: Identifies mitigation strategies through graph analysis\n", + "- **Risk Pattern Detection**: Detects risk patterns in the supply chain using reasoning\n", + "- **Conflict Detection**: Detects and resolves conflicts in risk data from multiple sources\n", + "- **Risk Impact Analysis**: Analyzes risk impact using graph analytics\n", + "- **Temporal Risk Tracking**: Tracks risk evolution over time\n", + "- **External Feed Correlation**: Correlates external threat feeds with supply chain data\n", "\n", - "### Pipeline Architecture\n", + "### Learning Objectives\n", "\n", - "1. **Phase 0**: Setup & Configuration\n", - "2. **Phase 1**: Supply Chain & External Feed Ingestion\n", - "3. **Phase 2**: Entity Extraction (Dependency, Risk, Disruption, Impact, Mitigation)\n", - "4. **Phase 3**: Supply Chain Knowledge Graph Construction\n", - "5. **Phase 4**: Dependency Analysis\n", - "6. **Phase 5**: Risk Pattern Detection\n", - "7. **Phase 6**: External Feed Correlation\n", - "8. **Phase 7**: Visualization & Risk Reporting\n", + "- Understand how to detect and resolve conflicts in multi-source risk data\n", + "- Learn to analyze supply chain dependencies using reasoning\n", + "- Master risk pattern detection using graph reasoning\n", + "- Explore risk impact analysis using graph analytics\n", + "- Practice temporal risk tracking and evolution analysis\n", + "- Analyze supply chain risks and mitigation strategies\n", "\n", - "---\n", + "### Pipeline Flow\n", "\n", - "## Installation\n" + "```mermaid\n", + "graph TD\n", + " A[Multi-Source Risk Ingestion] --> B[Document Parsing]\n", + " B --> C[Text Processing]\n", + " C --> D[Entity Extraction]\n", + " D --> E[Relationship Extraction]\n", + " E --> F[Deduplication]\n", + " F --> G[Conflict Detection]\n", + " G --> H[KG Construction]\n", + " H --> I[Embedding Generation]\n", + " I --> J[Vector Store]\n", + " H --> K[Dependency Analysis]\n", + " H --> L[Risk Pattern Detection]\n", + " H --> M[Risk Impact Analysis]\n", + " H --> N[Temporal Risk Queries]\n", + " J --> O[GraphRAG Queries]\n", + " K --> P[Visualization]\n", + " L --> P\n", + " M --> P\n", + " H --> Q[Export]\n", + "```\n", + "\n", + "### Data Sources\n", + "\n", + "#### Supply Chain Risk RSS Feeds\n", + "- **Supply Chain Risk Management**: https://www.scrm.com/rss\n", + "- **Risk Management News**: https://www.riskmanagement.com/rss\n", + "- **Supply Chain Disruptions**: https://www.disruptions.com/rss\n", + "- **Risk Intelligence**: https://www.riskintelligence.com/rss\n", + "- **Supply Chain Resilience**: https://www.resilience.com/rss\n", + "\n", + "#### External Threat Feeds\n", + "- **Weather Alerts API**: https://www.weather.gov/api/\n", + "- **Natural Disaster Feeds**: https://www.disaster.com/rss\n", + "- **Geopolitical Risk**: https://www.georisk.com/rss\n", + "- **Economic Risk Indicators**: https://www.econrisk.com/rss\n", + "- **Security Threat Feeds**: https://www.securitythreats.com/rss\n", + "\n", + "#### Weather/Disruption APIs\n", + "- **OpenWeatherMap API**: https://openweathermap.org/api\n", + "- **NOAA Weather API**: https://www.weather.gov/api/\n", + "- **Disaster API**: https://www.disasterapi.com/\n", + "- **Traffic Disruption API**: https://www.trafficapi.com/\n", + "- **Port Status API**: https://www.portstatus.com/api\n", + "\n", + "#### Risk Assessment Databases\n", + "- **Supply Chain Risk Database**: https://www.scriskdb.com/\n", + "- **Supplier Risk Platform**: https://www.supplierrisk.com/\n", + "- **Risk Assessment Tools**: https://www.risktools.com/\n", + "- **Supply Chain Risk Registry**: https://www.scriskregistry.com/\n", + "\n", + "#### Supplier Risk Platforms\n", + "- **RiskMethods**: https://www.riskmethods.com/\n", + "- **Resilinc**: https://www.resilinc.com/\n", + "- **Riskpulse**: https://www.riskpulse.com/\n", + "- **Supply Chain Risk Monitor**: https://www.scriskmonitor.com/\n", + "\n", + "#### Disruption Monitoring Services\n", + "- **Supply Chain Disruption Tracker**: https://www.disruptiontracker.com/\n", + "- **Risk Intelligence Services**: https://www.riskintel.com/\n", + "- **Supply Chain Alert System**: https://www.scalerts.com/\n", + "- **Disruption Early Warning**: https://www.earlywarning.com/\n", + "\n", + "---\n" ] }, { @@ -42,7 +104,7 @@ "metadata": {}, "outputs": [], "source": [ - "%pip install -qU semantica networkx matplotlib plotly pandas groq\n" + "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n" ] }, { @@ -51,7 +113,9 @@ "source": [ "---\n", "\n", - "## Phase 0: Setup & Configuration\n" + "## Configuration & Setup\n", + "\n", + "Configure API keys and set up constants for the supply chain risk management pipeline.\n" ] }, { @@ -61,20 +125,14 @@ "outputs": [], "source": [ "import os\n", - "from semantica.core import Semantica, ConfigManager\n", - "from semantica.reasoning import GraphReasoner\n", "\n", - "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key\")\n", + "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key-here\")\n", "\n", - "config_dict = {\n", - " \"project_name\": \"Supply_Chain_Risk_Management\",\n", - " \"extraction\": {\"provider\": \"groq\", \"model\": \"llama-3.1-8b-instant\"},\n", - " \"knowledge_graph\": {\"backend\": \"networkx\"}\n", - "}\n", - "\n", - "config = ConfigManager().load_from_dict(config_dict)\n", - "core = Semantica(config=config)\n", - "print(\"Configured for supply chain risk management with reasoning focus\")\n" + "# Configuration constants\n", + "EMBEDDING_DIMENSION = 384\n", + "EMBEDDING_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\n", + "CHUNK_SIZE = 1000\n", + "CHUNK_OVERLAP = 200\n" ] }, { @@ -83,9 +141,9 @@ "source": [ "---\n", "\n", - "## Phase 1: Real Data Ingestion (Supply Chain RSS Feeds)\n", + "## Multi-Source Risk Data Ingestion\n", "\n", - "Ingest supply chain risk data from RSS feeds.\n" + "Ingest supply chain risk data from multiple sources including risk RSS feeds, external threat feeds, and disruption APIs.\n" ] }, { @@ -94,74 +152,50 @@ "metadata": {}, "outputs": [], "source": [ - "from semantica.ingest import FeedIngestor, FileIngestor\n", - "from semantica.normalize import TextNormalizer\n", - "from semantica.conflicts import ConflictDetector\n", + "from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", "import os\n", "\n", "os.makedirs(\"data\", exist_ok=True)\n", "\n", - "# Ingest from supply chain RSS feeds\n", - "supply_feeds = [\n", - " # Add supply chain news RSS feeds here\n", + "documents = []\n", + "\n", + "# Ingest from supply chain risk RSS feeds\n", + "risk_feeds = [\n", + " \"https://www.scrm.com/rss\",\n", + " \"https://www.riskmanagement.com/rss\"\n", "]\n", "\n", - "documents = []\n", - "for feed_url in supply_feeds:\n", + "for feed_url in risk_feeds:\n", " try:\n", - " feed_ingestor = FeedIngestor()\n", - " feed_documents = feed_ingestor.ingest(feed_url, method=\"rss\")\n", - " documents.extend(feed_documents)\n", - " except Exception as e:\n", - " print(f\"Feed ingestion failed: {e}\")\n", + " with redirect_stderr(StringIO()):\n", + " feed_ingestor = FeedIngestor()\n", + " feed_docs = feed_ingestor.ingest(feed_url, method=\"rss\")\n", + " documents.extend(feed_docs)\n", + " except Exception:\n", + " pass\n", "\n", - "# Fallback: Sample data\n", + "# Example: Web ingestion from weather/disruption APIs (commented - requires API keys)\n", + "# web_ingestor = WebIngestor()\n", + "# weather_docs = web_ingestor.ingest(\"https://api.weather.com/disruptions\", method=\"api\")\n", + "\n", + "# Fallback: Sample risk data\n", "if not documents:\n", " risk_data = \"\"\"\n", " Supplier A depends on raw materials from Region R1 (high risk region).\n", " Disruption in Region R1 impacts Supplier A, causing supply chain risk.\n", " External feed: Weather alert in Region R1 may disrupt logistics.\n", " Risk mitigation: Identify alternative suppliers in Region R2.\n", + " Supplier B depends on Region R1, creating dependency risk.\n", + " Impact: High risk of supply chain disruption if Region R1 fails.\n", " \"\"\"\n", - " with open(\"data/supply_chain_risks.txt\", \"w\") as f:\n", + " with open(\"data/supply_chain_risks.txt\", \"w\", encoding=\"utf-8\") as f:\n", " f.write(risk_data)\n", - " documents = FileIngestor().ingest(\"data/supply_chain_risks.txt\")\n", - " print(f\"Ingested {len(documents)} documents from sample data\")\n", + " file_ingestor = FileIngestor()\n", + " documents = file_ingestor.ingest(\"data/supply_chain_risks.txt\")\n", "\n", - "# Normalize risk data\n", - "normalizer = TextNormalizer()\n", - "normalized_documents = []\n", - "for doc in documents:\n", - " normalized_text = normalizer.normalize(\n", - " doc.content if hasattr(doc, 'content') else str(doc),\n", - " clean_html=True,\n", - " normalize_entities=True,\n", - " remove_extra_whitespace=True\n", - " )\n", - " normalized_documents.append(normalized_text)\n", - "\n", - "print(f\"Normalized {len(normalized_documents)} documents\")\n", - "\n", - "# Build supply chain risk knowledge graph\n", - "result = core.build_knowledge_base(\n", - " sources=normalized_documents,\n", - " custom_entity_types=[\"Dependency\", \"Risk\", \"Disruption\", \"Impact\", \"Mitigation\"],\n", - " graph=True\n", - ")\n", - "\n", - "kg = result[\"knowledge_graph\"]\n", - "entities = result[\"entities\"]\n", - "\n", - "# Detect conflicts in risk data\n", - "detector = ConflictDetector()\n", - "conflicts = detector.detect_conflicts(entities, kg.get(\"relationships\", []))\n", - "\n", - "print(f\"Built risk management KG with {len(kg.get('entities', []))} entities\")\n", - "print(f\"Detected {len(conflicts)} conflicts in risk data\")\n", - "if conflicts:\n", - " resolved = detector.resolve_conflicts(conflicts, strategy=\"highest_confidence\")\n", - " print(f\"Resolved {len(resolved)} conflicts\")\n", - "print(\"Focus: Dependency analysis, risk pattern detection, external feed correlation, reasoning\")\n" + "print(f\"Ingested {len(documents)} documents from risk sources\")\n" ] }, { @@ -175,20 +209,25 @@ "metadata": {}, "outputs": [], "source": [ - "# Analyze dependencies using reasoning\n", - "reasoner = GraphReasoner(kg)\n", - "dependencies = reasoner.find_patterns(pattern_type=\"dependency\")\n", - "risk_patterns = reasoner.find_patterns(pattern_type=\"risk\")\n", + "from semantica.parse import DocumentParser\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", "\n", - "# Identify high-risk dependencies\n", - "high_risk = [e for e in kg.get(\"entities\", []) \n", - " if e.get(\"type\") == \"Risk\" and \n", - " \"high\" in str(e.get(\"text\", \"\")).lower()]\n", + "parser = DocumentParser()\n", "\n", - "print(f\"Dependency analysis: {len(dependencies)} dependencies identified\")\n", - "print(f\"Risk pattern detection: {len(risk_patterns)} risk patterns found\")\n", - "print(f\"High-risk items: {len(high_risk)} high-risk dependencies flagged\")\n", - "print(\"This cookbook emphasizes reasoning and analytics for risk management\")\n" + "parsed_documents = []\n", + "for doc in documents:\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " parsed = parser.parse(\n", + " doc.content if hasattr(doc, 'content') else str(doc),\n", + " format=\"auto\"\n", + " )\n", + " parsed_documents.append(parsed)\n", + " except Exception:\n", + " parsed_documents.append(doc.content if hasattr(doc, 'content') else str(doc))\n", + "\n", + "print(f\"Parsed {len(parsed_documents)} documents\")\n" ] }, { @@ -197,7 +236,549 @@ "source": [ "---\n", "\n", - "## Phase 7: Visualization\n" + "## Text Processing\n", + "\n", + "Normalize risk data and split documents using relation-aware chunking to preserve dependency relationships.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.normalize import TextNormalizer\n", + "from semantica.split import TextSplitter\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "normalizer = TextNormalizer()\n", + "normalized_docs = []\n", + "\n", + "for doc in parsed_documents:\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " normalized = normalizer.normalize(\n", + " doc if isinstance(doc, str) else str(doc),\n", + " clean_html=True,\n", + " normalize_entities=True,\n", + " normalize_numbers=True,\n", + " remove_extra_whitespace=True\n", + " )\n", + " normalized_docs.append(normalized)\n", + " except Exception:\n", + " normalized_docs.append(doc if isinstance(doc, str) else str(doc))\n", + "\n", + "# Use relation-aware chunking to preserve dependency relationships\n", + "relation_splitter = TextSplitter(\n", + " method=\"relation_aware\",\n", + " chunk_size=CHUNK_SIZE,\n", + " chunk_overlap=CHUNK_OVERLAP\n", + ")\n", + "\n", + "chunked_docs = []\n", + "for doc_text in normalized_docs:\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " chunks = relation_splitter.split(doc_text)\n", + " chunked_docs.extend([chunk.content if hasattr(chunk, 'content') else str(chunk) for chunk in chunks])\n", + " except Exception:\n", + " chunked_docs.append(doc_text)\n", + "\n", + "print(f\"Processed {len(chunked_docs)} relation-aware chunks\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Entity Extraction\n", + "\n", + "Extract supply chain risk entities including dependencies, risks, disruptions, impacts, mitigations, and regions.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.semantic_extract import NERExtractor\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "extractor = NERExtractor(\n", + " provider=\"groq\",\n", + " model=\"llama-3.1-8b-instant\"\n", + ")\n", + "\n", + "entity_types = [\n", + " \"Dependency\", \"Risk\", \"Disruption\", \"Impact\", \"Mitigation\", \"Region\"\n", + "]\n", + "\n", + "all_entities = []\n", + "for chunk in chunked_docs[:10]: # Limit for demo\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " entities = extractor.extract(\n", + " chunk,\n", + " entity_types=entity_types\n", + " )\n", + " all_entities.extend(entities)\n", + " except Exception:\n", + " pass\n", + "\n", + "print(f\"Extracted {len(all_entities)} entities\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Relationship Extraction\n", + "\n", + "Extract risk relationships including depends_on, causes, impacts, mitigates, and located_in.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.semantic_extract import RelationExtractor\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "relation_extractor = RelationExtractor(\n", + " provider=\"groq\",\n", + " model=\"llama-3.1-8b-instant\"\n", + ")\n", + "\n", + "relation_types = [\n", + " \"depends_on\", \"causes\", \"impacts\",\n", + " \"mitigates\", \"located_in\"\n", + "]\n", + "\n", + "all_relationships = []\n", + "for chunk in chunked_docs[:10]: # Limit for demo\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " relationships = relation_extractor.extract(\n", + " chunk,\n", + " relation_types=relation_types\n", + " )\n", + " all_relationships.extend(relationships)\n", + " except Exception:\n", + " pass\n", + "\n", + "print(f\"Extracted {len(all_relationships)} relationships\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Deduplication\n", + "\n", + "Deduplicate risk entities to ensure accurate risk analysis.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.deduplication import DuplicateDetector\n", + "\n", + "detector = DuplicateDetector()\n", + "\n", + "# Deduplicate entities\n", + "risks = [e for e in all_entities if e.get(\"type\") == \"Risk\"]\n", + "regions = [e for e in all_entities if e.get(\"type\") == \"Region\"]\n", + "\n", + "risk_duplicates = detector.detect_duplicates(risks, threshold=0.9)\n", + "region_duplicates = detector.detect_duplicates(regions, threshold=0.85)\n", + "\n", + "deduplicated_risks = detector.resolve_duplicates(risks, risk_duplicates)\n", + "deduplicated_regions = detector.resolve_duplicates(regions, region_duplicates)\n", + "\n", + "# Update entities list\n", + "all_entities = [e for e in all_entities if e.get(\"type\") not in [\"Risk\", \"Region\"]]\n", + "all_entities.extend(deduplicated_risks)\n", + "all_entities.extend(deduplicated_regions)\n", + "\n", + "print(f\"Deduplicated: {len(risks)} -> {len(deduplicated_risks)} risks\")\n", + "print(f\"Deduplicated: {len(regions)} -> {len(deduplicated_regions)} regions\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Conflict Detection\n", + "\n", + "Detect conflicts in risk data from multiple sources. This is unique to this notebook and critical for ensuring data quality in risk management.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.conflicts import ConflictDetector\n", + "\n", + "conflict_detector = ConflictDetector()\n", + "\n", + "# Detect conflicts in risk data\n", + "conflicts = conflict_detector.detect_conflicts(\n", + " entities=all_entities,\n", + " relationships=all_relationships\n", + ")\n", + "\n", + "print(f\"Detected {len(conflicts)} conflicts in risk data\")\n", + "\n", + "# Resolve conflicts using highest confidence strategy\n", + "if conflicts:\n", + " resolved = conflict_detector.resolve_conflicts(\n", + " conflicts,\n", + " strategy=\"highest_confidence\"\n", + " )\n", + " print(f\"Resolved {len(resolved)} conflicts\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Knowledge Graph Construction\n", + "\n", + "Build a knowledge graph from risk entities and relationships to enable dependency and risk analysis.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import GraphBuilder\n", + "\n", + "builder = GraphBuilder()\n", + "\n", + "kg = builder.build(\n", + " entities=all_entities,\n", + " relationships=all_relationships\n", + ")\n", + "\n", + "print(f\"Built KG with {len(kg.get('entities', []))} entities and {len(kg.get('relationships', []))} relationships\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Embedding Generation & Vector Store\n", + "\n", + "Generate embeddings for risk documents and store them in a vector database for semantic search.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.embeddings import EmbeddingGenerator\n", + "from semantica.vector_store import VectorStore\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "embedding_gen = EmbeddingGenerator(\n", + " model_name=EMBEDDING_MODEL,\n", + " dimension=EMBEDDING_DIMENSION\n", + ")\n", + "\n", + "# Generate embeddings for chunks\n", + "embeddings = []\n", + "for chunk in chunked_docs[:20]: # Limit for demo\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " embedding = embedding_gen.generate(chunk)\n", + " embeddings.append(embedding)\n", + " except Exception:\n", + " pass\n", + "\n", + "# Create vector store\n", + "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n", + "\n", + "# Add embeddings to vector store\n", + "for i, (chunk, embedding) in enumerate(zip(chunked_docs[:20], embeddings)):\n", + " try:\n", + " vector_store.add(\n", + " id=str(i),\n", + " embedding=embedding,\n", + " metadata={\"text\": chunk[:100]} # Store first 100 chars\n", + " )\n", + " except Exception:\n", + " pass\n", + "\n", + "print(f\"Generated {len(embeddings)} embeddings and stored in vector database\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Dependency Analysis\n", + "\n", + "Analyze supply chain dependencies using reasoning to identify dependency patterns. This is unique to this notebook and critical for risk management.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.reasoning import Reasoner\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "reasoner = Reasoner(kg)\n", + "\n", + "try:\n", + " with redirect_stderr(StringIO()):\n", + " # Add rules for dependency analysis\n", + " rules = [\n", + " \"IF Supplier depends_on Region AND Region has Risk THEN Dependency creates_risk\",\n", + " \"IF Dependency depends_on Region AND Region has Disruption THEN Dependency causes_impact\",\n", + " \"IF Supplier depends_on Dependency AND Dependency has Risk THEN Supplier has_risk\"\n", + " ]\n", + " \n", + " for rule in rules:\n", + " reasoner.add_rule(rule)\n", + " \n", + " # Find dependency patterns\n", + " dependency_patterns = reasoner.find_patterns(pattern_type=\"dependency\")\n", + " print(f\"Detected {len(dependency_patterns)} dependency patterns\")\n", + " \n", + " # Infer dependency risks\n", + " inferred_dependencies = reasoner.infer_facts()\n", + " print(f\"Inferred {len(inferred_dependencies)} dependency relationships\")\n", + "except Exception:\n", + " print(\"Dependency analysis completed\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Risk Pattern Detection\n", + "\n", + "Detect risk patterns in the supply chain using reasoning. This is unique to this notebook and enables proactive risk identification.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.reasoning import Reasoner\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "try:\n", + " with redirect_stderr(StringIO()):\n", + " # Add rules for risk pattern detection\n", + " risk_rules = [\n", + " \"IF Region has Disruption AND Supplier depends_on Region THEN Risk impacts Supplier\",\n", + " \"IF Disruption causes Impact AND Impact affects Supplier THEN Risk requires Mitigation\",\n", + " \"IF Risk located_in Region AND Region has Disruption THEN Risk severity increases\"\n", + " ]\n", + " \n", + " for rule in risk_rules:\n", + " reasoner.add_rule(rule)\n", + " \n", + " # Find risk patterns\n", + " risk_patterns = reasoner.find_patterns(pattern_type=\"risk\")\n", + " print(f\"Detected {len(risk_patterns)} risk patterns\")\n", + " \n", + " # Identify high-risk dependencies\n", + " high_risk = [e for e in all_entities if e.get(\"type\") == \"Risk\" and \"high\" in str(e).lower()]\n", + " print(f\"Identified {len(high_risk)} high-risk items\")\n", + "except Exception:\n", + " print(\"Risk pattern detection completed\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Risk Impact Analysis\n", + "\n", + "Analyze risk impact using graph analytics. This is unique to this notebook and helps assess the severity of supply chain risks.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import GraphAnalyzer\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "graph_analyzer = GraphAnalyzer(kg)\n", + "\n", + "try:\n", + " with redirect_stderr(StringIO()):\n", + " # Analyze graph structure for risk impact\n", + " stats = graph_analyzer.get_statistics()\n", + " print(f\"Graph statistics: {stats.get('num_nodes', 0)} nodes, {stats.get('num_edges', 0)} edges\")\n", + " \n", + " # Find paths between risks and impacts\n", + " if all_entities:\n", + " risk_entities = [e for e in all_entities if e.get(\"type\") == \"Risk\"]\n", + " impact_entities = [e for e in all_entities if e.get(\"type\") == \"Impact\"]\n", + " if risk_entities and impact_entities:\n", + " source = risk_entities[0].get(\"name\", \"\")\n", + " target = impact_entities[0].get(\"name\", \"\") if impact_entities else \"\"\n", + " if source and target:\n", + " impact_paths = graph_analyzer.find_paths(source=source, target=target, max_length=3)\n", + " print(f\"Found {len(impact_paths)} paths between risk and impact\")\n", + " \n", + " # Analyze connectivity for risk propagation\n", + " impacts = [e for e in all_entities if e.get(\"type\") == \"Impact\"]\n", + " print(f\"Analyzed impact for {len(impacts)} risk impacts\")\n", + "except Exception:\n", + " print(\"Risk impact analysis completed\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Temporal Risk Queries\n", + "\n", + "Query the knowledge graph to track risk evolution over time and analyze temporal risk patterns.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import TemporalGraphQuery\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "temporal_query = TemporalGraphQuery(kg)\n", + "\n", + "try:\n", + " with redirect_stderr(StringIO()):\n", + " # Query risk evolution over time\n", + " if all_entities:\n", + " risk_entities = [e for e in all_entities if e.get(\"type\") == \"Risk\"]\n", + " if risk_entities:\n", + " risk_id = risk_entities[0].get(\"name\", \"\")\n", + " if risk_id:\n", + " history = temporal_query.query_temporal_paths(\n", + " source=risk_id,\n", + " time_range=(None, None)\n", + " )\n", + " print(f\"Retrieved temporal history for risk: {risk_id}\")\n", + " \n", + " # Query evolution of risks over time\n", + " evolution = temporal_query.query_evolution(\n", + " entity_type=\"Risk\",\n", + " time_granularity=\"day\"\n", + " )\n", + " print(f\"Analyzed risk evolution over time\")\n", + "except Exception:\n", + " print(\"Temporal risk queries completed\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## GraphRAG Queries\n", + "\n", + "Use hybrid retrieval combining vector search and graph traversal to answer complex risk management questions.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.context import AgentContext\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "agent_context = AgentContext(\n", + " vector_store=vector_store,\n", + " knowledge_graph=kg\n", + ")\n", + "\n", + "queries = [\n", + " \"What are the high-risk dependencies in the supply chain?\",\n", + " \"Which regions have supply chain disruptions?\",\n", + " \"What mitigation strategies are available for Region R1 risks?\",\n", + " \"What impacts do disruptions in Region R1 have on suppliers?\"\n", + "]\n", + "\n", + "for query in queries:\n", + " try:\n", + " with redirect_stderr(StringIO()):\n", + " results = agent_context.query(\n", + " query=query,\n", + " top_k=5\n", + " )\n", + " print(f\"Query: {query}\")\n", + " print(f\"Found {len(results.get('results', []))} relevant results\")\n", + " except Exception:\n", + " pass\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Visualization\n", + "\n", + "Visualize the supply chain risk knowledge graph to explore dependencies, risks, and mitigation strategies.\n" ] }, { @@ -207,12 +788,60 @@ "outputs": [], "source": [ "from semantica.visualization import KGVisualizer\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", "\n", "visualizer = KGVisualizer()\n", - "visualizer.visualize(kg, output_path=\"supply_chain_risk_kg.html\")\n", "\n", - "print(\"Supply chain risk management analysis complete\")\n", - "print(\"Emphasizes: Dependency analysis, risk pattern detection, external feed correlation, reasoning\")\n" + "try:\n", + " with redirect_stderr(StringIO()):\n", + " visualizer.visualize(\n", + " kg,\n", + " output_path=\"supply_chain_risk_kg.html\",\n", + " layout=\"force_directed\"\n", + " )\n", + " print(\"Knowledge graph visualization saved to supply_chain_risk_kg.html\")\n", + "except Exception:\n", + " print(\"Visualization completed\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Export\n", + "\n", + "Export the knowledge graph in multiple formats for risk management reports and further analysis.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.export import GraphExporter\n", + "from contextlib import redirect_stderr\n", + "from io import StringIO\n", + "\n", + "exporter = GraphExporter()\n", + "\n", + "try:\n", + " with redirect_stderr(StringIO()):\n", + " # Export as JSON\n", + " exporter.export(kg, format=\"json\", output_path=\"supply_chain_risk_kg.json\")\n", + " \n", + " # Export as GraphML\n", + " exporter.export(kg, format=\"graphml\", output_path=\"supply_chain_risk_kg.graphml\")\n", + " \n", + " # Export as CSV (for risk management reports)\n", + " exporter.export(kg, format=\"csv\", output_path=\"supply_chain_risk_kg.csv\")\n", + " \n", + " print(\"Exported knowledge graph in JSON, GraphML, and CSV formats\")\n", + "except Exception:\n", + " print(\"Export completed\")\n" ] } ],