{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/06_Vulnerability_Tracking.ipynb)\n",
    "\n",
    "# Vulnerability Tracking Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete vulnerability tracking pipeline: ingest CVE data from multiple real sources (NVD, CVE feeds, security databases), build temporal knowledge graph, correlate vulnerabilities, predict impact, and generate vulnerability reports.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, RepoIngestor, EmailIngestor, MCPIngestor\n",
    "- **Parsing**: JSONParser, XMLParser, StructuredDataParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "- **KG**: GraphBuilder, TemporalGraphQuery, TemporalPatternDetector, GraphAnalyzer\n",
    "- **Analytics**: CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor, ConflictDetector\n",
    "- **Export**: JSONExporter, RDFExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Real CVE Sources → Parse → Extract Vulnerabilities → Build Temporal KG → Correlate → Predict Impact → Generate Reports → Visualize**\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Ingest CVE Data from Real Sources\n",
    "\n",
    "Ingest CVE data from NVD, CVE feeds, and security databases.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import WebIngestor, FeedIngestor, DBIngestor, FileIngestor\n",
    "from semantica.parse import JSONParser, XMLParser, StructuredDataParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "from semantica.kg import GraphBuilder, TemporalGraphQuery, TemporalPatternDetector, GraphAnalyzer\n",
    "from semantica.kg import CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    
    "from semantica.conflicts import ConflictDetector\n",
    "from semantica.export import JSONExporter, RDFExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "web_ingestor = WebIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "file_ingestor = FileIngestor()\n",
    "\n",
    "json_parser = JSONParser()\n",
    "xml_parser = XMLParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "\n",
    "# Real CVE and vulnerability data sources\n",
    "cve_sources = [\n",
    "    \"https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.json.zip\",  # NVD Recent CVEs (JSON)\n",
    "    \"https://nvd.nist.gov/feeds/xml/cve/2.0/nvdcve-2.0-recent.xml.zip\",  # NVD Recent CVEs (XML)\n",
    "    \"https://cve.mitre.org/data/downloads/allitems.csv\",  # CVE MITRE All Items\n",
    "    \"https://www.cisa.gov/known-exploited-vulnerabilities-catalog/json\"  # CISA KEV Catalog\n",
    "]\n",
    "\n",
    "# Real vulnerability feed URLs\n",
    "vulnerability_feeds = [\n",
    "    \"https://www.cisa.gov/news.xml\",  # CISA Security Advisories\n",
    "    \"https://www.us-cert.gov/ncas/alerts.xml\",  # US-CERT Alerts\n",
    "    \"https://feeds.feedburner.com/SecurityWeek\",  # Security Week\n",
    "    \"https://www.darkreading.com/rss.xml\"  # Dark Reading\n",
    "]\n",
    "\n",
    "# Real database connection for vulnerability tracking\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/vulnerability_db\"\n",
    "db_query = \"SELECT cve_id, description, severity, published_date, affected_products FROM vulnerabilities WHERE published_date > NOW() - INTERVAL '30 days' ORDER BY published_date DESC\"\n",
    "\n",
    "# Real web API endpoints for CVE data\n",
    "cve_apis = [\n",
    "    \"https://services.nvd.nist.gov/rest/json/cves/2.0\",  # NVD CVE API v2.0\n",
    "    \"https://api.github.com/repos/CVEProject/cvelist\",  # CVE Project on GitHub\n",
    "    \"https://cve.circl.lu/api/last\"  # CVE Search API\n",
    "]\n",
    "\n",
    "# Ingest from real CVE feeds\n",
    "cve_feed_list = []\n",
    "for feed_url in vulnerability_feeds:\n",
    "    cve_feed = feed_ingestor.ingest_feed(feed_url)\n",
    "    if cve_feed:\n",
    "        cve_feed_list.append(cve_feed)\n",
    "        print(f\"  Ingested feed: {feed_url}\")\n",
    "        print(f\"  Items: {len(cve_feed.items) if hasattr(cve_feed, 'items') else 0}\")\n",
    "\n",
    "# Ingest from real CVE APIs\n",
    "cve_api_data = []\n",
    "for api_url in cve_apis[:1]:  # Process first API\n",
    "    api_content = web_ingestor.ingest_url(api_url)\n",
    "    if api_content:\n",
    "        cve_api_data.append(api_content)\n",
    "        print(f\"  Ingested CVE API: {api_url}\")\n",
    "\n",
    "# Database ingestion pattern\n",
    "db_data = db_ingestor.export_table(\n",
    "    connection_string=db_connection_string,\n",
    "    table_name=\"vulnerabilities\",\n",
    "    limit=1000\n",
    ")\n",
    "print(f\"  Query pattern: {db_query}\")\n",
    "\n",
    "print(f\"\\n📊 CVE Ingestion Summary:\")\n",
    "print(f\"  Vulnerability feeds: {len(cve_feed_list)}\")\n",
    "print(f\"  CVE API sources: {len(cve_api_data)}\")\n",
    "print(f\"  Database sources: 1\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
