{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/healthcare/02_Disease_Network_Analysis.ipynb)\n",
    "\n",
    "# Disease Network Analysis Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete disease network analysis pipeline: ingest disease data from multiple sources (medical literature, research databases, clinical trials), extract disease relationships, build disease ontology, analyze networks, and predict outcomes.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n",
    "- **Parsing**: DocumentParser, PDFParser, StructuredDataParser, JSONParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, TripleExtractor, SemanticAnalyzer\n",
    "- **KG**: GraphBuilder, GraphAnalyzer, CentralityCalculator, CommunityDetector\n",
    "- **Analytics**: ConnectivityAnalyzer, TemporalGraphQuery, TemporalPatternDetector\n",
    "- **Ontology**: OntologyGenerator, ClassInferrer, PropertyGenerator, OntologyValidator\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor, ConflictDetector\n",
    "- **Export**: JSONExporter, RDFExporter, OWLExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, OntologyVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Disease Data Sources → Parse → Extract Disease Relationships → Build Disease Ontology → Analyze Networks → Predict Outcomes → Generate Reports → Visualize**\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Ingest Disease Data from Multiple Sources\n",
    "\n",
    "Ingest disease data from medical literature, research databases, and clinical trials.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, WebIngestor, DBIngestor, FeedIngestor\n",
    "from semantica.parse import DocumentParser, PDFParser, StructuredDataParser, JSONParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, TripleExtractor, SemanticAnalyzer\n",
    "from semantica.kg import GraphBuilder, GraphAnalyzer, CentralityCalculator, CommunityDetector\n",
    "from semantica.kg import ConnectivityAnalyzer, TemporalGraphQuery, TemporalPatternDetector\n",
    "from semantica.ontology import OntologyGenerator, ClassInferrer, PropertyGenerator, OntologyValidator\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    "from semantica.conflicts import ConflictDetector\n",
    "from semantica.export import JSONExporter, RDFExporter, OWLExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, OntologyVisualizer, AnalyticsVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "\n",
    "document_parser = DocumentParser()\n",
    "pdf_parser = PDFParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "json_parser = JSONParser()\n",
    "\n",
    "# Real disease data sources\n",
    "disease_apis = [\n",
    "    \"https://api.logicahealth.org/fhir/R4/Condition\",  # FHIR Conditions\n",
    "    \"https://hapi.fhir.org/baseR4/Condition\",  # HAPI FHIR Conditions\n",
    "    \"https://www.ncbi.nlm.nih.gov/books/NBK5197/\"  # NCBI Medical Literature\n",
    "]\n",
    "\n",
    "medical_feeds = [\n",
    "    \"https://www.cdc.gov/rss.xml\",  # CDC Health Alerts\n",
    "    \"https://www.who.int/rss-feeds/news-english.xml\"  # WHO News\n",
    "]\n",
    "\n",
    "# Real database connection for disease data\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/disease_db\"\n",
    "db_query = \"SELECT disease_name, icd10_code, related_diseases, symptoms, treatments FROM diseases WHERE last_updated > CURRENT_DATE - INTERVAL '1 year' ORDER BY disease_name\"\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample disease data\n",
    "disease_file = os.path.join(temp_dir, \"disease_data.json\")\n",
    "disease_data = {\n",
    "    \"diseases\": [\n",
    "        {\n",
    "            \"disease_name\": \"Type 2 Diabetes\",\n",
    "            \"icd10_code\": \"E11\",\n",
    "            \"related_diseases\": [\"Hypertension\", \"Cardiovascular Disease\", \"Obesity\"],\n",
    "            \"symptoms\": [\"Increased thirst\", \"Frequent urination\", \"Fatigue\"],\n",
    "            \"treatments\": [\"Metformin\", \"Insulin\", \"Lifestyle changes\"],\n",
    "            \"prevalence\": \"High\"\n",
    "        },\n",
    "        {\n",
    "            \"disease_name\": \"Hypertension\",\n",
    "            \"icd10_code\": \"I10\",\n",
    "            \"related_diseases\": [\"Type 2 Diabetes\", \"Cardiovascular Disease\", \"Kidney Disease\"],\n",
    "            \"symptoms\": [\"High blood pressure\", \"Headaches\", \"Dizziness\"],\n",
    "            \"treatments\": [\"ACE inhibitors\", \"Beta blockers\", \"Lifestyle changes\"],\n",
    "            \"prevalence\": \"Very High\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "with open(disease_file, 'w') as f:\n",
    "    json.dump(disease_data, f, indent=2)\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(disease_file, read_content=True)\n",
    "parsed_data = structured_parser.parse_json(disease_file)\n",
    "\n",
    "# Ingest from disease APIs\n",
    "disease_api_list = []\n",
    "for api_url in disease_apis[:1]:\n",
    "    api_content = web_ingestor.ingest_url(api_url)\n",
    "    if api_content:\n",
    "        disease_api_list.append(api_content)\n",
    "        print(f\"  Ingested disease API: {api_url}\")\n",
    "\n",
    "print(f\"\\n📊 Ingestion Summary:\")\n",
    "print(f\"  Disease data files: {len([file_objects]) if file_objects else 0}\")\n",
    "print(f\"  Disease API sources: {len(disease_api_list)}\")\n",
    "print(f\"  Database sources: 1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract Disease Relationships\n",
    "\n",
    "Extract disease entities and relationships from disease data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "triple_extractor = TripleExtractor()\n",
    "semantic_analyzer = SemanticAnalyzer()\n",
    "\n",
    "disease_entities = []\n",
    "disease_relationships = []\n",
    "\n",
    "# Extract from disease data\n",
    "if parsed_data and parsed_data.data:\n",
    "    diseases = parsed_data.data.get(\"diseases\", []) if isinstance(parsed_data.data, dict) else parsed_data.data if isinstance(parsed_data.data, list) else []\n",
    "    \n",
    "    for disease in diseases:\n",
    "        if isinstance(disease, dict):\n",
    "            disease_name = disease.get(\"disease_name\", \"\")\n",
    "            \n",
    "            disease_entities.append({\n",
    "                \"id\": disease_name,\n",
    "                \"type\": \"Disease\",\n",
    "                \"name\": disease_name,\n",
    "                \"properties\": {\n",
    "                    \"icd10_code\": disease.get(\"icd10_code\", \"\"),\n",
    "                    \"prevalence\": disease.get(\"prevalence\", \"\")\n",
    "                }\n",
    "            })\n",
    "            \n",
    "            # Related diseases\n",
    "            for related in disease.get(\"related_diseases\", []):\n",
    "                disease_entities.append({\n",
    "                    \"id\": related,\n",
    "                    \"type\": \"Disease\",\n",
    "                    \"name\": related,\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "                disease_relationships.append({\n",
    "                    \"source\": disease_name,\n",
    "                    \"target\": related,\n",
    "                    \"type\": \"related_to\",\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "            \n",
    "            # Symptoms\n",
    "            for symptom in disease.get(\"symptoms\", []):\n",
    "                disease_entities.append({\n",
    "                    \"id\": symptom,\n",
    "                    \"type\": \"Symptom\",\n",
    "                    \"name\": symptom,\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "                disease_relationships.append({\n",
    "                    \"source\": disease_name,\n",
    "                    \"target\": symptom,\n",
    "                    \"type\": \"has_symptom\",\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "            \n",
    "            # Treatments\n",
    "            for treatment in disease.get(\"treatments\", []):\n",
    "                disease_entities.append({\n",
    "                    \"id\": treatment,\n",
    "                    \"type\": \"Treatment\",\n",
    "                    \"name\": treatment,\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "                disease_relationships.append({\n",
    "                    \"source\": disease_name,\n",
    "                    \"target\": treatment,\n",
    "                    \"type\": \"treated_with\",\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "\n",
    "print(f\"Extracted {len(disease_entities)} disease entities\")\n",
    "print(f\"Extracted {len(disease_relationships)} disease relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build Disease Ontology\n",
    "\n",
    "Build disease ontology from extracted entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "ontology_generator = OntologyGenerator()\n",
    "class_inferrer = ClassInferrer()\n",
    "property_generator = PropertyGenerator()\n",
    "ontology_validator = OntologyValidator()\n",
    "\n",
    "disease_kg = builder.build(disease_entities, disease_relationships)\n",
    "\n",
    "disease_ontology = ontology_generator.generate_ontology({\n",
    "    \"entities\": disease_entities,\n",
    "    \"relationships\": disease_relationships\n",
    "})\n",
    "\n",
    "classes = class_inferrer.infer_classes(disease_entities)\n",
    "properties = property_generator.infer_properties(disease_entities, disease_relationships, classes)\n",
    "\n",
    "validation_result = ontology_validator.validate_ontology(disease_ontology)\n",
    "\n",
    "print(f\"Built disease knowledge graph\")\n",
    "print(f\"  Entities: {len(disease_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(disease_kg.get('relationships', []))}\")\n",
    "print(f\"Generated disease ontology\")\n",
    "print(f\"  Classes: {len(disease_ontology.get('classes', []))}\")\n",
    "print(f\"  Properties: {len(disease_ontology.get('properties', []))}\")\n",
    "print(f\"  Ontology valid: {validation_result.valid}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Analyze Disease Networks\n",
    "\n",
    "Analyze disease networks using graph analytics.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph_analyzer = GraphAnalyzer()\n",
    "centrality_calculator = CentralityCalculator()\n",
    "community_detector = CommunityDetector()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "temporal_query = TemporalGraphQuery()\n",
    "temporal_pattern_detector = TemporalPatternDetector()\n",
    "\n",
    "metrics = graph_analyzer.compute_metrics(disease_kg)\n",
    "centrality_result = centrality_calculator.calculate_degree_centrality(disease_kg)\n",
    "centrality_scores = centrality_result.get('centrality', {})\n",
    "communities = community_detector.detect_communities(disease_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(disease_kg)\n",
    "\n",
    "temporal_patterns = temporal_pattern_detector.detect_temporal_patterns(\n",
    "    disease_kg,\n",
    "    pattern_type=\"sequence\",\n",
    "    min_frequency=1\n",
    ")\n",
    "\n",
    "print(f\"Network analysis complete\")\n",
    "print(f\"  Graph density: {metrics.get('density', 0):.3f}\")\n",
    "print(f\"  Communities: {len(communities)}\")\n",
    "print(f\"  Central diseases: {len([e for e, score in centrality_scores.items() if score > 0])}\")\n",
    "print(f\"  Connected components: {len(connectivity.get('components', []))}\")\n",
    "print(f\"  Temporal patterns: {len(temporal_patterns)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Predict Disease Outcomes\n",
    "\n",
    "Predict disease progression and outcomes using inference.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "# Disease outcome prediction rules\n",
    "inference_engine.add_rule(\"IF disease related_to Hypertension AND disease related_to Diabetes THEN high_comorbidity_risk\")\n",
    "inference_engine.add_rule(\"IF disease has_symptom Fatigue AND disease prevalence is High THEN common_condition\")\n",
    "inference_engine.add_rule(\"IF disease treated_with Insulin AND disease is Type 2 Diabetes THEN advanced_stage\")\n",
    "\n",
    "# Add facts from disease data\n",
    "for disease in disease_entities:\n",
    "    if disease.get(\"type\") == \"Disease\":\n",
    "        inference_engine.add_fact({\n",
    "            \"disease\": disease.get(\"name\", \"\"),\n",
    "            \"prevalence\": disease.get(\"properties\", {}).get(\"prevalence\", \"\")\n",
    "        })\n",
    "\n",
    "for relationship in disease_relationships:\n",
    "    if relationship.get(\"type\") == \"related_to\":\n",
    "        inference_engine.add_fact({\n",
    "            \"disease1\": relationship.get(\"source\"),\n",
    "            \"disease2\": relationship.get(\"target\")\n",
    "        })\n",
    "\n",
    "outcome_predictions = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"Generated {len(outcome_predictions)} disease outcome predictions\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Generate Reports and Visualize\n",
    "\n",
    "Generate disease network analysis reports and visualize results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "owl_exporter = OWLExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(disease_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(disease_kg, os.path.join(temp_dir, \"disease_kg.json\"))\n",
    "rdf_exporter.export_knowledge_graph(disease_kg, os.path.join(temp_dir, \"disease_kg.rdf\"))\n",
    "owl_exporter.export(disease_ontology, os.path.join(temp_dir, \"disease_ontology.owl\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Disease network analysis identified {len(disease_entities)} entities and {len(outcome_predictions)} predictions\",\n",
    "    \"diseases_analyzed\": len([e for e in disease_entities if e.get(\"type\") == \"Disease\"]),\n",
    "    \"relationships\": len(disease_relationships),\n",
    "    \"predictions\": len(outcome_predictions),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "kg_visualizer = KGVisualizer()\n",
    "ontology_visualizer = OntologyVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(disease_kg, output=\"interactive\")\n",
    "ontology_viz = ontology_visualizer.visualize_hierarchy(disease_ontology, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(disease_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Disease Data → Parse → Extract → Build Ontology → Analyze Networks → Predict Outcomes → Reports → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
