{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/healthcare/01_Clinical_Reports_Processing.ipynb)\n",
    "\n",
    "# Clinical Reports Processing Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete clinical reports processing pipeline: ingest clinical documents from multiple sources (EHR systems, HL7/FHIR APIs, medical databases), extract medical entities, build knowledge graph, store in triple store, and query patient data.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n",
    "- **Parsing**: DocumentParser, PDFParser, StructuredDataParser, CSVParser, MCPParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, CoreferenceResolver, TripleExtractor\n",
    "- **KG**: GraphBuilder, GraphValidator, EntityResolver, GraphAnalyzer\n",
    "- **Triple Store**: TripleStore, TripleManager, QueryEngine\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor, ValidationEngine\n",
    "- **Export**: JSONExporter, RDFExporter, OWLExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, OntologyVisualizer, TemporalVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Clinical Documents (Files, APIs, DB, MCP) → Parse → Extract Medical Entities → Build Medical KG → Store in Triple Store → Query Patient Data → Generate Reports → Visualize**\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Ingest Clinical Documents from Multiple Sources\n",
    "\n",
    "Ingest clinical documents from EHR systems, HL7/FHIR APIs, and medical databases.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, DBIngestor, StreamIngestor, WebIngestor, MCPIngestor, ingest_mcp\n",
    "from semantica.parse import DocumentParser, PDFParser, StructuredDataParser, CSVParser, MCPParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, CoreferenceResolver, TripleExtractor\n",
    "from semantica.kg import GraphBuilder, GraphValidator, EntityResolver, GraphAnalyzer\n",
    "from semantica.triple_store import TripleStore, TripleManager, QueryEngine\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    
    "from semantica.export import JSONExporter, RDFExporter, OWLExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, OntologyVisualizer, TemporalVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "stream_ingestor = StreamIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "mcp_ingestor = MCPIngestor()\n",
    "\n",
    "document_parser = DocumentParser()\n",
    "pdf_parser = PDFParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "csv_parser = CSVParser()\n",
    "mcp_parser = MCPParser()\n",
    "\n",
    "# Real healthcare data sources\n",
    "healthcare_apis = [\n",
    "    \"https://api.logicahealth.org/fhir/R4/Patient\",  # Logica Health FHIR API\n",
    "    \"https://hapi.fhir.org/baseR4/Patient\",  # HAPI FHIR Server\n",
    "    \"https://api.logicahealth.org/fhir/R4/Observation\"  # FHIR Observations\n",
    "]\n",
    "\n",
    "medical_feeds = [\n",
    "    \"https://www.cdc.gov/rss.xml\",  # CDC Health Alerts\n",
    "    \"https://www.who.int/rss-feeds/news-english.xml\"  # WHO News\n",
    "]\n",
    "\n",
    "# Real database connection for clinical records (HIPAA compliant)\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/clinical_records_db\"\n",
    "db_query = \"SELECT patient_id, visit_date, diagnosis, medication, procedure, doctor FROM clinical_visits WHERE visit_date > CURRENT_DATE - INTERVAL '1 year' ORDER BY visit_date DESC\"\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample clinical report data\n",
    "clinical_report_file = os.path.join(temp_dir, \"clinical_report.json\")\n",
    "clinical_data = {\n",
    "    \"patient_id\": \"P001\",\n",
    "    \"visit_date\": (datetime.now() - timedelta(days=30)).isoformat(),\n",
    "    \"diagnosis\": [\"Hypertension\", \"Type 2 Diabetes\"],\n",
    "    \"medications\": [\"Lisinopril 10mg\", \"Metformin 500mg\"],\n",
    "    \"procedures\": [\"Blood Pressure Check\", \"HbA1c Test\"],\n",
    "    \"doctor\": \"Dr. Smith\",\n",
    "    \"notes\": \"Patient shows improvement in blood pressure control. Continue current medications.\"\n",
    "}\n",
    "\n",
    "with open(clinical_report_file, 'w') as f:\n",
    "    json.dump(clinical_data, f, indent=2)\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(clinical_report_file, read_content=True)\n",
    "parsed_data = structured_parser.parse_data(clinical_report_file, data_format=\"json\")\n",
    "\n",
    "# Ingest from FHIR APIs\n",
    "fhir_content_list = []\n",
    "for api_url in healthcare_apis[:1]:\n",
    "    api_content = web_ingestor.ingest_url(api_url)\n",
    "    if api_content:\n",
    "        fhir_content_list.append(api_content)\n",
    "        print(f\"  Ingested FHIR API: {api_url}\")\n",
    "\n",
    "# Optional: Ingest from MCP server\n",
    "# Users can bring their own medical database MCP server via URL\n",
    "mcp_clinical_data = []\n",
    "# Connect to medical database MCP server via URL\n",
    "# Example: http://localhost:8000/mcp or https://api.example.com/medical-mcp\n",
    "medical_mcp_url = \"http://localhost:8000/mcp\"  # Replace with your MCP server URL\n",
    "\n",
    "mcp_ingestor.connect(\n",
    "    \"clinical_mcp_server\",\n",
    "    url=medical_mcp_url,\n",
    "    headers={\n",
    "        \"Authorization\": \"Bearer your_token\",\n",
    "        \"X-API-Key\": \"your_api_key\"\n",
    "    } if \"api.example.com\" in medical_mcp_url else {}\n",
    ")\n",
    "\n",
    "# Ingest patient records from MCP server\n",
    "mcp_data = mcp_ingestor.ingest_resources(\n",
    "    \"clinical_mcp_server\",\n",
    "    resource_uris=[\"resource://patients/records\"]\n",
    ")\n",
    "mcp_clinical_data.extend(mcp_data)\n",
    "print(f\"  Ingested MCP resources: {len(mcp_data)}\")\n",
    "\n",
    "# Or use tool-based ingestion to query patient records\n",
    "tool_data = mcp_ingestor.ingest_tool_output(\n",
    "    \"clinical_mcp_server\",\n",
    "    tool_name=\"query_patient_records\",\n",
    "    arguments={\n",
    "        \"patient_id\": \"P001\",\n",
    "        \"date_range\": {\n",
    "            \"start\": (datetime.now() - timedelta(days=365)).isoformat(),\n",
    "            \"end\": datetime.now().isoformat()\n",
    "        }\n",
    "    }\n",
    ")\n",
    "if tool_data:\n",
    "    mcp_clinical_data.append(tool_data)\n",
    "    print(f\"  Retrieved tool data\")\n",
    "\n",
    "# Parse MCP responses and merge with existing clinical data\n",
    "for mcp_item in mcp_clinical_data:\n",
    "    parsed_mcp = mcp_parser.parse_response(mcp_item, response_type=\"json\")\n",
    "    if isinstance(parsed_mcp, dict):\n",
    "        if \"patient_records\" in parsed_mcp:\n",
    "            # Merge patient records from MCP\n",
    "            if isinstance(parsed_data.get(\"data\"), list):\n",
    "                parsed_data[\"data\"].extend(parsed_mcp.get(\"patient_records\", []))\n",
    "            elif isinstance(parsed_data.get(\"data\"), dict):\n",
    "                parsed_data[\"data\"] = [parsed_data.get(\"data\")] + parsed_mcp.get(\"patient_records\", [])\n",
    "            print(f\"  Parsed MCP item\")\n",
    "\n",
    "mcp_ingestor.disconnect(\"clinical_mcp_server\")\n",
    "print(f\"  Disconnected from MCP server\")\n",
    "\n",
    "print(f\"\\n📊 Ingestion Summary:\")\n",
    "print(f\"  Clinical reports: {len([file_objects]) if file_objects else 0}\")\n",
    "print(f\"  FHIR API sources: {len(fhir_content_list)}\")\n",
    "print(f\"  Database sources: 1\")\n",
    "print(f\"  MCP server sources: {len(mcp_clinical_data)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract Medical Entities\n",
    "\n",
    "Extract medical entities (conditions, medications, procedures, doctors) from clinical reports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "coreference_resolver = CoreferenceResolver()\n",
    "triple_extractor = TripleExtractor()\n",
    "\n",
    "medical_entities = []\n",
    "relationships = []\n",
    "\n",
    "# Extract from clinical data\n",
    "if parsed_data and parsed_data.data:\n",
    "    clinical = parsed_data.data if isinstance(parsed_data.data, dict) else parsed_data.data[0] if isinstance(parsed_data.data, list) else {}\n",
    "    \n",
    "    if isinstance(clinical, dict):\n",
    "        patient_id = clinical.get(\"patient_id\", \"\")\n",
    "        \n",
    "        medical_entities.append({\n",
    "            \"id\": patient_id,\n",
    "            \"type\": \"Patient\",\n",
    "            \"name\": patient_id,\n",
    "            \"properties\": {}\n",
    "        })\n",
    "        \n",
    "        # Diagnoses\n",
    "        for diagnosis in clinical.get(\"diagnosis\", []):\n",
    "            medical_entities.append({\n",
    "                \"id\": diagnosis,\n",
    "                \"type\": \"Diagnosis\",\n",
    "                \"name\": diagnosis,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            relationships.append({\n",
    "                \"source\": patient_id,\n",
    "                \"target\": diagnosis,\n",
    "                \"type\": \"has_diagnosis\",\n",
    "                \"properties\": {\"timestamp\": clinical.get(\"visit_date\", \"\")}\n",
    "            })\n",
    "        \n",
    "        # Medications\n",
    "        for medication in clinical.get(\"medications\", []):\n",
    "            medical_entities.append({\n",
    "                \"id\": medication,\n",
    "                \"type\": \"Medication\",\n",
    "                \"name\": medication,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            relationships.append({\n",
    "                \"source\": patient_id,\n",
    "                \"target\": medication,\n",
    "                \"type\": \"prescribed\",\n",
    "                \"properties\": {\"timestamp\": clinical.get(\"visit_date\", \"\")}\n",
    "            })\n",
    "        \n",
    "        # Procedures\n",
    "        for procedure in clinical.get(\"procedures\", []):\n",
    "            medical_entities.append({\n",
    "                \"id\": procedure,\n",
    "                \"type\": \"Procedure\",\n",
    "                \"name\": procedure,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            relationships.append({\n",
    "                \"source\": patient_id,\n",
    "                \"target\": procedure,\n",
    "                \"type\": \"underwent\",\n",
    "                \"properties\": {\"timestamp\": clinical.get(\"visit_date\", \"\")}\n",
    "            })\n",
    "        \n",
    "        # Doctor\n",
    "        doctor = clinical.get(\"doctor\", \"\")\n",
    "        if doctor:\n",
    "            medical_entities.append({\n",
    "                \"id\": doctor,\n",
    "                \"type\": \"Doctor\",\n",
    "                \"name\": doctor,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            relationships.append({\n",
    "                \"source\": doctor,\n",
    "                \"target\": patient_id,\n",
    "                \"type\": \"treats\",\n",
    "                \"properties\": {\"timestamp\": clinical.get(\"visit_date\", \"\")}\n",
    "            })\n",
    "\n",
    "print(f\"Extracted {len(medical_entities)} medical entities\")\n",
    "print(f\"Extracted {len(relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build Medical Knowledge Graph\n",
    "\n",
    "Build knowledge graph from medical entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "entity_resolver = EntityResolver()\n",
    "graph_validator = GraphValidator()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "\n",
    "resolved_entities = entity_resolver.resolve(medical_entities)\n",
    "\n",
    "medical_kg = builder.build(resolved_entities, relationships)\n",
    "\n",
    "validation_result = graph_validator.validate(medical_kg)\n",
    "metrics = graph_analyzer.compute_metrics(medical_kg)\n",
    "\n",
    "print(f\"Built medical knowledge graph\")\n",
    "print(f\"  Entities: {len(medical_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(medical_kg.get('relationships', []))}\")\n",
    "print(f\"  Graph valid: {validation_result.get('valid', False)}\")\n",
    "print(f\"  Graph density: {metrics.get('density', 0):.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Store in Triple Store and Query Patient Data\n",
    "\n",
    "Store knowledge graph in triple store and query patient information.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "triple_store = TripleStore()\n",
    "triple_manager = TripleManager()\n",
    "query_engine = QueryEngine()\n",
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "triple_store.store_knowledge_graph(medical_kg)\n",
    "\n",
    "# Query patient data\n",
    "patient_id = \"P001\"\n",
    "patient_query = f\"SELECT * WHERE {{ ?patient :hasDiagnosis ?diagnosis . ?patient :prescribed ?medication }}\"\n",
    "\n",
    "query_results = query_engine.query(patient_query, knowledge_graph=medical_kg)\n",
    "\n",
    "# Medical inference rules\n",
    "inference_engine.add_rule(\"IF patient has_diagnosis Hypertension AND patient prescribed Lisinopril THEN treatment_appropriate\")\n",
    "inference_engine.add_rule(\"IF patient has_diagnosis Diabetes AND patient prescribed Metformin THEN treatment_appropriate\")\n",
    "\n",
    "for relationship in relationships:\n",
    "    if relationship.get(\"type\") == \"has_diagnosis\":\n",
    "        inference_engine.add_fact({\n",
    "            \"patient\": relationship.get(\"source\"),\n",
    "            \"diagnosis\": relationship.get(\"target\")\n",
    "        })\n",
    "    if relationship.get(\"type\") == \"prescribed\":\n",
    "        inference_engine.add_fact({\n",
    "            \"patient\": relationship.get(\"source\"),\n",
    "            \"medication\": relationship.get(\"target\")\n",
    "        })\n",
    "\n",
    "medical_insights = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"Stored medical knowledge graph in triple store\")\n",
    "print(f\"Query returned {len(query_results) if query_results else 0} results\")\n",
    "print(f\"Generated {len(medical_insights)} medical insights\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Generate Reports and Visualize\n",
    "\n",
    "Generate clinical reports and visualize results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "owl_exporter = OWLExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(medical_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(medical_kg, os.path.join(temp_dir, \"clinical_kg.json\"))\n",
    "rdf_exporter.export_knowledge_graph(medical_kg, os.path.join(temp_dir, \"clinical_kg.rdf\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Clinical reports processing identified {len(medical_entities)} entities and {len(medical_insights)} insights\",\n",
    "    \"patients_processed\": len([e for e in medical_entities if e.get(\"type\") == \"Patient\"]),\n",
    "    \"diagnoses\": len([e for e in medical_entities if e.get(\"type\") == \"Diagnosis\"]),\n",
    "    \"medications\": len([e for e in medical_entities if e.get(\"type\") == \"Medication\"]),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "kg_visualizer = KGVisualizer()\n",
    "ontology_visualizer = OntologyVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(medical_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(medical_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Clinical Documents → Parse → Extract → Build KG → Triple Store → Query → Reports → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
