{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/healthcare/04_Healthcare_GraphRAG_Hybrid.ipynb)\n",
    "\n",
    "# Healthcare GraphRAG System with Semantica\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates building a **Retrieval-Augmented Generation (GraphRAG) system** using **Semantica as the core framework** that leverages heterogeneous healthcare resources through a combination of materialized knowledge graphs and virtually integrated data sources.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/concepts/)\n",
    "\n",
    "### Why Semantica?\n",
    "\n",
    "Semantica provides a comprehensive, unified framework for building GraphRAG systems in healthcare:\n",
    "\n",
    "- **Materialized Knowledge Graphs**: Semantica's KG modules enable building persistent knowledge graphs from medical ontologies, clinical documents, and reports\n",
    "- **Virtual Data Integration**: Semantica's DBIngestor and QueryEngine allow virtual integration with Electronic Health Records (EHRs) without data replication\n",
    "- **Hybrid Design**: Semantica's architecture naturally separates structural knowledge from patient-level data\n",
    "- **Dynamic Query Orchestration**: Semantica's Reasoning and Triple Store modules enable orchestration of queries across ontologies, documents, and EHRs\n",
    "- **Temporal & Semantic Dimensions**: Semantica's Temporal and Context modules provide historical analysis and semantic understanding\n",
    "- **Traceable & Explainable**: Semantica's ExplanationGenerator and ContextRetriever provide traceable, explainable answers\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- Materialized knowledge graphs from medical ontologies (SNOMED CT, ICD-10), clinical documents, and reports\n",
    "- Virtual integration with Electronic Health Records (EHRs) without data replication\n",
    "- Hybrid design separating structural knowledge from patient-level data\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "- Dynamic orchestration of queries across ontologies, documents, and EHRs using Semantica\n",
    "- Temporal and semantic dimensions via graph for historical analysis\n",
    "- Traceable, explainable, and historically contextualized answers\n",
    "\n",
    "### Semantica Modules Used (20+)\n",
    "\n",
    "- **Ingest**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, RepoIngestor, EmailIngestor, MCPIngestor (for EHR virtual connections and medical literature)\n",
    "- **Parse**: DocumentParser, PDFParser (clinical documents), StructuredDataParser (for structured medical data)\n",
    "- **Normalize**: TextNormalizer (for text normalization)\n",
    "- **Semantic Extract**: NERExtractor, RelationExtractor, TripleExtractor (medical entities and relationships)\n",
    "- **Ontology**: OntologyGenerator, OWLGenerator (medical ontologies like SNOMED CT, ICD-10)\n",
    "- **KG**: GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer (materialized knowledge graph)\n",
    "- **Embeddings**: EmbeddingGenerator, TextEmbedder (for embeddings)\n",
    "- **Vector Store**: VectorStore, HybridSearch, MetadataFilter (for RAG)\n",
    "- **Triple Store**: TripleManager, QueryEngine (for SPARQL queries on ontologies)\n",
    "- **Reasoning**: InferenceEngine, RuleManager (for query orchestration and medical reasoning)\n",
    "- **Context**: ContextRetriever, ContextGraphBuilder (for contextual retrieval)\n",
    "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer (for visualization)\n",
    "- **Export**: JSONExporter, RDFExporter, ReportGenerator\n",
    "- **Pipeline**: PipelineBuilder, ExecutionEngine (for orchestrating the complete pipeline)\n",
    "\n",
    "### Pipeline Overview\n",
    "\n",
    "**Medical Ontologies + Clinical Documents + EHRs (Virtual) → Parse → Extract Medical Entities → Build Materialized KG → Generate Embeddings → Vector Store → GraphRAG Setup → Query Orchestration → Generate Answers → Visualize & Export**\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Setup and Import Semantica Modules\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import all Semantica modules - using Semantica as the core framework\n",
    "from semantica.ingest import FileIngestor, DBIngestor, WebIngestor\n",
    "from semantica.parse import DocumentParser, PDFParser, StructuredDataParser\n",
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, TripleExtractor\n",
    "from semantica.ontology import OntologyGenerator, OWLGenerator\n",
    "from semantica.kg import GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer\n",
    "from semantica.embeddings import EmbeddingGenerator, TextEmbedder\n",
    "from semantica.vector_store import VectorStore, HybridSearch, MetadataFilter\n",
    "from semantica.triple_store import TripleManager, QueryEngine\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    "from semantica.context import ContextRetriever, ContextGraphBuilder\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "from semantica.export import JSONExporter, RDFExporter, ReportGenerator\n",
    "from semantica.pipeline import PipelineBuilder, ExecutionEngine\n",
    "\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "import numpy as np\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Ingest Medical Ontologies and Clinical Documents\n",
    "\n",
    "Using Semantica's ingest modules to load medical ontologies, clinical documents, and set up virtual connections to EHRs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica ingestors\n",
    "file_ingestor = FileIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "\n",
    "# Create temporary directory for sample data\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample medical ontology data (SNOMED CT concepts)\n",
    "snomed_data = {\n",
    "    \"concepts\": [\n",
    "        {\n",
    "            \"concept_id\": \"73211009\",\n",
    "            \"concept_name\": \"Diabetes mellitus\",\n",
    "            \"hierarchy\": \"Clinical finding\",\n",
    "            \"parent_concept\": \"Disorder of glucose metabolism\"\n",
    "        },\n",
    "        {\n",
    "            \"concept_id\": \"44054006\",\n",
    "            \"concept_name\": \"Type 2 diabetes mellitus\",\n",
    "            \"hierarchy\": \"Clinical finding\",\n",
    "            \"parent_concept\": \"Diabetes mellitus\"\n",
    "        },\n",
    "        {\n",
    "            \"concept_id\": \"46635009\",\n",
    "            \"concept_name\": \"Type 1 diabetes mellitus\",\n",
    "            \"hierarchy\": \"Clinical finding\",\n",
    "            \"parent_concept\": \"Diabetes mellitus\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "# Sample clinical document\n",
    "clinical_document = {\n",
    "    \"document_id\": \"DOC-001\",\n",
    "    \"patient_id\": \"PATIENT-001\",\n",
    "    \"document_type\": \"Clinical Note\",\n",
    "    \"content\": \"Patient presents with Type 2 diabetes mellitus. Current medications include Metformin 500mg twice daily. Blood glucose levels are well-controlled. Patient reports adherence to dietary recommendations.\",\n",
    "    \"date\": datetime.now().isoformat(),\n",
    "    \"physician\": \"Dr. Smith\"\n",
    "}\n",
    "\n",
    "# Save sample data\n",
    "snomed_file = os.path.join(temp_dir, \"snomed_concepts.json\")\n",
    "clinical_file = os.path.join(temp_dir, \"clinical_note.json\")\n",
    "\n",
    "with open(snomed_file, 'w') as f:\n",
    "    json.dump(snomed_data, f, indent=2)\n",
    "\n",
    "with open(clinical_file, 'w') as f:\n",
    "    json.dump(clinical_document, f, indent=2)\n",
    "\n",
    "# Ingest using Semantica FileIngestor\n",
    "snomed_file_obj = file_ingestor.ingest_file(snomed_file, read_content=True)\n",
    "clinical_file_obj = file_ingestor.ingest_file(clinical_file, read_content=True)\n",
    "\n",
    "# Virtual EHR connection setup (using Semantica DBIngestor)\n",
    "# In production, this would connect to actual EHR database\n",
    "ehr_connection_config = {\n",
    "    \"connection_string\": \"postgresql://user:password@localhost:5432/ehr_db\",\n",
    "    \"query\": \"SELECT patient_id, diagnosis, medications, lab_results, visit_date FROM patient_records WHERE patient_id = %s\"\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Parse and Normalize Data Using Semantica\n",
    "\n",
    "Using Semantica's parse and normalize modules to process the ingested data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica parsers and normalizer\n",
    "document_parser = DocumentParser()\n",
    "pdf_parser = PDFParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "text_normalizer = TextNormalizer()\n",
    "\n",
    "# Parse structured data (SNOMED CT)\n",
    "parsed_snomed = structured_parser.parse_json(snomed_file)\n",
    "snomed_concepts = parsed_snomed.data if hasattr(parsed_snomed, 'data') else parsed_snomed\n",
    "\n",
    "# Parse clinical document\n",
    "parsed_clinical = structured_parser.parse_json(clinical_file)\n",
    "clinical_data = parsed_clinical.data if hasattr(parsed_clinical, 'data') else parsed_clinical\n",
    "\n",
    "# Normalize clinical text using Semantica\n",
    "if isinstance(clinical_data, dict) and 'content' in clinical_data:\n",
    "    normalized_text = text_normalizer.normalize_text(clinical_data['content'])\n",
    "    clinical_data['normalized_content'] = normalized_text\n"
  ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Extract Medical Entities and Relationships Using Semantica\n",
    "\n",
    "Using Semantica's semantic extraction modules to extract medical entities, relationships, and triples from the parsed data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica extractors\n",
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "triple_extractor = TripleExtractor()\n",
    "\n",
    "# Extract entities from clinical document\n",
    "clinical_text = clinical_data.get('normalized_content', clinical_data.get('content', ''))\n",
    "extracted_entities = ner_extractor.extract(clinical_text)\n",
    "\n",
    "# Extract relationships\n",
    "extracted_relationships = relation_extractor.extract(clinical_text, entities=extracted_entities)\n",
    "\n",
    "# Extract triples\n",
    "extracted_triples = triple_extractor.extract(clinical_text)\n",
    "\n",
    "# Build entity list for knowledge graph\n",
    "medical_entities = []\n",
    "medical_relationships = []\n",
    "\n",
    "# Add SNOMED CT concepts as entities\n",
    "if isinstance(snomed_concepts, dict):\n",
    "    for concept in snomed_concepts.get('concepts', []):\n",
    "        medical_entities.append({\n",
    "            \"id\": f\"snomed_{concept.get('concept_id', '')}\",\n",
    "            \"type\": \"Medical_Concept\",\n",
    "            \"name\": concept.get('concept_name', ''),\n",
    "            \"properties\": {\n",
    "                \"concept_id\": concept.get('concept_id', ''),\n",
    "                \"hierarchy\": concept.get('hierarchy', ''),\n",
    "                \"source\": \"SNOMED_CT\"\n",
    "            }\n",
    "        })\n",
    "\n",
    "# Add extracted entities from clinical document\n",
    "for entity in extracted_entities:\n",
    "    medical_entities.append({\n",
    "        \"id\": f\"entity_{entity.get('id', len(medical_entities))}\",\n",
    "        \"type\": entity.get('type', 'Entity'),\n",
    "        \"name\": entity.get('text', ''),\n",
    "        \"properties\": {\n",
    "            \"source\": \"clinical_document\",\n",
    "            \"document_id\": clinical_data.get('document_id', '')\n",
    "        }\n",
    "    })\n",
    "\n",
    "# Add relationships\n",
    "for rel in extracted_relationships:\n",
    "    medical_relationships.append({\n",
    "        \"source\": rel.get('source', ''),\n",
    "        \"target\": rel.get('target', ''),\n",
    "        \"type\": rel.get('type', 'related_to'),\n",
    "        \"properties\": {\n",
    "            \"source\": \"clinical_document\"\n",
    "        }\n",
    "    })\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Build Materialized Knowledge Graph Using Semantica\n",
    "\n",
    "Using Semantica's KG modules to build a materialized knowledge graph from the extracted entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica KG builders and analyzers\n",
    "graph_builder = GraphBuilder()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "\n",
    "# Build materialized knowledge graph using Semantica\n",
    "materialized_kg = graph_builder.build(medical_entities, medical_relationships)\n",
    "\n",
    "# Analyze the graph using Semantica\n",
    "kg_metrics = graph_analyzer.compute_metrics(materialized_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(materialized_kg)\n",
    "\n",
    "print(f\"  - Entities: {len(materialized_kg.get('entities', []))}\")\n",
    "print(f\"  - Relationships: {len(materialized_kg.get('relationships', []))}\")\n",
    "print(f\"  - Graph density: {kg_metrics.get('density', 0):.4f}\")\n",
    "print(f\"  - Connected components: {connectivity.get('num_components', 0)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Generate Ontology Using Semantica\n",
    "\n",
    "Using Semantica's ontology modules to generate medical ontologies from the knowledge graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica ontology engine\n",
    "from semantica.ontology import OntologyEngine\n",
    "engine = OntologyEngine(base_uri=\"http://semantica.example.org/healthcare#\")\n",
    "\n",
    "# Generate ontology from knowledge graph entities/relationships\n",
    "ontology_result = engine.from_data({\n",
    "    \"entities\": medical_entities,\n",
    "    \"relationships\": medical_relationships\n",
    "}, name=\"HealthcareOntology\", entities=medical_entities, relationships=medical_relationships)\n",
    "\n",
    "# Generate OWL representation using Semantica\n",
    "owl_ontology = engine.to_owl(ontology_result, format=\"turtle\")\n",
    "\n",
    "print(f\"  - Classes: {len(ontology_result.get('classes', []))}\")\n",
    "print(f\"  - Properties: {len(ontology_result.get('properties', []))}\")\n",
    "print(f\"  - OWL generated: {len(owl_ontology) if owl_ontology else 0} characters\")\n",
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Generate Embeddings Using Semantica\n",
    "\n",
    "Using Semantica's embedding modules to generate vector embeddings for RAG.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica embedding generators\n",
    "embedding_generator = EmbeddingGenerator()\n",
    "text_embedder = TextEmbedder()\n",
    "\n",
    "# Generate embeddings for clinical documents using Semantica\n",
    "documents_for_embedding = [clinical_data.get('normalized_content', clinical_data.get('content', ''))]\n",
    "\n",
    "# Generate embeddings using Semantica\n",
    "document_embeddings = []\n",
    "for doc in documents_for_embedding:\n",
    "    embedding = text_embedder.embed(doc)\n",
    "    document_embeddings.append(embedding)\n",
    "\n",
    "# Generate embeddings for entities\n",
    "entity_embeddings = {}\n",
    "for entity in medical_entities[:10]:  # Limit for demo\n",
    "    entity_text = f\"{entity.get('name', '')} {entity.get('type', '')}\"\n",
    "    embedding = text_embedder.embed(entity_text)\n",
    "    entity_embeddings[entity.get('id', '')] = embedding\n",
    "\n",
    "print(f\"  - Document embeddings: {len(document_embeddings)}\")\n",
    "print(f\"  - Entity embeddings: {len(entity_embeddings)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Setup Vector Store and Hybrid Search Using Semantica\n",
    "\n",
    "Using Semantica's vector store modules to set up RAG with hybrid search capabilities.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica vector store and hybrid search\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=768)\n",
    "hybrid_search = HybridSearch()\n",
    "\n",
    "# Prepare metadata for documents\n",
    "document_metadata = [{\n",
    "    \"document_id\": clinical_data.get('document_id', ''),\n",
    "    \"document_type\": clinical_data.get('document_type', ''),\n",
    "    \"date\": clinical_data.get('date', ''),\n",
    "    \"source\": \"clinical_document\"\n",
    "}]\n",
    "\n",
    "# Store document embeddings using Semantica\n",
    "document_ids = vector_store.store_vectors(\n",
    "    vectors=document_embeddings,\n",
    "    metadata=document_metadata\n",
    ")\n",
    "\n",
    "# Store entity embeddings\n",
    "entity_metadata = [{\"entity_id\": eid, \"type\": \"entity\", \"source\": \"kg\"} for eid in entity_embeddings.keys()]\n",
    "entity_ids = vector_store.store_vectors(\n",
    "    vectors=list(entity_embeddings.values()),\n",
    "    metadata=entity_metadata\n",
    ")\n",
    "\n",
    "print(f\"  - Document vectors stored: {len(document_ids)}\")\n",
    "print(f\"  - Entity vectors stored: {len(entity_ids)}\")\n",
    "print(f\"  - Hybrid search ready for GraphRAG\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9: Setup Triple Store for Ontology Queries Using Semantica\n",
    "\n",
    "Using Semantica's triple store modules to enable SPARQL queries on medical ontologies.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica triple store and query engine\n",
    "triple_manager = TripleManager()\n",
    "query_engine = QueryEngine()\n",
    "\n",
    "# Register triple store (using in-memory for demo)\n",
    "store = triple_manager.register_store(\"healthcare_ontology\", \"jena\", \"http://localhost:3030/healthcare\")\n",
    "\n",
    "# Convert ontology to triples and add to store\n",
    "# In production, this would load the OWL ontology\n",
    "sample_triples = [\n",
    "    {\n",
    "        \"subject\": \"http://semantica.example.org/healthcare#Type2Diabetes\",\n",
    "        \"predicate\": \"http://www.w3.org/2000/01/rdf-schema#subClassOf\",\n",
    "        \"object\": \"http://semantica.example.org/healthcare#Diabetes\",\n",
    "        \"confidence\": 1.0\n",
    "    },\n",
    "    {\n",
    "        \"subject\": \"http://semantica.example.org/healthcare#Type1Diabetes\",\n",
    "        \"predicate\": \"http://www.w3.org/2000/01/rdf-schema#subClassOf\",\n",
    "        \"object\": \"http://semantica.example.org/healthcare#Diabetes\",\n",
    "        \"confidence\": 1.0\n",
    "    }\n",
    "]\n",
    "\n",
    "# Add triples using Semantica\n",
    "for triple in sample_triples:\n",
    "    triple_manager.add_triple(triple, store_id=\"healthcare_ontology\")\n",
    "\n",
    "print(f\"  - Triples added: {len(sample_triples)}\")\n",
    "print(f\"  - SPARQL queries enabled for ontology\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 10: Implement GraphRAG Query Orchestration Using Semantica\n",
    "\n",
    "Using Semantica's reasoning and context modules to orchestrate queries across ontologies, documents, and EHRs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica reasoning and context modules\n",
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "context_retriever = ContextRetriever()\n",
    "context_graph_builder = ContextGraphBuilder()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "# Define medical reasoning rules using Semantica\n",
    "medical_rules = [\n",
    "    {\n",
    "        \"rule_id\": \"diabetes_treatment_rule\",\n",
    "        \"condition\": \"IF patient has Type2Diabetes THEN recommend Metformin\",\n",
    "        \"action\": \"suggest_treatment\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Add rules using Semantica\n",
    "for rule in medical_rules:\n",
    "    rule_manager.add_rule(rule)\n",
    "\n",
    "# Example query orchestration function using Semantica\n",
    "def orchestrate_graphrag_query(query_text, patient_id=None):\n",
    "    \"\"\"\n",
    "    Orchestrate GraphRAG query across ontologies, documents, and EHRs using Semantica.\n",
    "    \"\"\"\n",
    "    results = {\n",
    "        \"ontology_results\": [],\n",
    "        \"document_results\": [],\n",
    "        \"ehr_results\": [],\n",
    "        \"graph_results\": [],\n",
    "        \"context\": {}\n",
    "    }\n",
    "    \n",
    "    # 1. Query ontology using Semantica Triple Store\n",
    "    sparql_query = f\"\"\"\n",
    "    SELECT ?concept WHERE {{\n",
    "        ?concept rdfs:label ?label .\n",
    "        FILTER(CONTAINS(LCASE(?label), \"{query_text.lower()}\"))\n",
    "    }}\n",
    "    \"\"\"\n",
    "    ontology_results = query_engine.execute_query(sparql_query, store)\n",
    "    results[\"ontology_results\"] = ontology_results.get(\"bindings\", [])\n",
    "    \n",
    "    # 2. Search documents using Semantica Hybrid Search\n",
    "    query_embedding = text_embedder.embed(query_text)\n",
    "    document_results = hybrid_search.search(\n",
    "        query_vector=query_embedding,\n",
    "        vectors=document_embeddings,\n",
    "        metadata=document_metadata,\n",
    "        vector_ids=document_ids,\n",
    "        k=5\n",
    "    )\n",
    "    results[\"document_results\"] = document_results\n",
    "    \n",
    "    # 3. Query knowledge graph using Semantica\n",
    "    graph_results = graph_analyzer.query_graph(materialized_kg, query_text)\n",
    "    results[\"graph_results\"] = graph_results\n",
    "    \n",
    "    # 4. Build context using Semantica\n",
    "    context_graph = context_graph_builder.build(\n",
    "        entities=medical_entities,\n",
    "        relationships=medical_relationships,\n",
    "        query=query_text\n",
    "    )\n",
    "    results[\"context\"] = context_graph\n",
    "    \n",
    "    # 5. Virtual EHR query (would execute in production)\n",
    "    if patient_id:\n",
    "        # In production: ehr_results = db_ingestor.query(ehr_connection_config, patient_id)\n",
    "        results[\"ehr_results\"] = {\"note\": \"EHR query would execute here in production\"}\n",
    "    \n",
    "    return results\n",
    "\n",
    "# Example query\n",
    "query = \"What are the treatment options for Type 2 diabetes?\"\n",
    "orchestrated_results = orchestrate_graphrag_query(query)\n",
    "\n",
    "print(f\"  - Query: '{query}'\")\n",
    "print(f\"  - Ontology results: {len(orchestrated_results['ontology_results'])}\")\n",
    "print(f\"  - Document results: {len(orchestrated_results['document_results'])}\")\n",
    "print(f\"  - Graph results: {len(orchestrated_results['graph_results'])}\")\n",
    "print(f\"  - Context built: {bool(orchestrated_results['context'])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate explanation using Semantica\n",
    "explanation = explanation_generator.generate(\n",
    "    query=query,\n",
    "    results=orchestrated_results,\n",
    "    knowledge_graph=materialized_kg,\n",
    "    context=orchestrated_results['context']\n",
    ")\n",
    "\n",
    "print(f\"  {explanation.get('answer', 'Answer generated')}\")\n",
    "print(f\"\\nSources:\")\n",
    "print(f\"  - Ontology: {len(orchestrated_results['ontology_results'])} results\")\n",
    "print(f\"  - Documents: {len(orchestrated_results['document_results'])} results\")\n",
    "print(f\"  - Knowledge Graph: {len(orchestrated_results['graph_results'])} results\")\n",
    "print(f\"  - Traceability: Enabled via Semantica ContextRetriever\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 12: Visualize Knowledge Graph and Results Using Semantica\n",
    "\n",
    "Using Semantica's visualization modules to visualize the knowledge graph and analysis results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica visualizers\n",
    "kg_visualizer = KGVisualizer(layout=\"force\", color_scheme=\"vibrant\")\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "\n",
    "# Visualize knowledge graph using Semantica\n",
    "kg_fig = kg_visualizer.visualize_network(\n",
    "    materialized_kg,\n",
    "    output=\"interactive\"\n",
    ")\n",
    "\n",
    "# Visualize analytics\n",
    "centrality = graph_analyzer.compute_centrality(materialized_kg, method=\"pagerank\")\n",
    "analytics_data = {\n",
    "    \"graph\": materialized_kg,\n",
    "    \"centrality\": centrality\n",
    "}\n",
    "analytics_fig = analytics_visualizer.visualize_centrality_rankings(\n",
    "    centrality,\n",
    "    centrality_type=\"pagerank\",\n",
    "    top_n=10,\n",
    "    output=\"interactive\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 13: Export Results Using Semantica\n",
    "\n",
    "Using Semantica's export modules to export the knowledge graph, ontology, and reports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Semantica exporters\n",
    "json_exporter = JSONExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "# Export knowledge graph as JSON using Semantica\n",
    "kg_json_file = os.path.join(temp_dir, \"healthcare_kg.json\")\n",
    "json_exporter.export(materialized_kg, kg_json_file)\n",
    "\n",
    "# Export ontology as RDF using Semantica\n",
    "ontology_rdf_file = os.path.join(temp_dir, \"healthcare_ontology.rdf\")\n",
    "rdf_exporter.export(ontology_result, ontology_rdf_file)\n",
    "\n",
    "# Generate comprehensive report using Semantica\n",
    "report_data = {\n",
    "    \"title\": \"Healthcare GraphRAG System Report\",\n",
    "    \"knowledge_graph_metrics\": kg_metrics,\n",
    "    \"query_results\": orchestrated_results,\n",
    "    \"explanation\": explanation\n",
    "}\n",
    "report_file = os.path.join(temp_dir, \"healthcare_graphrag_report.html\")\n",
    "report_generator.generate_report(report_data, report_file, format=\"html\")\n",
    "\n",
    "print(f\"  - Knowledge graph JSON: {kg_json_file}\")\n",
    "print(f\"  - Ontology RDF: {ontology_rdf_file}\")\n",
    "print(f\"  - Report HTML: {report_file}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 14: Complete Pipeline Orchestration Using Semantica\n",
    "\n",
    "Using Semantica's pipeline module to orchestrate the complete GraphRAG pipeline.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build complete pipeline using Semantica PipelineBuilder\n",
    "pipeline_builder = PipelineBuilder()\n",
    "\n",
    "# Define handlers using existing Semantica modules initialized earlier\n",
    "def ingest_handler(data, **config):\n",
    "    source_dir = config.get(\"source\", temp_dir)\n",
    "    files = []\n",
    "    try:\n",
    "        files = [os.path.join(source_dir, f) for f in os.listdir(source_dir)]\n",
    "    except Exception:\n",
    "        pass\n",
    "    if files:\n",
    "        file_obj = file_ingestor.ingest_file(files[0], read_content=True)\n",
    "        return {**data, \"file\": file_obj}\n",
    "    return data\n",
    "\n",
    "def parse_handler(data, **config):\n",
    "    file_obj = data.get(\"file\")\n",
    "    if file_obj and getattr(file_obj, \"path\", None):\n",
    "        parsed = document_parser.parse_document(file_obj.path)\n",
    "        text = parsed.get(\"text\") if isinstance(parsed, dict) else None\n",
    "        return {**data, \"text\": text or data.get(\"text\")}\n",
    "    return data\n",
    "\n",
    "def extract_handler(data, **config):\n",
    "    text = data.get(\"text\", \"\")\n",
    "    entities = ner_extractor.extract_entities(text) if hasattr(ner_extractor, 'extract_entities') else ner_extractor.extract(text)\n",
    "    entity_dicts = [{\"id\": f\"e{i}\", \"name\": getattr(e, 'text', str(e)), \"type\": getattr(e, 'label', getattr(e, 'type', 'Entity'))} for i, e in enumerate(entities)]\n",
    "    return {**data, \"entities\": entity_dicts}\n",
    "\n",
    "def build_kg_handler(data, **config):\n",
    "    entities = data.get(\"entities\", [])\n",
    "    kg = graph_builder.build({\"entities\": entities})\n",
    "    return {**data, \"knowledge_graph\": kg}\n",
    "healthcare_graphrag_pipeline = (\n",
    "    pipeline_builder\n",
    "    .add_step(\"ingest\", \"ingest\", handler=ingest_handler, source=temp_dir)\n",
    "    .add_step(\"parse\", \"parse\", dependencies=[\"ingest\"], handler=parse_handler)\n",
    "    .add_step(\"extract\", \"extract\", dependencies=[\"parse\"], handler=extract_handler)\n",
    "    .add_step(\"build_kg\", \"build_kg\", dependencies=[\"extract\"], handler=build_kg_handler)\n",
    "    .build()\n",
    "\n",
    "# Execute pipeline using Semantica ExecutionEngine\n",
    "execution_engine = ExecutionEngine()\n",
    "pipeline_result = execution_engine.execute_pipeline(healthcare_graphrag_pipeline, {\"files\": []})\n",
    "\n",
    "print(f\"  - Pipeline steps: {len(healthcare_graphrag_pipeline.steps)}\")\n",
    "print(f\"  - Execution status: {getattr(pipeline_result, 'success', True)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion and Best Practices\n",
    "\n",
    "### Key Takeaways\n",
    "\n",
    "1. **Semantica as Core Framework**: This notebook demonstrated using Semantica as the exclusive framework for building a Healthcare GraphRAG system\n",
    "2. **Materialized Knowledge Graphs**: Semantica's KG modules enable building persistent knowledge graphs from medical ontologies and documents\n",
    "3. **Virtual Data Integration**: Semantica's DBIngestor allows virtual integration with EHRs without data replication\n",
    "4. **Hybrid Search**: Semantica's HybridSearch combines vector similarity with knowledge graph queries\n",
    "5. **Query Orchestration**: Semantica's Reasoning and Triple Store modules enable dynamic query orchestration\n",
    "6. **Explainability**: Semantica's ExplanationGenerator provides traceable, explainable answers\n",
    "\n",
    "### Semantica-Specific Performance Considerations\n",
    "\n",
    "- **Vector Store**: Use Semantica's VectorStore with appropriate backend (FAISS for local, Pinecone/Weaviate for cloud)\n",
    "- **Graph Analytics**: Leverage Semantica's GraphAnalyzer for efficient centrality and community detection\n",
    "- **Pipeline Execution**: Use Semantica's ExecutionEngine for parallel execution of pipeline steps\n",
    "- **Caching**: Utilize Semantica's ContextRetriever caching for frequently accessed contexts\n",
    "\n",
    "### Deployment Recommendations Using Semantica\n",
    "\n",
    "1. **Production Setup**:\n",
    "   - Use Semantica's configuration management for environment-specific settings\n",
    "   - Leverage Semantica's Pipeline module for production workflows\n",
    "   - Use Semantica's export modules for data persistence\n",
    "\n",
    "2. **Scalability**:\n",
    "   - Use Semantica's batch processing capabilities for large-scale data ingestion\n",
    "   - Leverage Semantica's vector store adapters for distributed storage\n",
    "   - Utilize Semantica's parallel execution features\n",
    "\n",
    "3. **Compliance**:\n",
    "   - Semantica's virtual data integration ensures EHR data remains in original systems\n",
    "   - Use Semantica's audit logging for traceability\n",
    "   - Leverage Semantica's export modules for compliance reporting\n",
    "\n",
    "### How Semantica's Architecture Benefits Healthcare GraphRAG\n",
    "\n",
    "- **Unified Framework**: Single framework for all operations reduces integration complexity\n",
    "- **Modular Design**: Semantica's modular architecture allows flexible deployment\n",
    "- **Extensibility**: Semantica's registry system enables custom method registration\n",
    "- **Type Safety**: Semantica's structured data models ensure data consistency\n",
    "- **Performance**: Semantica's optimized algorithms provide efficient graph operations\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
