{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)\n",
    "\n",
    "#  Ontology Generation \n",
    "\n",
    "Welcome to the comprehensive guide on Semantica's Ontology Module. This module is the powerhouse for structuring your data into meaningful knowledge graphs, providing a complete 6-stage pipeline from raw data to validated OWL ontologies.\n",
    "\n",
    "In this notebook, we will dive deep into:\n",
    "1.  **The 6-Stage Generation Pipeline**: Understanding how Semantica transforms data into knowledge.\n",
    "2.  **Core Components in Focus**: Detailed usage of `ClassInferrer`, `PropertyGenerator`, and `OntologyOptimizer`.\n",
    "3.  **Visualize**: exploring your ontology with interactive charts and hierarchies.\n",
    "4.  **Advanced Usage**: Text-to-Ontology (LLM), Competency Questions, and Lifecycle Management.\n",
    "5.  **Exporting & Interoperability**: Saving your work in standard formats like Turtle and RDF/XML.\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/ontology/)\n",
    "\n",
    "## Getting Started\n",
    "\n",
    "First, let's setup our environment and initialize the `OntologyEngine`. This engine is the unified entry point for all ontology operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fcea810",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q semantica"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from semantica.ontology import OntologyEngine, OntologyGenerator\n",
    "from semantica.utils.logging import get_logger\n",
    "\n",
    "# Initialize logger for visibility\n",
    "logger = get_logger(\"ontology_guide\")\n",
    "\n",
    "# Initialize the Engine\n",
    "# base_uri defines the namespace root for your ontology\n",
    "engine = OntologyEngine(base_uri=\"https://docs.semantica.dev/ontology/\", min_occurrences=1)\n",
    "\n",
    "print(\"Ontology Engine initialized successfully!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## The 5-Stage Generation Pipeline\n",
    "\n",
    "Semantica uses a sophisticated 6-stage pipeline to robustly generate ontologies. This automated process takes raw entity and relationship data and produces a high-quality OWL ontology.\n",
    "\n",
    "### The Stages:\n",
    "1.  **Semantic Network Parsing**: Extracts raw concepts and connections from your inputs.\n",
    "2.  **YAML-to-Definition**: Transforms concepts into structured class definitions.\n",
    "3.  **Definition-to-Types**: Maps definitions to formal OWL types (e.g., `owl:Class`, `owl:ObjectProperty`).\n",
    "4.  **Hierarchy Generation**: Builds a taxonomic structure (parent-child relationships) using `associatedWith` or linguistic patterns.\n",
    "5.  **TTL Generation**: Serializes the in-memory structure into Turtle format logic.\n",
    "\n",
    "Let's see this in action with some sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample Data: A simple corporate structure\n",
    "entities = [\n",
    "    {\"id\": \"e1\", \"type\": \"Company\", \"name\": \"TechCorp\", \"founded\": \"2010\"},\n",
    "    {\"id\": \"e2\", \"type\": \"Person\", \"name\": \"Alice\", \"role\": \"CEO\"},\n",
    "    {\"id\": \"e3\", \"type\": \"Person\", \"name\": \"Bob\", \"role\": \"CTO\"},\n",
    "    {\"id\": \"e4\", \"type\": \"Department\", \"name\": \"Engineering\"},\n",
    "    {\"id\": \"e5\", \"type\": \"Project\", \"name\": \"Project Phoenix\"},\n",
    "    # Added entities to ensure robust class inference\n",
    "    {\"id\": \"e6\", \"type\": \"Company\", \"name\": \"TechSolutions\", \"founded\": \"2015\"},\n",
    "    {\"id\": \"e7\", \"type\": \"Person\", \"name\": \"Charlie\", \"role\": \"Lead\"},\n",
    "    {\"id\": \"e8\", \"type\": \"Person\", \"name\": \"Dave\", \"role\": \"Manager\"},\n",
    "    {\"id\": \"e9\", \"type\": \"Department\", \"name\": \"Research\"},\n",
    "    {\"id\": \"e10\", \"type\": \"Project\", \"name\": \"Project Orion\"}\n",
    "]\n",
    "\n",
    "relationships = [\n",
    "    {\"source\": \"e2\", \"target\": \"e1\", \"type\": \"leads\"},\n",
    "    {\"source\": \"e3\", \"target\": \"e4\", \"type\": \"manages\"},\n",
    "    {\"source\": \"e4\", \"target\": \"e1\", \"type\": \"part_of\"},\n",
    "    {\"source\": \"e3\", \"target\": \"e5\", \"type\": \"works_on\"},\n",
    "    # Additional relationships for new entities\n",
    "    {\"source\": \"e7\", \"target\": \"e6\", \"type\": \"leads\"},\n",
    "    {\"source\": \"e8\", \"target\": \"e9\", \"type\": \"manages\"},\n",
    "    {\"source\": \"e9\", \"target\": \"e6\", \"type\": \"part_of\"},\n",
    "    {\"source\": \"e7\", \"target\": \"e10\", \"type\": \"works_on\"}\n",
    "]\n",
    "\n",
    "data = {\n",
    "    \"entities\": entities,\n",
    "    \"relationships\": relationships\n",
    "}\n",
    "\n",
    "# Run the full pipeline\n",
    "ontology = engine.from_data(data, name=\"CorporateOntology\")\n",
    "\n",
    "print(f\"Generated Ontology: {ontology['name']}\")\n",
    "print(f\"Classes Found: {len(ontology['classes'])}\")\n",
    "print(f\"Properties Found: {len(ontology['properties'])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting the Results\n",
    "\n",
    "The generated `ontology` object is a rich dictionary containing all the inferred structure. Let's peek inside to see what Classes and Properties were created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect Classes\n",
    "print(\"--- Inferred Classes ---\")\n",
    "for cls in ontology['classes']:\n",
    "    print(f\"Class: {cls['name']}\")\n",
    "    print(f\"  URI: {cls.get('uri')}\")\n",
    "    # Check if a hierarchy was inferred\n",
    "    if cls.get('subClassOf'):\n",
    "        print(f\"  Parent: {cls['subClassOf']}\")\n",
    "    print(\"\")\n",
    "\n",
    "# Inspect Properties\n",
    "print(\"--- Inferred Properties ---\")\n",
    "for prop in ontology['properties']:\n",
    "    type_label = \"Object Property\" if prop['type'] == 'object' else \"Data Property\"\n",
    "    print(f\"{prop['name']} [{type_label}]\")\n",
    "    print(f\"  Domain: {prop.get('domain')}\")\n",
    "    print(f\"  Range: {prop.get('range')}\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Deep Dive: Component by Component\n",
    "\n",
    "While `OntologyEngine` is great for one-shot generation, you often need fine-grained control. Let's look at the individual tools that power the engine.\n",
    "\n",
    "### 1. `ClassInferrer`: Mastering Class Discovery\n",
    "\n",
    "The `ClassInferrer` analyzes entities to find patterns. It can handle noise and only creates classes for types that appear frequently enough.\n",
    "\n",
    "*   **`min_occurrences`**: Ignores types with fewer entities than this count.\n",
    "*   **`build_class_hierarchy`**: Toggles automatic parent-child detection.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ontology import ClassInferrer\n",
    "\n",
    "# Initialize inferrer with a threshold\n",
    "# We set min_occurrences=1 here to capture everything in our small example\n",
    "inferrer = ClassInferrer(min_occurrences=1)\n",
    "\n",
    "raw_entities = [\n",
    "    {\"type\": \"Manager\", \"name\": \"Dave\", \"level\": 5},\n",
    "    {\"type\": \"Manager\", \"name\": \"Eve\", \"level\": 4},\n",
    "    {\"type\": \"Employee\", \"name\": \"Frank\"}, # Only 1 employee\n",
    "    {\"type\": \"TemporaryWorker\", \"name\": \"Grace\"} \n",
    "]\n",
    "\n",
    "# Infer classes\n",
    "classes = inferrer.infer_classes(raw_entities, build_hierarchy=True)\n",
    "\n",
    "print(f\"Inferred {len(classes)} classes from raw entities.\")\n",
    "for c in classes:\n",
    "    print(f\"- {c['name']} (Count: {c['entity_count']})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. `PropertyGenerator`: The Glue of the Ontology\n",
    "\n",
    "Properties define relationships. Semantica distinguishes between:\n",
    "*   **Object Properties**: Links between two entities (e.g., `leads` between Person and Company).\n",
    "*   **Data Properties**: Attributes of an entity (e.g., `founded` year of a Company).\n",
    "\n",
    "The `PropertyGenerator` automatically detects this distinction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ontology import PropertyGenerator\n",
    "\n",
    "prop_gen = PropertyGenerator()\n",
    "\n",
    "# We need the classes first to help property generation context\n",
    "context_classes = classes  # reusing from previous step\n",
    "\n",
    "# Let's define some relationships and attributes implicitly via entities\n",
    "# Note: 'level' in Manager entities is a potential data property\n",
    "complex_entities = [\n",
    "    {\"id\": \"m1\", \"type\": \"Manager\", \"name\": \"Dave\", \"level\": 5},\n",
    "    {\"id\": \"e1\", \"type\": \"Employee\", \"name\": \"Frank\"}\n",
    "]\n",
    "complex_relationships = [\n",
    "    {\"source\": \"m1\", \"target\": \"e1\", \"type\": \"supervises\"} # Object property\n",
    "]\n",
    "\n",
    "properties = prop_gen.infer_properties(\n",
    "    entities=complex_entities,\n",
    "    relationships=complex_relationships,\n",
    "    classes=context_classes\n",
    ")\n",
    "\n",
    "print(\"--- Property Types Identified ---\")\n",
    "for p in properties:\n",
    "    print(f\"Property: {p['name']}\")\n",
    "    print(f\"  Type: {p['type']}\")\n",
    "    print(f\"  Domain: {p['domain']} -> Range: {p['range']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. `OntologyOptimizer`: Refining the Structure\n",
    "\n",
    "Before finalizing, it's good practice to optimize. The optimizer removes redundancies and improves coherence, such as ensuring all classes have proper labels and valid URIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ontology import OntologyOptimizer\n",
    "\n",
    "optimizer = OntologyOptimizer()\n",
    "\n",
    "# Let's pretend we have a messy ontology dict\n",
    "messy_ontology = {\n",
    "    \"classes\": [\n",
    "        {\"name\": \"Person\", \"uri\": \"...Person\"},\n",
    "        {\"name\": \"Person\", \"uri\": \"...Person\"} # Duplicate!\n",
    "    ],\n",
    "    \"properties\": []\n",
    "}\n",
    "\n",
    "clean_ontology = optimizer.optimize_ontology(messy_ontology, remove_redundancy=True)\n",
    "\n",
    "print(f\"Original Classes: {len(messy_ontology['classes'])}\")\n",
    "print(f\"Optimized Classes: {len(clean_ontology['classes'])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Visualization\n",
    "\n",
    "A picture is worth a thousand triplets! The `OntologyVisualizer` lets you explore your ontology's structure interactively.\n",
    "\n",
    "We can visualize:\n",
    "*   **Class Hierarchies**: Tree diagrams of class inheritance.\n",
    "*   **Structure Networks**: The full graph of classes and properties.\n",
    "*   **Metrics Dashboards**: High-level stats at a glance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.visualization import OntologyVisualizer\n",
    "\n",
    "viz = OntologyVisualizer()\n",
    "\n",
    "# 1. Interactive Class Hierarchy\n",
    "# Returns a Plotly figure you can interact with\n",
    "fig_hierarchy = viz.visualize_hierarchy(ontology, output=\"interactive\")\n",
    "if fig_hierarchy:\n",
    "    fig_hierarchy.show()\n",
    "\n",
    "# 2. Ontology Structure Network\n",
    "# See how classes and properties connect\n",
    "fig_structure = viz.visualize_structure(ontology, output=\"interactive\")\n",
    "if fig_structure:\n",
    "    fig_structure.show()\n",
    "\n",
    "# 3. Metrics Dashboard\n",
    "# View counts, depths, and statistics\n",
    "fig_metrics = viz.visualize_metrics(ontology, output=\"interactive\")\n",
    "if fig_metrics:\n",
    "    fig_metrics.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Advanced Usage: Lifecycle & AI\n",
    "\n",
    "Enterprise ontologies are living artifacts. Semantica provides tools to manage their entire lifecycle and accelerate creation with AI.\n",
    "\n",
    "### 1. Text-to-Ontology (LLM Integration)\n",
    "\n",
    "Instead of manually creating entities, use the `LLMOntologyGenerator` to extract an ontology directly from text requirements or documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ontology import LLMOntologyGenerator\n",
    "\n",
    "try:\n",
    "    # Note: Requires an API key in your environment variables\n",
    "    llm_gen = LLMOntologyGenerator(provider=\"openai\", model=\"gpt-4\")\n",
    "\n",
    "    text_description = \"\"\"\n",
    "    A University has many Departments. Each Department offers several Courses.\n",
    "    Professors teach Courses and belong to a Department.\n",
    "    Students enroll in Courses.\n",
    "    \"\"\"\n",
    "\n",
    "    llm_ontology = llm_gen.generate_ontology_from_text(\n",
    "        text=text_description,\n",
    "        name=\"UniversityOntology\"\n",
    "    )\n",
    "\n",
    "    print(\"Generated Classes:\", [c['name'] for c in llm_ontology['classes']])\n",
    "except Exception:\n",
    "    print(\"Skipping LLM generation: No API key or provider configured in this environment.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Test-Driven Design (Competency Questions)\n",
    "\n",
    "Formalize your requirements as \"Competency Questions\" (CQs). The `CompetencyQuestionsManager` can check if your ontology contains the necessary terms to answer them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ontology import CompetencyQuestionsManager\n",
    "\n",
    "cq_manager = CompetencyQuestionsManager()\n",
    "\n",
    "# Define what our ontology SHOULD answer\n",
    "cq_manager.add_question(\"Who leads TechCorp?\", category=\"organizational\")\n",
    "cq_manager.add_question(\"Which projects does Bob manage?\", category=\"operational\")\n",
    "\n",
    "# Validate our 'ontology' against these questions\n",
    "validation_results = cq_manager.validate_ontology(ontology)\n",
    "\n",
    "print(f\"Answerable Questions: {validation_results['answerable']} / {validation_results['total_questions']}\")\n",
    "for q in cq_manager.questions:\n",
    "    status = \"✅\" if q.answerable else \"❌\"\n",
    "    print(f\"{status} {q.question}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Lifecycle Management (Versioning & Reuse)\n",
    "\n",
    "Manage iterations with `VersionManager` and import external standards like FOAF or Dublin Core with `ReuseManager`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ontology import VersionManager, ReuseManager\n",
    "\n",
    "# --- Versioning ---\n",
    "v_manager = VersionManager(base_uri=\"https://example.org/ontology/\")\n",
    "v1 = v_manager.create_version(\"1.0\", ontology, changes=[\"Initial creation\"])\n",
    "print(f\"Created Version: {v1.version} at {v1.ontology_iri}\")\n",
    "\n",
    "# --- Reuse ---\n",
    "reuse_manager = ReuseManager()\n",
    "\n",
    "# Check if we can reuse FOAF\n",
    "foaf_info = reuse_manager.research_ontology(\"http://xmlns.com/foaf/0.1/\")\n",
    "if foaf_info:\n",
    "    print(f\"Found standard ontology: {foaf_info['name']}\")\n",
    "    # We could now import this into our ontology\n",
    "    ontology['imports'].append(foaf_info['uri'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Exporting Your Ontology\n",
    "\n",
    "Once your ontology is built and validated, you'll want to save it. Semantica focuses on **Turtle (`.ttl`)** as the primary format, but supports others via `rdflib`.\n",
    "\n",
    "You can export to a string or directly to a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Turtle string representation\n",
    "ttl_output = engine.to_owl(ontology, format=\"turtle\")\n",
    "\n",
    "print(\"--- Turtle Preview (First 500 chars) ---\")\n",
    "print(ttl_output[:500])\n",
    "print(\"...\")\n",
    "\n",
    "# Save to file\n",
    "output_path = \"corporate_ontology.ttl\"\n",
    "engine.export_owl(ontology, path=output_path, format=\"turtle\")\n",
    "print(f\"Successfully saved ontology to {output_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "You have now mastered the essentials of Semantica's Ontology Module!\n",
    "\n",
    "*   **Automated Generation**: Used the 6-stage pipeline to go from raw data to a structured ontology.\n",
    "*   **Component Control**: Used `ClassInferrer` and `PropertyGenerator` for fine-tuned modeling.\n",
    "*   **Visualization**: Explored the ontology structure interactively.\n",
    "*   **Advanced Lifecycle**: Used AI generation, competency questions, and versioning.\n",
    "*   **Export**: Serialized your knowledge graph for use in other semantic web tools.\n",
    "\n",
    "**Next Steps**:\n",
    "*   Try customizing the `NamespaceManager` to use your organization's URL.\n",
    "*   Explore `OntologyEvaluator` for deeper quality metrics.\n",
    "*   Feed the generated ontology into the **Knowledge Graph** module to start reasoning over your data!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
