{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)\n",
                "\n",
                "# Mastering Ontology Generation with Semantica\n",
                "\n",
                "Welcome to the comprehensive guide on Semantica's Ontology Module. This module is the powerhouse for structuring your data into meaningful knowledge graphs, providing a complete 6-stage pipeline from raw data to validated OWL ontologies.\n",
                "\n",
                "In this notebook, we will dive deep into:\n",
                "1.  **The 6-Stage Generation Pipeline**: Understanding how Semantica transforms data into knowledge.\n",
                "2.  **Core Components in Focus**: Detailed usage of `ClassInferrer`, `PropertyGenerator`, and `OntologyOptimizer`.\n",
                "3.  **Validation & Quality**: ensuring your ontology is consistent and structurally sound.\n",
                "4.  **Visualize**: exploring your ontology with interactive charts and hierarchies.\n",
                "5.  **Advanced Usage**: Text-to-Ontology (LLM), Competency Questions, and Lifecycle Management.\n",
                "6.  **Exporting & Interoperability**: Saving your work in standard formats like Turtle and RDF/XML.\n",
                "\n",
                "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/ontology/)\n",
                "\n",
                "## Getting Started\n",
                "\n",
                "First, let's setup our environment and initialize the `OntologyEngine`. This engine is the unified entry point for all ontology operations."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Install semantica if not already installed\n",
                "# !pip install semantica\n",
                "# !pip install plotly # Required for visualization\n",
                "\n",
                "from semantica.ontology import OntologyEngine, OntologyGenerator\n",
                "from semantica.utils.logging import get_logger\n",
                "\n",
                "# Initialize logger for visibility\n",
                "logger = get_logger(\"ontology_guide\")\n",
                "\n",
                "# Initialize the Engine\n",
                "# base_uri defines the namespace root for your ontology\n",
                "engine = OntologyEngine(base_uri=\"https://docs.semantica.dev/ontology/\")\n",
                "\n",
                "print(\"Ontology Engine initialized successfully!\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "---\n",
                "\n",
                "## The 6-Stage Generation Pipeline\n",
                "\n",
                "Semantica uses a sophisticated 6-stage pipeline to robustly generate ontologies. This automated process takes raw entity and relationship data and produces a high-quality OWL ontology.\n",
                "\n",
                "### The Stages:\n",
                "1.  **Semantic Network Parsing**: Extracts raw concepts and connections from your inputs.\n",
                "2.  **YAML-to-Definition**: Transforms concepts into structured class definitions.\n",
                "3.  **Definition-to-Types**: Maps definitions to formal OWL types (e.g., `owl:Class`, `owl:ObjectProperty`).\n",
                "4.  **Hierarchy Generation**: Builds a taxonomic structure (parent-child relationships) using `associatedWith` or linguistic patterns.\n",
                "5.  **TTL Generation**: Serializes the in-memory structure into Turtle format logic.\n",
                "6.  **Symbolic Validation**: Validates the result using reasoners like HermiT (if available) or structural checks.\n",
                "\n",
                "Let's see this in action with some sample data."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Sample Data: A simple corporate structure\n",
                "entities = [\n",
                "    {\"id\": \"e1\", \"type\": \"Company\", \"name\": \"TechCorp\", \"founded\": \"2010\"},\n",
                "    {\"id\": \"e2\", \"type\": \"Person\", \"name\": \"Alice\", \"role\": \"CEO\"},\n",
                "    {\"id\": \"e3\", \"type\": \"Person\", \"name\": \"Bob\", \"role\": \"CTO\"},\n",
                "    {\"id\": \"e4\", \"type\": \"Department\", \"name\": \"Engineering\"},\n",
                "    {\"id\": \"e5\", \"type\": \"Project\", \"name\": \"Project Phoenix\"}\n",
                "]\n",
                "\n",
                "relationships = [\n",
                "    {\"source\": \"e2\", \"target\": \"e1\", \"type\": \"leads\"},\n",
                "    {\"source\": \"e3\", \"target\": \"e4\", \"type\": \"manages\"},\n",
                "    {\"source\": \"e4\", \"target\": \"e1\", \"type\": \"part_of\"},\n",
                "    {\"source\": \"e3\", \"target\": \"e5\", \"type\": \"works_on\"}\n",
                "]\n",
                "\n",
                "data = {\n",
                "    \"entities\": entities,\n",
                "    \"relationships\": relationships\n",
                "}\n",
                "\n",
                "# Run the full pipeline\n",
                "ontology = engine.from_data(data, name=\"CorporateOntology\")\n",
                "\n",
                "print(f\"Generated Ontology: {ontology['name']}\")\n",
                "print(f\"Classes Found: {len(ontology['classes'])}\")\n",
                "print(f\"Properties Found: {len(ontology['properties'])}\")\n",
                "print(f\"Validation Status: Valid={ontology.get('validation_result', {}).get('valid', 'Unknown')}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Inspecting the Results\n",
                "\n",
                "The generated `ontology` object is a rich dictionary containing all the inferred structure. Let's peek inside to see what Classes and Properties were created."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Inspect Classes\n",
                "print(\"--- Inferred Classes ---\")\n",
                "for cls in ontology['classes']:\n",
                "    print(f\"Class: {cls['name']}\")\n",
                "    print(f\"  URI: {cls.get('uri')}\")\n",
                "    # Check if a hierarchy was inferred\n",
                "    if cls.get('subClassOf'):\n",
                "        print(f\"  Parent: {cls['subClassOf']}\")\n",
                "    print(\"\")\n",
                "\n",
                "# Inspect Properties\n",
                "print(\"--- Inferred Properties ---\")\n",
                "for prop in ontology['properties']:\n",
                "    type_label = \"Object Property\" if prop['type'] == 'object' else \"Data Property\"\n",
                "    print(f\"{prop['name']} [{type_label}]\")\n",
                "    print(f\"  Domain: {prop.get('domain')}\")\n",
                "    print(f\"  Range: {prop.get('range')}\")\n",
                "    print(\"\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "---\n",
                "\n",
                "## Deep Dive: Component by Component\n",
                "\n",
                "While `OntologyEngine` is great for one-shot generation, you often need fine-grained control. Let's look at the individual tools that power the engine.\n",
                "\n",
                "### 1. `ClassInferrer`: Mastering Class Discovery\n",
                "\n",
                "The `ClassInferrer` analyzes entities to find patterns. It can handle noise and only creates classes for types that appear frequently enough.\n",
                "\n",
                "*   **`min_occurrences`**: Ignores types with fewer entities than this count.\n",
                "*   **`build_class_hierarchy`**: Toggles automatic parent-child detection.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ontology import ClassInferrer\n",
                "\n",
                "# Initialize inferrer with a threshold\n",
                "# We set min_occurrences=1 here to capture everything in our small example\n",
                "inferrer = ClassInferrer(min_occurrences=1)\n",
                "\n",
                "raw_entities = [\n",
                "    {\"type\": \"Manager\", \"name\": \"Dave\", \"level\": 5},\n",
                "    {\"type\": \"Manager\", \"name\": \"Eve\", \"level\": 4},\n",
                "    {\"type\": \"Employee\", \"name\": \"Frank\"}, # Only 1 employee\n",
                "    {\"type\": \"TemporaryWorker\", \"name\": \"Grace\"} \n",
                "]\n",
                "\n",
                "# Infer classes\n",
                "classes = inferrer.infer_classes(raw_entities, build_hierarchy=True)\n",
                "\n",
                "print(f\"Inferred {len(classes)} classes from raw entities.\")\n",
                "for c in classes:\n",
                "    print(f\"- {c['name']} (Count: {c['entity_count']})\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 2. `PropertyGenerator`: The Glue of the Ontology\n",
                "\n",
                "Properties define relationships. Semantica distinguishes between:\n",
                "*   **Object Properties**: Links between two entities (e.g., `leads` between Person and Company).\n",
                "*   **Data Properties**: Attributes of an entity (e.g., `founded` year of a Company).\n",
                "\n",
                "The `PropertyGenerator` automatically detects this distinction."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ontology import PropertyGenerator\n",
                "\n",
                "prop_gen = PropertyGenerator()\n",
                "\n",
                "# We need the classes first to help property generation context\n",
                "context_classes = classes  # reusing from previous step\n",
                "\n",
                "# Let's define some relationships and attributes implicitly via entities\n",
                "# Note: 'level' in Manager entities is a potential data property\n",
                "complex_entities = [\n",
                "    {\"id\": \"m1\", \"type\": \"Manager\", \"name\": \"Dave\", \"level\": 5},\n",
                "    {\"id\": \"e1\", \"type\": \"Employee\", \"name\": \"Frank\"}\n",
                "]\n",
                "complex_relationships = [\n",
                "    {\"source\": \"m1\", \"target\": \"e1\", \"type\": \"supervises\"} # Object property\n",
                "]\n",
                "\n",
                "properties = prop_gen.infer_properties(\n",
                "    entities=complex_entities,\n",
                "    relationships=complex_relationships,\n",
                "    classes=context_classes\n",
                ")\n",
                "\n",
                "print(\"--- Property Types Identified ---\")\n",
                "for p in properties:\n",
                "    print(f\"Property: {p['name']}\")\n",
                "    print(f\"  Type: {p['type']}\")\n",
                "    print(f\"  Domain: {p['domain']} -> Range: {p['range']}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 3. `OntologyOptimizer`: Refining the Structure\n",
                "\n",
                "Before finalizing, it's good practice to optimize. The optimizer removes redundancies and improves coherence, such as ensuring all classes have proper labels and valid URIs."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ontology import OntologyOptimizer\n",
                "\n",
                "optimizer = OntologyOptimizer()\n",
                "\n",
                "# Let's pretend we have a messy ontology dict\n",
                "messy_ontology = {\n",
                "    \"classes\": [\n",
                "        {\"name\": \"Person\", \"uri\": \"...Person\"},\n",
                "        {\"name\": \"Person\", \"uri\": \"...Person\"} # Duplicate!\n",
                "    ],\n",
                "    \"properties\": []\n",
                "}\n",
                "\n",
                "clean_ontology = optimizer.optimize_ontology(messy_ontology, remove_redundancy=True)\n",
                "\n",
                "print(f\"Original Classes: {len(messy_ontology['classes'])}\")\n",
                "print(f\"Optimized Classes: {len(clean_ontology['classes'])}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "---\n",
                "\n",
                "## Validation and Quality Control\n",
                "\n",
                "Semantica includes a robust `OntologyValidator`. It checks for:\n",
                "1.  **Structure**: Missing fields, malformed URIs.\n",
                "2.  **Consistency**: Circular hierarchies, contradictory definitions.\n",
                "3.  **Metrics**: Depth of hierarchy, property usage.\n",
                "\n",
                "If you have `Owlready2` installed, it can even run a reasoner (HermiT or Pellet) to prove logical consistency."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ontology import OntologyValidator\n",
                "\n",
                "validator = OntologyValidator(\n",
                "    check_consistency=True,\n",
                "    check_satisfiability=True\n",
                ")\n",
                "\n",
                "# Validate our previously generated 'ontology'\n",
                "result = validator.validate_ontology(ontology)\n",
                "\n",
                "print(f\"Is Valid? {result.valid}\")\n",
                "print(f\"Is Consistent? {result.consistent}\")\n",
                "\n",
                "if result.errors:\n",
                "    print(\"Errors Found:\", result.errors)\n",
                "if result.warnings:\n",
                "    print(\"Warnings:\", result.warnings)\n",
                "    \n",
                "# Check Metrics\n",
                "print(\"Metrics:\", result.metrics)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "---\n",
                "\n",
                "## Visualization\n",
                "\n",
                "A picture is worth a thousand triples! The `OntologyVisualizer` lets you explore your ontology's structure interactively.\n",
                "\n",
                "We can visualize:\n",
                "*   **Class Hierarchies**: Tree diagrams of class inheritance.\n",
                "*   **Structure Networks**: The full graph of classes and properties.\n",
                "*   **Metrics Dashboards**: High-level stats at a glance."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.visualization import OntologyVisualizer\n",
                "\n",
                "viz = OntologyVisualizer()\n",
                "\n",
                "# 1. Interactive Class Hierarchy\n",
                "# Returns a Plotly figure you can interact with\n",
                "fig_hierarchy = viz.visualize_hierarchy(ontology, output=\"interactive\")\n",
                "if fig_hierarchy:\n",
                "    fig_hierarchy.show()\n",
                "\n",
                "# 2. Ontology Structure Network\n",
                "# See how classes and properties connect\n",
                "fig_structure = viz.visualize_structure(ontology, output=\"interactive\")\n",
                "if fig_structure:\n",
                "    fig_structure.show()\n",
                "\n",
                "# 3. Metrics Dashboard\n",
                "# View counts, depths, and statistics\n",
                "fig_metrics = viz.visualize_metrics(ontology, output=\"interactive\")\n",
                "if fig_metrics:\n",
                "    fig_metrics.show()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "---\n",
                "\n",
                "## Advanced Usage: Lifecycle & AI\n",
                "\n",
                "Enterprise ontologies are living artifacts. Semantica provides tools to manage their entire lifecycle and accelerate creation with AI.\n",
                "\n",
                "### 1. Text-to-Ontology (LLM Integration)\n",
                "\n",
                "Instead of manually creating entities, use the `LLMOntologyGenerator` to extract an ontology directly from text requirements or documents."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ontology import LLMOntologyGenerator\n",
                "\n",
                "try:\n",
                "    # Note: Requires an API key in your environment variables\n",
                "    llm_gen = LLMOntologyGenerator(provider=\"openai\", model=\"gpt-4\")\n",
                "\n",
                "    text_description = \"\"\"\n",
                "    A University has many Departments. Each Department offers several Courses.\n",
                "    Professors teach Courses and belong to a Department.\n",
                "    Students enroll in Courses.\n",
                "    \"\"\"\n",
                "\n",
                "    llm_ontology = llm_gen.generate_ontology_from_text(\n",
                "        text=text_description,\n",
                "        name=\"UniversityOntology\"\n",
                "    )\n",
                "\n",
                "    print(\"Generated Classes:\", [c['name'] for c in llm_ontology['classes']])\n",
                "except Exception:\n",
                "    print(\"Skipping LLM generation: No API key or provider configured in this environment.\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 2. Test-Driven Design (Competency Questions)\n",
                "\n",
                "Formalize your requirements as \"Competency Questions\" (CQs). The `CompetencyQuestionsManager` can check if your ontology contains the necessary terms to answer them."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ontology import CompetencyQuestionsManager\n",
                "\n",
                "cq_manager = CompetencyQuestionsManager()\n",
                "\n",
                "# Define what our ontology SHOULD answer\n",
                "cq_manager.add_question(\"Who leads TechCorp?\", category=\"organizational\")\n",
                "cq_manager.add_question(\"Which projects does Bob manage?\", category=\"operational\")\n",
                "\n",
                "# Validate our 'ontology' against these questions\n",
                "validation_results = cq_manager.validate_ontology(ontology)\n",
                "\n",
                "print(f\"Answerable Questions: {validation_results['answerable']} / {validation_results['total_questions']}\")\n",
                "for q in cq_manager.questions:\n",
                "    status = \"✅\" if q.answerable else \"❌\"\n",
                "    print(f\"{status} {q.question}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 3. Lifecycle Management (Versioning & Reuse)\n",
                "\n",
                "Manage iterations with `VersionManager` and import external standards like FOAF or Dublin Core with `ReuseManager`."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ontology import VersionManager, ReuseManager\n",
                "\n",
                "# --- Versioning ---\n",
                "v_manager = VersionManager(base_uri=\"https://example.org/ontology/\")\n",
                "v1 = v_manager.create_version(\"1.0\", ontology, changes=[\"Initial creation\"])\n",
                "print(f\"Created Version: {v1.version} at {v1.ontology_iri}\")\n",
                "\n",
                "# --- Reuse ---\n",
                "reuse_manager = ReuseManager()\n",
                "\n",
                "# Check if we can reuse FOAF\n",
                "foaf_info = reuse_manager.research_ontology(\"http://xmlns.com/foaf/0.1/\")\n",
                "if foaf_info:\n",
                "    print(f\"Found standard ontology: {foaf_info['name']}\")\n",
                "    # We could now import this into our ontology\n",
                "    ontology['imports'].append(foaf_info['uri'])"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "---\n",
                "\n",
                "## Exporting Your Ontology\n",
                "\n",
                "Once your ontology is built and validated, you'll want to save it. Semantica focuses on **Turtle (`.ttl`)** as the primary format, but supports others via `rdflib`.\n",
                "\n",
                "You can export to a string or directly to a file."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Get Turtle string representation\n",
                "ttl_output = engine.to_owl(ontology, format=\"turtle\")\n",
                "\n",
                "print(\"--- Turtle Preview (First 500 chars) ---\")\n",
                "print(ttl_output[:500])\n",
                "print(\"...\")\n",
                "\n",
                "# Save to file\n",
                "output_path = \"corporate_ontology.ttl\"\n",
                "engine.export_owl(ontology, path=output_path, format=\"turtle\")\n",
                "print(f\"Successfully saved ontology to {output_path}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Summary\n",
                "\n",
                "You have now mastered the essentials of Semantica's Ontology Module!\n",
                "\n",
                "*   **Automated Generation**: Used the 6-stage pipeline to go from raw data to a structured ontology.\n",
                "*   **Component Control**: Used `ClassInferrer` and `PropertyGenerator` for fine-tuned modeling.\n",
                "*   **Quality Assurance**: Validated your model against strict standards.\n",
                "*   **Visualization**: Explored the ontology structure interactively.\n",
                "*   **Advanced Lifecycle**: Used AI generation, competency questions, and versioning.\n",
                "*   **Export**: Serialized your knowledge graph for use in other semantic web tools.\n",
                "\n",
                "**Next Steps**:\n",
                "*   Try customizing the `NamespaceManager` to use your organization's URL.\n",
                "*   Explore `OntologyEvaluator` for deeper quality metrics.\n",
                "*   Feed the generated ontology into the **Knowledge Graph** module to start reasoning over your data!"
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.8.10"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 5
}