Update cookbooks: remove version checks and ensure pip install

This commit is contained in:
KaifAhmad1
2025-12-13 23:03:29 +05:30
parent 6be868e067
commit ccb3f43104
69 changed files with 4410 additions and 4548 deletions
@@ -1,297 +1,286 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/12_Unstructured_to_Ontology.ipynb)\n",
"\n",
"# Advanced: Unstructured Text to Ontology\n",
"\n",
"Welcome to the advanced guide on extracting structured ontologies from unstructured text. This notebook explores two powerful paradigms available in Semantica:\n",
"\n",
"1. **Classical NLP Pipeline**: Using Named Entity Recognition (NER) and Relation Extraction.\n",
"2. **Generative AI Pipeline**: Using Large Language Models (LLMs) for direct conceptual modeling.\n",
"\n",
"We will compare both approaches, visualize the results, and validate the generated ontologies.\n",
"\n",
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/ontology/)\n",
"\n",
"## Setup and Installation\n",
"\n",
"Ensure you have Semantica installed with all dependencies."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install -U \"semantica[all]\"\n",
"import semantica\n",
"print(semantica.__version__)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# !pip install semantica[all]\n",
"\n",
"from semantica.utils.logging import get_logger\n",
"\n",
"logger = get_logger(\"unstructured_guide\")\n",
"print(\"Environment setup complete.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## The Input Text\n",
"\n",
"We will use a rich paragraph of text describing a technology company to test both extraction methods."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"text_corpus = \"\"\"\n",
"QuantumDynamics is a leading AI research lab founded by Dr. Elena Rostova in 2018. \n",
"The lab is headquartered in Zurich, Switzerland, and focuses on quantum computing algorithms. \n",
"Dr. Rostova serves as the Chief Scientist. \n",
"The lab has released products like the Q-1 Processor and the NeuralBridge SDK. \n",
"QuantumDynamics collaborates with major universities such as MIT and ETH Zurich.\n",
"\"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Approach 1: The Classical NLP Pipeline\n",
"\n",
"This approach builds the ontology from the bottom up:\n",
"1. **Extract Entities**: Identify nouns/proper nouns (e.g., \"QuantumDynamics\", \"Zurich\").\n",
"2. **Extract Relations**: Identify verbs connecting them (e.g., \"headquartered in\").\n",
"3. **Generate Ontology**: Map these triplets to Classes and Properties.\n",
"\n",
"**Pros**: Deterministic, traceable, works offline.\n",
"**Cons**: Dependent on the underlying NLP model's vocabulary and flexibility."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
"from semantica.ontology import OntologyGenerator, OntologyOptimizer\n",
"\n",
"# 1. Initialize Extractors\n",
"ner = NERExtractor()\n",
"re = RelationExtractor()\n",
"\n",
"# 2. Extract Entities\n",
"print(\"Extracting entities...\")\n",
"entities = ner.extract(text_corpus)\n",
"print(f\"Found {len(entities)} entities: {[e['text'] for e in entities]}\")\n",
"\n",
"# 3. Extract Relationships\n",
"print(\"Extracting relationships...\")\n",
"relationships = re.extract(text_corpus, entities)\n",
"for r in relationships:\n",
" print(f\" - {r['source']} -> {r['type']} -> {r['target']}\")\n",
"\n",
"# 4. Generate Structure\n",
"generator = OntologyGenerator()\n",
"nlp_ontology = generator.generate(entities, relationships, name=\"QuantumOntologyNLP\")\n",
"\n",
"# 5. Optimize (Clean up)\n",
"optimizer = OntologyOptimizer()\n",
"nlp_ontology = optimizer.optimize_ontology(nlp_ontology, remove_redundancy=True)\n",
"\n",
"print(f\"\\nGenerated NLP Ontology with {len(nlp_ontology['classes'])} classes and {len(nlp_ontology['properties'])} properties.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Approach 2: The Generative AI Pipeline (LLM)\n",
"\n",
"This approach uses a Large Language Model to \"read\" the text and directly propose a schema.\n",
"\n",
"**Pros**: Context-aware, can handle ambiguity, generates human-like class names.\n",
"**Cons**: Non-deterministic, requires API access.\n",
"\n",
"*Note: This step requires a configured LLM provider (e.g., OpenAI).* "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.ontology import LLMOntologyGenerator\n",
"\n",
"try:\n",
" # Initialize LLM Generator (ensure OPENAI_API_KEY is set in env)\n",
" llm_gen = LLMOntologyGenerator(provider=\"openai\", model=\"gpt-4\")\n",
" \n",
" print(\"Generating ontology with LLM...\")\n",
" llm_ontology = llm_gen.generate_ontology_from_text(\n",
" text=text_corpus,\n",
" name=\"QuantumOntologyLLM\"\n",
" )\n",
" \n",
" print(f\"Generated LLM Ontology with {len(llm_ontology['classes'])} classes and {len(llm_ontology['properties'])} properties.\")\n",
" print(\"Classes detected:\", [c['name'] for c in llm_ontology['classes']])\n",
" \n",
"except Exception as e:\n",
" print(f\"Skipping LLM generation: {e}\")\n",
" llm_ontology = None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Comparing Results with Visualization\n",
"\n",
"Let's visualize both ontologies side-by-side (if available) to see the difference in structure. The NLP model tends to be more literal, while the LLM model tends to be more conceptual."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.visualization import OntologyVisualizer\n",
"\n",
"visualizer = OntologyVisualizer()\n",
"\n",
"print(\"--- NLP Approach Visualization ---\")\n",
"fig_nlp = visualizer.visualize_structure(nlp_ontology, output=\"interactive\")\n",
"if fig_nlp: fig_nlp.show()\n",
"\n",
"if llm_ontology:\n",
" print(\"--- LLM Approach Visualization ---\")\n",
" fig_llm = visualizer.visualize_structure(llm_ontology, output=\"interactive\")\n",
" if fig_llm: fig_llm.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Validation\n",
"\n",
"No matter the method, validation is crucial. We check for structural integrity and logical consistency."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.ontology import OntologyValidator\n",
"\n",
"validator = OntologyValidator()\n",
"\n",
"def print_report(name, ont):\n",
" if not ont: return\n",
" res = validator.validate_ontology(ont)\n",
" print(f\"[{name}] Valid: {res.valid}, Errors: {len(res.errors)}\")\n",
" if res.metrics:\n",
" print(f\" Depth: {res.metrics.get('hierarchy_depth')}, Concepts: {res.metrics.get('class_count')}\")\n",
"\n",
"print_report(\"Classical NLP\", nlp_ontology)\n",
"print_report(\"Generative AI\", llm_ontology)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Export to OWL\n",
"\n",
"Finally, we choose the best model (or merge them using `ReuseManager`, covered in other guides) and export it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.export import OWLExporter\n",
"\n",
"exporter = OWLExporter()\n",
"\n",
"# Export the NLP ontology by default, or the LLM one if preferred\n",
"target_ontology = llm_ontology if llm_ontology else nlp_ontology\n",
"\n",
"output_file = \"quantum_ontology.ttl\"\n",
"exporter.export(target_ontology, output_file, format=\"turtle\")\n",
"print(f\"Successfully exported ontology to {output_file}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"You have learned to:\n",
"1. **Extract Ontologies Programmatically**: Using `NERExtractor` for reliable, data-driven modeling.\n",
"2. **Generate Ontologies with AI**: Using `LLMOntologyGenerator` for conceptual, high-level modeling.\n",
"3. **Visualize and Compare**: Using `OntologyVisualizer` to inspect the structural differences.\n",
"4. **Validate and Export**: Ensuring quality before saving to OWL standards."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/12_Unstructured_to_Ontology.ipynb)\n",
"\n",
"# Advanced: Unstructured Text to Ontology\n",
"\n",
"Welcome to the advanced guide on extracting structured ontologies from unstructured text. This notebook explores two powerful paradigms available in Semantica:\n",
"\n",
"1. **Classical NLP Pipeline**: Using Named Entity Recognition (NER) and Relation Extraction.\n",
"2. **Generative AI Pipeline**: Using Large Language Models (LLMs) for direct conceptual modeling.\n",
"\n",
"We will compare both approaches, visualize the results, and validate the generated ontologies.\n",
"\n",
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/ontology/)\n",
"\n",
"## Setup and Installation\n",
"\n",
"Ensure you have Semantica installed with all dependencies."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# !pip install semantica[all]\n",
"\n",
"from semantica.utils.logging import get_logger\n",
"\n",
"logger = get_logger(\"unstructured_guide\")\n",
"print(\"Environment setup complete.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## The Input Text\n",
"\n",
"We will use a rich paragraph of text describing a technology company to test both extraction methods."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"text_corpus = \"\"\"\n",
"QuantumDynamics is a leading AI research lab founded by Dr. Elena Rostova in 2018. \n",
"The lab is headquartered in Zurich, Switzerland, and focuses on quantum computing algorithms. \n",
"Dr. Rostova serves as the Chief Scientist. \n",
"The lab has released products like the Q-1 Processor and the NeuralBridge SDK. \n",
"QuantumDynamics collaborates with major universities such as MIT and ETH Zurich.\n",
"\"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Approach 1: The Classical NLP Pipeline\n",
"\n",
"This approach builds the ontology from the bottom up:\n",
"1. **Extract Entities**: Identify nouns/proper nouns (e.g., \"QuantumDynamics\", \"Zurich\").\n",
"2. **Extract Relations**: Identify verbs connecting them (e.g., \"headquartered in\").\n",
"3. **Generate Ontology**: Map these triplets to Classes and Properties.\n",
"\n",
"**Pros**: Deterministic, traceable, works offline.\n",
"**Cons**: Dependent on the underlying NLP model's vocabulary and flexibility."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
"from semantica.ontology import OntologyGenerator, OntologyOptimizer\n",
"\n",
"# 1. Initialize Extractors\n",
"ner = NERExtractor()\n",
"re = RelationExtractor()\n",
"\n",
"# 2. Extract Entities\n",
"print(\"Extracting entities...\")\n",
"entities = ner.extract(text_corpus)\n",
"print(f\"Found {len(entities)} entities: {[e['text'] for e in entities]}\")\n",
"\n",
"# 3. Extract Relationships\n",
"print(\"Extracting relationships...\")\n",
"relationships = re.extract(text_corpus, entities)\n",
"for r in relationships:\n",
" print(f\" - {r['source']} -> {r['type']} -> {r['target']}\")\n",
"\n",
"# 4. Generate Structure\n",
"generator = OntologyGenerator()\n",
"nlp_ontology = generator.generate(entities, relationships, name=\"QuantumOntologyNLP\")\n",
"\n",
"# 5. Optimize (Clean up)\n",
"optimizer = OntologyOptimizer()\n",
"nlp_ontology = optimizer.optimize_ontology(nlp_ontology, remove_redundancy=True)\n",
"\n",
"print(f\"\\nGenerated NLP Ontology with {len(nlp_ontology['classes'])} classes and {len(nlp_ontology['properties'])} properties.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Approach 2: The Generative AI Pipeline (LLM)\n",
"\n",
"This approach uses a Large Language Model to \"read\" the text and directly propose a schema.\n",
"\n",
"**Pros**: Context-aware, can handle ambiguity, generates human-like class names.\n",
"**Cons**: Non-deterministic, requires API access.\n",
"\n",
"*Note: This step requires a configured LLM provider (e.g., OpenAI).* "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.ontology import LLMOntologyGenerator\n",
"\n",
"try:\n",
" # Initialize LLM Generator (ensure OPENAI_API_KEY is set in env)\n",
" llm_gen = LLMOntologyGenerator(provider=\"openai\", model=\"gpt-4\")\n",
" \n",
" print(\"Generating ontology with LLM...\")\n",
" llm_ontology = llm_gen.generate_ontology_from_text(\n",
" text=text_corpus,\n",
" name=\"QuantumOntologyLLM\"\n",
" )\n",
" \n",
" print(f\"Generated LLM Ontology with {len(llm_ontology['classes'])} classes and {len(llm_ontology['properties'])} properties.\")\n",
" print(\"Classes detected:\", [c['name'] for c in llm_ontology['classes']])\n",
" \n",
"except Exception as e:\n",
" print(f\"Skipping LLM generation: {e}\")\n",
" llm_ontology = None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Comparing Results with Visualization\n",
"\n",
"Let's visualize both ontologies side-by-side (if available) to see the difference in structure. The NLP model tends to be more literal, while the LLM model tends to be more conceptual."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.visualization import OntologyVisualizer\n",
"\n",
"visualizer = OntologyVisualizer()\n",
"\n",
"print(\"--- NLP Approach Visualization ---\")\n",
"fig_nlp = visualizer.visualize_structure(nlp_ontology, output=\"interactive\")\n",
"if fig_nlp: fig_nlp.show()\n",
"\n",
"if llm_ontology:\n",
" print(\"--- LLM Approach Visualization ---\")\n",
" fig_llm = visualizer.visualize_structure(llm_ontology, output=\"interactive\")\n",
" if fig_llm: fig_llm.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Validation\n",
"\n",
"No matter the method, validation is crucial. We check for structural integrity and logical consistency."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.ontology import OntologyValidator\n",
"\n",
"validator = OntologyValidator()\n",
"\n",
"def print_report(name, ont):\n",
" if not ont: return\n",
" res = validator.validate_ontology(ont)\n",
" print(f\"[{name}] Valid: {res.valid}, Errors: {len(res.errors)}\")\n",
" if res.metrics:\n",
" print(f\" Depth: {res.metrics.get('hierarchy_depth')}, Concepts: {res.metrics.get('class_count')}\")\n",
"\n",
"print_report(\"Classical NLP\", nlp_ontology)\n",
"print_report(\"Generative AI\", llm_ontology)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Export to OWL\n",
"\n",
"Finally, we choose the best model (or merge them using `ReuseManager`, covered in other guides) and export it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.export import OWLExporter\n",
"\n",
"exporter = OWLExporter()\n",
"\n",
"# Export the NLP ontology by default, or the LLM one if preferred\n",
"target_ontology = llm_ontology if llm_ontology else nlp_ontology\n",
"\n",
"output_file = \"quantum_ontology.ttl\"\n",
"exporter.export(target_ontology, output_file, format=\"turtle\")\n",
"print(f\"Successfully exported ontology to {output_file}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"You have learned to:\n",
"1. **Extract Ontologies Programmatically**: Using `NERExtractor` for reliable, data-driven modeling.\n",
"2. **Generate Ontologies with AI**: Using `LLMOntologyGenerator` for conceptual, high-level modeling.\n",
"3. **Visualize and Compare**: Using `OntologyVisualizer` to inspect the structural differences.\n",
"4. **Validate and Export**: Ensuring quality before saving to OWL standards."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}