mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
KaifAhmad1
7a6f1d0417
Add a Cite Us section to the README with BibTeX citation info, and align it with docs/citation.md (author/organization: Semantica, 2026). Update LICENSE and docs/project-license.md copyright holder to Semantica, and replace the stale Hawksight-AI GitHub org slug with semantica-agi across READMEs, plugin manifests, cookbook notebooks, and GitHub templates.
312 lines
11 KiB
Plaintext
312 lines
11 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/12_Unstructured_to_Ontology.ipynb)\n",
|
|
"\n",
|
|
"# Unstructured Text to Ontology\n",
|
|
"\n",
|
|
"Welcome to the advanced guide on extracting structured ontologies from unstructured text. This notebook explores two powerful paradigms available in Semantica:\n",
|
|
"\n",
|
|
"1. **Classical NLP Pipeline**: Using Named Entity Recognition (NER) and Relation Extraction.\n",
|
|
"2. **Generative AI Pipeline**: Using Large Language Models (LLMs) for direct conceptual modeling.\n",
|
|
"\n",
|
|
"We will compare both approaches, visualize the results, and validate the generated ontologies.\n",
|
|
"\n",
|
|
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/ontology/)\n",
|
|
"\n",
|
|
"## Setup and Installation\n",
|
|
"\n",
|
|
"Ensure you have Semantica installed with all dependencies."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9c21e116",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install -qU semantica"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"\n",
|
|
"from semantica.utils.logging import get_logger\n",
|
|
"\n",
|
|
"logger = get_logger(\"unstructured_guide\")\n",
|
|
"print(\"Environment setup complete.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## The Input Text\n",
|
|
"\n",
|
|
"We will use a rich paragraph of text describing a technology company to test both extraction methods."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"text_corpus = \"\"\"\n",
|
|
"QuantumDynamics is a leading AI research lab founded by Dr. Elena Rostova in 2018. \n",
|
|
"The lab is headquartered in Zurich, Switzerland, and focuses on quantum computing algorithms. \n",
|
|
"Dr. Rostova serves as the Chief Scientist. \n",
|
|
"The lab has released products like the Q-1 Processor and the NeuralBridge SDK. \n",
|
|
"QuantumDynamics collaborates with major universities such as MIT and ETH Zurich.\n",
|
|
"\"\"\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## Approach 1: The Classical NLP Pipeline\n",
|
|
"\n",
|
|
"This approach builds the ontology from the bottom up:\n",
|
|
"1. **Extract Entities**: Identify nouns/proper nouns (e.g., \"QuantumDynamics\", \"Zurich\").\n",
|
|
"2. **Extract Relations**: Identify verbs connecting them (e.g., \"headquartered in\").\n",
|
|
"3. **Generate Ontology**: Map these triplets to Classes and Properties.\n",
|
|
"\n",
|
|
"**Pros**: Deterministic, traceable, works offline.\n",
|
|
"**Cons**: Dependent on the underlying NLP model's vocabulary and flexibility."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "75f896b9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
|
|
"from semantica.ontology import OntologyGenerator, OntologyOptimizer\n",
|
|
"\n",
|
|
"# 1. Initialize Extractors\n",
|
|
"ner = NERExtractor()\n",
|
|
"re = RelationExtractor()\n",
|
|
"\n",
|
|
"# 2. Extract Entities\n",
|
|
"print(\"Extracting entities...\")\n",
|
|
"entities = ner.extract(text_corpus)\n",
|
|
"\n",
|
|
"# Note: entities are returned as Entity objects (dataclasses), not dictionaries.\n",
|
|
"# We access properties using dot notation (e.g., entity.text, entity.label).\n",
|
|
"print(f\"Found {len(entities)} entities.\")\n",
|
|
"for e in entities[:5]:\n",
|
|
" print(f\" - {e.text} ({e.label}) [Conf: {e.confidence}]\")\n",
|
|
"\n",
|
|
"# 3. Extract Relationships\n",
|
|
"print(\"\\nExtracting relationships...\")\n",
|
|
"relationships = re.extract(text_corpus, entities)\n",
|
|
"\n",
|
|
"# Note: relationships are returned as Relation objects.\n",
|
|
"print(f\"Found {len(relationships)} relationships.\")\n",
|
|
"for r in relationships:\n",
|
|
" print(f\" - {r.subject.text} -> {r.predicate} -> {r.object.text}\")\n",
|
|
"\n",
|
|
"# 4. Prepare Data for Ontology Generation\n",
|
|
"# The OntologyGenerator expects dictionaries, so we convert our objects.\n",
|
|
"# We also ensure we handle both object attributes and potential dictionary keys for robustness.\n",
|
|
"entities_data = []\n",
|
|
"for e in entities:\n",
|
|
" if hasattr(e, 'to_dict'):\n",
|
|
" entities_data.append(e.to_dict())\n",
|
|
" else:\n",
|
|
" # Manual conversion for dataclasses without to_dict\n",
|
|
" entities_data.append({\n",
|
|
" \"id\": getattr(e, \"text\", str(e)),\n",
|
|
" \"text\": getattr(e, \"text\", str(e)),\n",
|
|
" \"type\": getattr(e, \"label\", getattr(e, \"type\", \"Unknown\")),\n",
|
|
" \"confidence\": getattr(e, \"confidence\", 1.0)\n",
|
|
" })\n",
|
|
"\n",
|
|
"relationships_data = []\n",
|
|
"for r in relationships:\n",
|
|
" if hasattr(r, 'to_dict'):\n",
|
|
" relationships_data.append(r.to_dict())\n",
|
|
" else:\n",
|
|
" # Manual conversion for dataclasses without to_dict\n",
|
|
" # Handle nested Entity objects in subject/object fields\n",
|
|
" subj = r.subject\n",
|
|
" obj = r.object\n",
|
|
" subj_text = getattr(subj, \"text\", str(subj))\n",
|
|
" obj_text = getattr(obj, \"text\", str(obj))\n",
|
|
" \n",
|
|
" relationships_data.append({\n",
|
|
" \"source\": subj_text,\n",
|
|
" \"target\": obj_text,\n",
|
|
" \"type\": getattr(r, \"predicate\", getattr(r, \"type\", \"related_to\")),\n",
|
|
" \"confidence\": getattr(r, \"confidence\", 1.0)\n",
|
|
" })\n",
|
|
"\n",
|
|
"# 5. Generate Structure\n",
|
|
"generator = OntologyGenerator()\n",
|
|
"nlp_ontology = generator.generate_ontology(\n",
|
|
" {\"entities\": entities_data, \"relationships\": relationships_data},\n",
|
|
" name=\"QuantumOntologyNLP\"\n",
|
|
")\n",
|
|
"\n",
|
|
"# 6. Optimize (Clean up)\n",
|
|
"optimizer = OntologyOptimizer()\n",
|
|
"nlp_ontology = optimizer.optimize_ontology(nlp_ontology, remove_redundancy=True)\n",
|
|
"\n",
|
|
"print(f\"\\nGenerated NLP Ontology with {len(nlp_ontology['classes'])} classes and {len(nlp_ontology['properties'])} properties.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## Approach 2: The Generative AI Pipeline (LLM)\n",
|
|
"\n",
|
|
"This approach uses a Large Language Model to \"read\" the text and directly propose a schema.\n",
|
|
"\n",
|
|
"**Pros**: Context-aware, can handle ambiguity, generates human-like class names.\n",
|
|
"**Cons**: Non-deterministic, requires API access.\n",
|
|
"\n",
|
|
"*Note: This step requires a configured LLM provider (e.g., OpenAI).* "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.ontology import LLMOntologyGenerator\n",
|
|
"\n",
|
|
"try:\n",
|
|
" # Initialize LLM Generator (ensure OPENAI_API_KEY is set in env)\n",
|
|
" llm_gen = LLMOntologyGenerator(provider=\"openai\", model=\"gpt-4\")\n",
|
|
" \n",
|
|
" print(\"Generating ontology with LLM...\")\n",
|
|
" llm_ontology = llm_gen.generate_ontology_from_text(\n",
|
|
" text=text_corpus,\n",
|
|
" name=\"QuantumOntologyLLM\"\n",
|
|
" )\n",
|
|
" \n",
|
|
" print(f\"Generated LLM Ontology with {len(llm_ontology['classes'])} classes and {len(llm_ontology['properties'])} properties.\")\n",
|
|
" print(\"Classes detected:\", [c['name'] for c in llm_ontology['classes']])\n",
|
|
" \n",
|
|
"except Exception as e:\n",
|
|
" print(f\"Skipping LLM generation: {e}\")\n",
|
|
" llm_ontology = None"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## Comparing Results with Visualization\n",
|
|
"\n",
|
|
"Let's visualize both ontologies side-by-side (if available) to see the difference in structure. The NLP model tends to be more literal, while the LLM model tends to be more conceptual."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.visualization import OntologyVisualizer\n",
|
|
"\n",
|
|
"visualizer = OntologyVisualizer()\n",
|
|
"\n",
|
|
"print(\"--- NLP Approach Visualization ---\")\n",
|
|
"fig_nlp = visualizer.visualize_structure(nlp_ontology, output=\"interactive\")\n",
|
|
"if fig_nlp: fig_nlp.show()\n",
|
|
"\n",
|
|
"if llm_ontology:\n",
|
|
" print(\"--- LLM Approach Visualization ---\")\n",
|
|
" fig_llm = visualizer.visualize_structure(llm_ontology, output=\"interactive\")\n",
|
|
" if fig_llm: fig_llm.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## Export to OWL\n",
|
|
"\n",
|
|
"Finally, we choose the best model (or merge them using `ReuseManager`, covered in other guides) and export it."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.export import OWLExporter\n",
|
|
"\n",
|
|
"exporter = OWLExporter()\n",
|
|
"\n",
|
|
"# Export the NLP ontology by default, or the LLM one if preferred\n",
|
|
"target_ontology = llm_ontology if llm_ontology else nlp_ontology\n",
|
|
"\n",
|
|
"output_file = \"quantum_ontology.ttl\"\n",
|
|
"exporter.export(target_ontology, output_file, format=\"turtle\")\n",
|
|
"print(f\"Successfully exported ontology to {output_file}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Summary\n",
|
|
"\n",
|
|
"You have learned to:\n",
|
|
"1. **Extract Ontologies Programmatically**: Using `NERExtractor` for reliable, data-driven modeling.\n",
|
|
"2. **Generate Ontologies with AI**: Using `LLMOntologyGenerator` for conceptual, high-level modeling.\n",
|
|
"3. **Visualize and Compare**: Using `OntologyVisualizer` to inspect the structural differences.\n",
|
|
"4. **Validate and Export**: Ensuring quality before saving to OWL standards."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.8.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|