mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-11 04:01:32 +00:00
363 lines
13 KiB
Plaintext
363 lines
13 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)\n",
|
|
"\n",
|
|
"# 🚀 Your First Knowledge Graph\n",
|
|
"\n",
|
|
"## Overview\n",
|
|
"\n",
|
|
"This notebook walks you through creating your first knowledge graph from a simple document. You'll learn the complete end-to-end workflow from ingesting a file to visualizing the resulting knowledge graph — and every step consumes the real output of the step before it.\n",
|
|
"\n",
|
|
"> [!TIP]\n",
|
|
"> This is the perfect starting point if you are new to Semantica. No prior knowledge of knowledge graphs is required!\n",
|
|
"\n",
|
|
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/kg/)\n",
|
|
"\n",
|
|
"### 🎯 Learning Objectives\n",
|
|
"\n",
|
|
"- **Understand the Workflow**: Learn the `File → Parse → Extract → Graph → Visualize` pipeline\n",
|
|
"- **Ingest Data**: Load documents using `FileIngestor`\n",
|
|
"- **Parse Content**: Extract text using `DocumentParser`\n",
|
|
"- **Extract Knowledge**: Identify entities and relations using `NERExtractor` and `RelationExtractor`\n",
|
|
"- **Build Graph**: Construct a graph using `GraphBuilder`\n",
|
|
"- **Visualize**: See your graph come to life with `KGVisualizer`\n",
|
|
"\n",
|
|
"## Installation\n",
|
|
"\n",
|
|
"Install Semantica from PyPI:\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"pip install semantica\n",
|
|
"# Or with all optional dependencies:\n",
|
|
"pip install semantica[all]\n",
|
|
"```\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"## 🔄 Simple End-to-End Workflow\n",
|
|
"\n",
|
|
"The complete workflow consists of five main steps:\n",
|
|
"\n",
|
|
"1. **📥 Ingest** - Load data from files or other sources\n",
|
|
"2. **📄 Parse** - Extract and structure content from documents\n",
|
|
"3. **⛏️ Extract** - Identify entities and relationships\n",
|
|
"4. **🕸️ Build Graph** - Construct the knowledge graph\n",
|
|
"5. **📊 Visualize** - Render and analyze the graph\n",
|
|
"\n",
|
|
"Each step is demonstrated in the code cells below, and each cell can be rerun on its own: the sample file is only removed by the optional cleanup cell at the very end.\n",
|
|
"\n",
|
|
"> [!TIP]\n",
|
|
"> **Alternative: Using Semantica Framework**\n",
|
|
">\n",
|
|
"> For a simpler, high-level approach, you can use the `Semantica` framework class which orchestrates all these steps:\n",
|
|
">\n",
|
|
"> ```python\n",
|
|
"> from semantica.core import Semantica\n",
|
|
">\n",
|
|
"> framework = Semantica()\n",
|
|
"> framework.initialize()\n",
|
|
">\n",
|
|
"> result = framework.build_knowledge_base(\n",
|
|
"> sources=[\"sample_document.txt\"],\n",
|
|
"> embeddings=True,\n",
|
|
"> graph=True\n",
|
|
"> )\n",
|
|
">\n",
|
|
"> framework.shutdown()\n",
|
|
"> ```\n",
|
|
">\n",
|
|
"> This notebook shows the step-by-step approach for learning. See [Core Module Usage Guide](../../../semantica/core/core_usage.md) for more details.\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"## 📂 Step 1: Ingest a File\n",
|
|
"\n",
|
|
"In this step, we'll use `FileIngestor` to load a document. The ingestor supports various file formats including PDF, DOCX, TXT, and more. Writing the sample file is idempotent, so this cell can be rerun at any time.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"%pip install semantica\n",
|
|
"\n",
|
|
"# spaCy models are distributed separately from the spaCy library. This lesson\n",
|
|
"# relies on the English model to recognize standalone places such as Cupertino.\n",
|
|
"import sys\n",
|
|
"import subprocess\n",
|
|
"import spacy\n",
|
|
"\n",
|
|
"try:\n",
|
|
" spacy.load(\"en_core_web_sm\")\n",
|
|
"except OSError:\n",
|
|
" subprocess.check_call([sys.executable, \"-m\", \"spacy\", \"download\", \"en_core_web_sm\"])\n"
|
|
],
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"from pathlib import Path\n",
|
|
"\n",
|
|
"from semantica.ingest import FileIngestor\n",
|
|
"\n",
|
|
"sample_text = \"\"\"Apple Inc. is headquartered in Cupertino, California.\n",
|
|
"In 1976, Steve Jobs founded Apple Inc.\n",
|
|
"Tim Cook is the CEO of Apple Inc.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"sample_file = Path(\"sample_document.txt\")\n",
|
|
"sample_file.write_text(sample_text)\n",
|
|
"\n",
|
|
"print(f\"File: {sample_file}\")\n",
|
|
"print(f\"Content length: {len(sample_text)} characters\")\n",
|
|
"\n",
|
|
"ingestor = FileIngestor()\n",
|
|
"file_object = ingestor.ingest_file(sample_file, read_content=True)\n",
|
|
"print(f\" File name: {file_object.name}\")\n",
|
|
"print(f\" File type: {file_object.file_type}\")\n",
|
|
"print(f\" Content available: {file_object.content is not None}\")"
|
|
],
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 📄 Step 2: Parse the Document\n",
|
|
"\n",
|
|
"After ingesting the file, we need to parse it to extract the text content. `DocumentParser.parse_document()` returns the extracted text under the `\"text\"` key.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"from semantica.parse import DocumentParser\n",
|
|
"\n",
|
|
"parser = DocumentParser()\n",
|
|
"parsed_document = parser.parse_document(str(sample_file))\n",
|
|
"\n",
|
|
"parsed_content = parsed_document.get(\"text\", \"\")\n",
|
|
"assert parsed_content.strip(), \"Parsing produced no text — check the input file\"\n",
|
|
"\n",
|
|
"print(f\"Parsed content length: {len(parsed_content)} characters\")\n",
|
|
"print(f\"Preview: {parsed_content[:120]}...\")"
|
|
],
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## ⛏️ Step 3: Extract Entities and Relations\n",
|
|
"\n",
|
|
"Now we'll extract entities and relations from the parsed text. `NERExtractor` identifies people, organizations, locations and dates; `RelationExtractor` finds relations between those mentions. Both operate on the *parsed content from Step 2* — not on a copy of the raw string.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
|
|
"\n",
|
|
"ner_extractor = NERExtractor()\n",
|
|
"relation_extractor = RelationExtractor()\n",
|
|
"\n",
|
|
"mentions = ner_extractor.extract(parsed_content)\n",
|
|
"relations = relation_extractor.extract(parsed_content, mentions)\n",
|
|
"\n",
|
|
"print(\"Entity mentions:\")\n",
|
|
"for mention in mentions:\n",
|
|
" print(f\" {mention.text!r:<13} {mention.label:<7} span=[{mention.start_char}:{mention.end_char}]\")\n",
|
|
"\n",
|
|
"print(\"\\nExtracted relations:\")\n",
|
|
"for rel in relations:\n",
|
|
" print(f\" {rel.subject.text!r} --{rel.predicate}--> {rel.object.text!r}\")"
|
|
],
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 🕸️ Step 4: Build the Knowledge Graph\n",
|
|
"\n",
|
|
"Using the extracted entities and relations, we construct a knowledge graph with `GraphBuilder`. Every mention gets a graph ID, and each edge is built from the actual `Relation.subject` / `Relation.object` endpoints.\n",
|
|
"\n",
|
|
"> [!NOTE]\n",
|
|
"> The graph will contain one node per *mention*, so `Apple Inc.` appears three times. Merging duplicate mentions into one canonical entity is covered in [07_Building_Knowledge_Graphs.ipynb](./07_Building_Knowledge_Graphs.ipynb).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"from semantica.kg import GraphBuilder\n",
|
|
"\n",
|
|
"entities = []\n",
|
|
"span_to_id = {}\n",
|
|
"for i, mention in enumerate(mentions, 1):\n",
|
|
" graph_id = f\"e{i}\"\n",
|
|
" span_to_id[(mention.start_char, mention.end_char)] = graph_id\n",
|
|
" entities.append({\n",
|
|
" \"id\": graph_id,\n",
|
|
" \"type\": mention.label,\n",
|
|
" \"name\": mention.text,\n",
|
|
" \"properties\": {},\n",
|
|
" })\n",
|
|
"\n",
|
|
"relationships = []\n",
|
|
"for rel in relations:\n",
|
|
" source_id = span_to_id.get((rel.subject.start_char, rel.subject.end_char))\n",
|
|
" target_id = span_to_id.get((rel.object.start_char, rel.object.end_char))\n",
|
|
" if source_id is None or target_id is None:\n",
|
|
" print(f\"Skipping relation with unmapped endpoint: \"\n",
|
|
" f\"{rel.subject.text!r} --{rel.predicate}--> {rel.object.text!r}\")\n",
|
|
" continue\n",
|
|
" relationships.append({\n",
|
|
" \"source\": source_id,\n",
|
|
" \"target\": target_id,\n",
|
|
" \"type\": rel.predicate,\n",
|
|
" \"properties\": {},\n",
|
|
" })\n",
|
|
"\n",
|
|
"builder = GraphBuilder()\n",
|
|
"knowledge_graph = builder.build({\"entities\": entities, \"relationships\": relationships})\n",
|
|
"\n",
|
|
"id_to_name = {entity[\"id\"]: entity[\"name\"] for entity in entities}\n",
|
|
"\n",
|
|
"print(f\"Nodes (entities): {len(knowledge_graph['entities'])}\")\n",
|
|
"for entity in knowledge_graph[\"entities\"]:\n",
|
|
" print(f\" {entity['id']}: {entity['name']} ({entity['type']})\")\n",
|
|
"\n",
|
|
"print(f\"\\nEdges (relationships): {len(knowledge_graph['relationships'])}\")\n",
|
|
"for relationship in knowledge_graph[\"relationships\"]:\n",
|
|
" print(f\" {id_to_name[relationship['source']]} \"\n",
|
|
" f\"--{relationship['type']}--> {id_to_name[relationship['target']]}\")\n",
|
|
"\n",
|
|
"edges = {\n",
|
|
" (id_to_name[r[\"source\"]], r[\"type\"], id_to_name[r[\"target\"]])\n",
|
|
" for r in knowledge_graph[\"relationships\"]\n",
|
|
"}\n",
|
|
"assert (\"Apple Inc.\", \"located_in\", \"Cupertino\") in edges\n",
|
|
"assert (\"Tim Cook\", \"works_for\", \"Apple Inc.\") in edges"
|
|
],
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 📊 Step 5: Visualize and Analyze\n",
|
|
"\n",
|
|
"Finally, we render the knowledge graph with `KGVisualizer` and look at its structure. `visualize_network()` accepts the `GraphBuilder` result directly and can save an interactive HTML file.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"from semantica.visualization import KGVisualizer\n",
|
|
"\n",
|
|
"visualizer = KGVisualizer()\n",
|
|
"fig = visualizer.visualize_network(\n",
|
|
" knowledge_graph, output=\"html\", file_path=\"knowledge_graph.html\"\n",
|
|
")\n",
|
|
"print(\"Saved interactive visualization to knowledge_graph.html\")\n",
|
|
"\n",
|
|
"entity_types = {}\n",
|
|
"for entity in knowledge_graph[\"entities\"]:\n",
|
|
" entity_types[entity[\"type\"]] = entity_types.get(entity[\"type\"], 0) + 1\n",
|
|
"\n",
|
|
"print(\"\\nEntities by type:\")\n",
|
|
"for entity_type, count in sorted(entity_types.items()):\n",
|
|
" print(f\" - {entity_type}: {count}\")\n",
|
|
"\n",
|
|
"relationship_types = {}\n",
|
|
"for relationship in knowledge_graph[\"relationships\"]:\n",
|
|
" relationship_types[relationship[\"type\"]] = (\n",
|
|
" relationship_types.get(relationship[\"type\"], 0) + 1\n",
|
|
" )\n",
|
|
"\n",
|
|
"print(\"\\nRelationships by type:\")\n",
|
|
"for relationship_type, count in sorted(relationship_types.items()):\n",
|
|
" print(f\" - {relationship_type}: {count}\")\n",
|
|
"\n",
|
|
"fig"
|
|
],
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 🧹 Optional: Clean Up\n",
|
|
"\n",
|
|
"Run this cell only when you are done with the notebook. Earlier cells read `sample_document.txt`, so they stay rerunnable until you delete it here.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"for path in [sample_file, Path(\"knowledge_graph.html\")]:\n",
|
|
" if path.exists():\n",
|
|
" path.unlink()\n",
|
|
" print(f\"Removed {path}\")"
|
|
],
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Summary\n",
|
|
"\n",
|
|
"You've built your first knowledge graph, end to end:\n",
|
|
"\n",
|
|
"- **FileIngestor** loaded the sample document\n",
|
|
"- **DocumentParser** returned its text under the `\"text\"` key\n",
|
|
"- **NERExtractor** / **RelationExtractor** produced real mentions and relations from that text\n",
|
|
"- **GraphBuilder** turned them into a graph whose edges come from the actual relation endpoints\n",
|
|
"- **KGVisualizer** rendered the result as an interactive network\n",
|
|
"\n",
|
|
"Next: merge duplicate mentions with `EntityResolver` in [07_Building_Knowledge_Graphs.ipynb](./07_Building_Knowledge_Graphs.ipynb), or explore graph metrics in the Graph Analytics notebook.\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|