semantica/cookbook/introduction/21_Amazon_Neptune_Store.ipynb

Amazon Neptune Graph Store

Overview

This notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).

Key Features

• IAM Authentication: Secure access using AWS SigV4 signatures via AuthManager
• OpenCypher Support: Query using standard OpenCypher syntax
• Bolt Protocol: Uses Neo4j Bolt driver for efficient binary communication
• Native ~id Support: Leverages Neptune's native element ID handling
• Full CRUD Operations: Create, read, update, delete nodes and relationships
• Automatic Retry: Built-in retry logic with exponential backoff for transient errors

Prerequisites

• An Amazon Neptune Database cluster
• AWS credentials configured (boto3, environment variables, or IAM role)
• Network access to your Neptune cluster (VPC, security groups)
• Your public IP address or VPN/office CIDR (run curl ifconfig.me to find your public IP), used below to restrict database access

Quick Setup with CloudFormation

If you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:

# Deploy the Neptune stack (takes ~15-20 minutes)
# Replace 203.0.113.25/32 with your own public IP (run `curl ifconfig.me` to find it)
# or your office/VPN CIDR. This restricts who can reach the database on the
# network level - never widen it to 0.0.0.0/0 outside of a short-lived local experiment.
aws cloudformation create-stack \
  --stack-name semantica-neptune \
  --template-body file://neptune-setup.yaml \
  --parameters ParameterKey=ClientCidr,ParameterValue=203.0.113.25/32 \
  --capabilities CAPABILITY_NAMED_IAM

# Wait for stack creation to complete
aws cloudformation wait stack-create-complete --stack-name semantica-neptune

# Get the outputs (endpoint, port, credentials)
aws cloudformation describe-stacks --stack-name semantica-neptune \
  --query 'Stacks[0].Outputs' --output table

The template creates:

• VPC with public subnets, Internet Gateway, and VPC Flow Logs (to CloudWatch Logs)
• Neptune cluster (db.t3.medium) with IAM authentication enabled
• IAM user with least-privilege access for OpenCypher queries
• Security group allowing Bolt protocol (port 8182) access only from the ClientCidr you specify

⚠️ Security Note: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager. The ClientCidr parameter is required (no default) precisely so the database is never silently exposed to the whole internet.

Outputs:

• NeptuneEndpoint - Cluster hostname (use as NEPTUNE_ENDPOINT)
• NeptunePort - 8182 (use as NEPTUNE_PORT)
• AwsAccessKeyId - IAM user access key (use as AWS_ACCESS_KEY_ID)
• AwsSecretAccessKey - IAM user secret key in plaintext (use as AWS_SECRET_ACCESS_KEY)
• AwsRegion - Deployment region (use as AWS_REGION)

Cleanup:

aws cloudformation delete-stack --stack-name semantica-neptune

Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):

Resource	Cost (USD)
Neptune db.t3.medium instance	~96/month (0.132/hr)
Storage (10 GB)	~1/month
I/O requests	~1-5/month
Public IPv4 address	~3.60/month (0.005/hr)
VPC Flow Logs (CloudWatch Logs)	~1-2/month depending on traffic
VPC, subnets, route tables, Internet Gateway, IAM	No Additional Charge

Free Tier: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.

---

Installation

# Install Semantica with Neptune support
pip install semantica

# Required dependencies (installed automatically)
pip install boto3 neo4j

In [ ]:

!pip install semantica

Configuration

Set your Neptune cluster endpoint and AWS credentials. Replace the placeholder values with your actual configuration.

In [ ]:

import os

# Neptune cluster configuration - REPLACE WITH YOUR VALUES
# (Get these from CloudFormation stack outputs)
os.environ["NEPTUNE_ENDPOINT"] = "your-cluster.us-east-1.neptune.amazonaws.com"
os.environ["NEPTUNE_PORT"] = "8182"
os.environ["AWS_REGION"] = "us-east-1"

# AWS credentials for IAM Authentication
# Option 1: IAM User (static credentials from CloudFormation template)
# os.environ["AWS_ACCESS_KEY_ID"] = "AKIA..."  # From AwsAccessKeyId output
# os.environ["AWS_SECRET_ACCESS_KEY"] = "..."  # From AwsSecretAccessKey output
# Note: No AWS_SESSION_TOKEN needed for IAM users

# Option 2: IAM Role / Temporary credentials (e.g., STS AssumeRole, EC2 instance role)
# os.environ["AWS_ACCESS_KEY_ID"] = "ASIA..."  # Temporary access key
# os.environ["AWS_SECRET_ACCESS_KEY"] = "..."  # Temporary secret key
# os.environ["AWS_SESSION_TOKEN"] = "..."      # REQUIRED for temporary credentials

print(f"Neptune Endpoint: {os.environ.get('NEPTUNE_ENDPOINT')}")
print(f"AWS Region: {os.environ.get('AWS_REGION')}")

Step 1: Initialize Neptune Store

Initialize a connection to your Amazon Neptune cluster with IAM authentication.

In [ ]:

import os
from semantica.graph_store import GraphStore

# Option 1: Using GraphStore factory (recommended)
neptune_store = GraphStore(
    backend="neptune",
    endpoint=os.environ.get("NEPTUNE_ENDPOINT"),
    port=int(os.environ.get("NEPTUNE_PORT", 8182)),
    region=os.environ.get("AWS_REGION", "us-east-1"),
    iam_auth=True,
)

# Connect to Neptune
neptune_store.connect()
print("Connected to Amazon Neptune!")

Development/Testing Without IAM Auth

For development or testing environments where IAM authentication is not required (e.g., Neptune notebooks or VPC-only access), you can disable IAM signing:

In [ ]:

# For dev/test environments without IAM authentication
neptune_store_dev = GraphStore(
    backend="neptune",
    endpoint=os.environ.get("NEPTUNE_ENDPOINT"),
    port=int(os.environ.get("NEPTUNE_PORT", 8182)),
    region=os.environ.get("AWS_REGION", "us-east-1"),
    iam_auth=False,  # Disable IAM signing for dev/test
)
neptune_store_dev.connect()

Authentication Options

IAM Authentication (recommended for production) automatically uses the AWS credential chain:

1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
2. AWS credentials file (~/.aws/credentials)
3. IAM role (for EC2, Lambda, ECS)

Step 2: Node Operations

Creating Nodes

Nodes represent entities in your graph. Each node can have:

• ID: A unique identifier (custom or auto-generated UUID)
• Labels: Categories/types (e.g., Person, Company)
• Properties: Key-value pairs (e.g., {"name": "Alice", "age": 30})

In [ ]:

# Create a single node with custom ID (id in properties)
alice = neptune_store.create_node(
    labels=["Person"],
    properties={"id": "alice", "name": "Alice", "age": 30, "role": "Engineer"}
)
print(f"Created node: {alice}")

# Create a node with auto-generated UUID (no id in properties)
bob = neptune_store.create_node(
    labels=["Person"],
    properties={"name": "Bob", "age": 25, "role": "Designer"}
)
print(f"Created node with UUID: {bob['id']}")

# Create a company node with auto-generated ID
acme = neptune_store.create_node(
    labels=["Company"],
    properties={"name": "Acme Corp", "industry": "Technology", "founded": 2010}
)
print(f"Created company: {acme}")

Creating Multiple Nodes (Batch)

In [ ]:

# Batch create nodes for better performance
# Include 'id' in properties for custom IDs
nodes_data = [
    {"labels": ["Person"], "properties": {"id": "charlie", "name": "Charlie", "age": 35}},
    {"labels": ["Person"], "properties": {"id": "diana", "name": "Diana", "age": 28}},
    {"labels": ["Location"], "properties": {"name": "San Francisco", "state": "CA"}},
]

created_nodes = neptune_store.create_nodes(nodes_data)
print(f"Created {len(created_nodes)} nodes in batch")

Retrieving Nodes

In [ ]:

# Get a specific node by ID
alice_node = neptune_store.get_node(node_id="alice")
print(f"Retrieved: {alice_node}")

# Get nodes by label
people = neptune_store.get_nodes(labels=["Person"], limit=10)
print(f"Found {len(people)} Person nodes:")
for person in people:
    print(f"  - {person.get('properties', {}).get('name')}")

# Get nodes by properties
engineers = neptune_store.get_nodes(
    labels=["Person"],
    properties={"role": "Engineer"},
    limit=5
)
print(f"Found {len(engineers)} engineers")

Updating Nodes

In [ ]:

# Update node properties (merge mode - default)
updated_alice = neptune_store.update_node(
    node_id="alice",
    properties={"age": 31, "department": "AI Research"},
    merge=True
)
print(f"Updated Alice: {updated_alice}")

# Replace all properties (merge=False)
# WARNING: This removes properties not in the update
replaced = neptune_store.update_node(
    node_id="charlie",
    properties={"name": "Charlie", "age": 36},
    merge=False
)

Deleting Nodes

In [ ]:

# Delete a node (with detach=True to also delete relationships)
deleted = neptune_store.delete_node(node_id="diana", detach=True)
print(f"Deleted diana: {deleted}")

# Without detach (fails if node has relationships)
# neptune_store.delete_node(node_id="alice", detach=False)

Step 3: Relationship Operations

Creating Relationships

Relationships connect nodes and represent connections between entities.

In [ ]:

# Create a relationship between Alice and Acme
works_at = neptune_store.create_relationship(
    start_node_id="alice",
    end_node_id=acme["id"],
    rel_type="WORKS_AT",
    properties={"since": 2020, "position": "Senior Engineer"}
)
print(f"Created relationship: {works_at}")

# Create a KNOWS relationship between people
knows_rel = neptune_store.create_relationship(
    start_node_id="alice",
    end_node_id=bob["id"],
    rel_type="KNOWS",
    properties={"since": 2019}
)

Retrieving Relationships

In [ ]:

# Get all relationships for a node
alice_rels = neptune_store.get_relationships(node_id="alice", direction="both")
print(f"Alice has {len(alice_rels)} relationships")

# Get outgoing relationships only
outgoing = neptune_store.get_relationships(node_id="alice", direction="out")

# Filter by relationship type
works_rels = neptune_store.get_relationships(
    node_id="alice",
    rel_type="WORKS_AT",
    direction="out"
)
print(f"Alice's work relationships: {len(works_rels)}")

Deleting Relationships

In [ ]:

# Delete a specific relationship by ID
if works_at.get("id"):
    deleted = neptune_store.delete_relationship(rel_id=works_at["id"])
    print(f"Deleted relationship: {deleted}")

Step 4: OpenCypher Queries

Amazon Neptune supports OpenCypher queries via the Bolt protocol. Execute complex graph patterns using standard Cypher syntax.

In [ ]:

# Simple query
results = neptune_store.execute_query(
    "MATCH (p:Person) RETURN p.name, p.age ORDER BY p.age"
)
print("People in the graph:")
for record in results.get("records", []):
    print(f"  - {record.get('p.name')}: {record.get('p.age')} years old")

In [ ]:

# Using parameters (safer and more efficient)
results = neptune_store.execute_query(
    "MATCH (p:Person) WHERE p.age > $min_age RETURN p.name, p.age",
    parameters={"min_age": 25}
)
print(f"People over 25: {len(results.get('records', []))}")

In [ ]:

# Find relationships between nodes
results = neptune_store.execute_query("""
    MATCH (p:Person)-[r:WORKS_AT]->(c:Company)
    RETURN p.name as employee, c.name as company, r.since as start_year
""")
for record in results.get("records", []):
    print(f"{record['employee']} works at {record['company']} since {record['start_year']}")

In [ ]:

# Count and aggregate
results = neptune_store.execute_query("""
    MATCH (p:Person)
    RETURN count(p) as total, avg(p.age) as avg_age, max(p.age) as max_age
""")
stats = results.get("records", [{}])[0]
print(f"Total: {stats.get('total')}, Avg Age: {stats.get('avg_age'):.1f}")

Step 5: Graph Analytics

Get Neighbors

Traverse the graph to find connected nodes.

In [ ]:

# Get immediate neighbors (depth=1)
neighbors = neptune_store.get_neighbors(
    node_id="alice",
    direction="both",
    depth=1
)
print(f"Alice's direct neighbors: {len(neighbors)}")

# Get neighbors up to 2 hops away
extended = neptune_store.get_neighbors(
    node_id="alice",
    direction="out",
    depth=2
)
print(f"Nodes within 2 hops: {len(extended)}")

Shortest Path

Find the shortest path between two nodes.

In [ ]:

# Find shortest path
path = neptune_store.shortest_path(
    start_node_id="alice",
    end_node_id="charlie",
    max_depth=5
)

if path:
    print("Path found!")
    print(f"  Length: {path.get('length')}")
    print(f"  Nodes: {len(path.get('nodes', []))}")
    print(f"  Relationships: {len(path.get('relationships', []))}")
else:
    print("No path found between nodes")

Step 6: Graph Statistics

Get comprehensive statistics about your graph.

In [ ]:

# Get graph statistics
stats = neptune_store.get_stats()

print("Graph Statistics:")
print(f"  Total nodes: {stats.get('node_count', 'N/A')}")
print(f"  Total relationships: {stats.get('relationship_count', 'N/A')}")

print("\nNode labels:")
for label, count in stats.get('label_counts', {}).items():
    print(f"  - {label}: {count}")

print("\nRelationship types:")
for rel_type, count in stats.get('relationship_type_counts', {}).items():
    print(f"  - {rel_type}: {count}")

Step 7: Connection Management

Always close connections when done to free resources.

In [ ]:

# Check connection status
status = neptune_store.get_status()
print(f"Connection status: {status}")

# Close the connection
neptune_store.close()
print("Connection closed")

Neptune-Specific Considerations

Native Element IDs

Neptune uses native ~id for element identification. Include id in properties to set a custom ID:

# Create a node with custom ID (include 'id' in properties)
node = neptune_store.create_node(
    labels=["Person"],
    properties={"id": "my-custom-id", "name": "Test"}
)

# Create a node with auto-generated UUID (omit 'id' from properties)
node = neptune_store.create_node(
    labels=["Person"],
    properties={"name": "Test"}
)

# The ID is used in id() function calls internally:
# MATCH (n) WHERE id(n) = 'my-custom-id' RETURN n

OpenCypher Considerations

Amazon Neptune Database's OpenCypher implementation has some differences from Neo4j:

1. No shortestPath() function: Use variable-length path patterns or allShortestPaths()
2. Labels syntax: Use labels(n) function to retrieve node labels
3. Property updates: Use SET n += {props} for merge behavior

For the complete OpenCypher specification supported by Amazon Neptune Database, see the AWS documentation.

Amazon Neptune Analytics

For analytical (OLAP) workloads such as graph algorithms, aggregations, and large-scale traversals, consider Amazon Neptune Analytics. Neptune Analytics complements Neptune Database by providing optimized performance for analytical queries while Neptune Database is optimized for transactional (OLTP) workloads.

Performance Tips

1. Use batch operations for creating multiple nodes/relationships
2. Use parameters in queries to enable query caching
3. Limit result sets with LIMIT clause

Summary

This notebook covered the Amazon Neptune Graph Store integration:

• IAM Authentication: Secure AWS SigV4 signing
• CRUD Operations: Full node and relationship management
• OpenCypher Queries: Standard graph query language
• Graph Analytics: Neighbors and shortest path algorithms
• Statistics & Monitoring: Graph metrics and status

Key Takeaways

• Neptune uses native ~id for element identification
• IAM authentication is recommended for production
• Bolt protocol provides efficient binary query interface
• Semantica abstracts Neptune-specific syntax differences

Next Steps

• Graph Store (Neo4j/FalkorDB) - Compare with other backends
• Building Knowledge Graphs - Build production KGs
• Graph Analytics - Advanced analytics algorithms