mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Added CloudFormation template that creates a dev cluster with a single [t3 instance](https://docs.aws.amazon.com/neptune/latest/userguide/manage-console-instances-t3.html) configured with a [public endpoint](https://docs.aws.amazon.com/neptune/latest/userguide/neptune-public-endpoints.html) and IAM Auth enabled (required for public endpoint), and creates an IAM User using least-privilege principles. See Get started with Neptune Database for free on the [Amazon Neptune pricing page](https://aws.amazon.com/neptune/pricing/).
Includes the CloudFormation template in the same directory as the [Amazon Neptune Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/21_Amazon_Neptune_Store.ipynb) and references it as a prerequisite in the cookbook.
This commit is contained in:
Don Simpson
@@ -5,6 +5,7 @@ repos:
|
||||
- id: trailing-whitespace
|
||||
- id: end-of-file-fixer
|
||||
- id: check-yaml
|
||||
exclude: 'neptune-setup\.yaml$'
|
||||
- id: check-json
|
||||
- id: check-toml
|
||||
- id: check-added-large-files
|
||||
@@ -49,9 +50,15 @@ repos:
|
||||
hooks:
|
||||
- id: yamllint
|
||||
args: ['-d', '{extends: default, rules: {line-length: {max: 120}}}']
|
||||
exclude: 'neptune-setup\.yaml$'
|
||||
|
||||
- repo: https://github.com/aws-cloudformation/cfn-lint
|
||||
rev: v1.43.3
|
||||
hooks:
|
||||
- id: cfn-lint
|
||||
files: 'neptune-setup\.yaml$'
|
||||
|
||||
# Removed slow hooks for faster development:
|
||||
# - mypy: Type checking (can be run manually or in CI)
|
||||
# - bandit: Security scanning (can be run separately)
|
||||
# - pytest: Testing (should be run manually, not on every commit)
|
||||
|
||||
|
||||
@@ -25,6 +25,57 @@
|
||||
"- AWS credentials configured (boto3, environment variables, or IAM role)\n",
|
||||
"- Network access to your Neptune cluster (VPC, security groups)\n",
|
||||
"\n",
|
||||
"#### Quick Setup with CloudFormation\n",
|
||||
"\n",
|
||||
"If you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n",
|
||||
"\n",
|
||||
"```bash\n",
|
||||
"# Deploy the Neptune stack (takes ~15-20 minutes)\n",
|
||||
"aws cloudformation create-stack \\\n",
|
||||
" --stack-name semantica-neptune \\\n",
|
||||
" --template-body file://neptune-setup.yaml \\\n",
|
||||
" --capabilities CAPABILITY_NAMED_IAM\n",
|
||||
"\n",
|
||||
"# Wait for stack creation to complete\n",
|
||||
"aws cloudformation wait stack-create-complete --stack-name semantica-neptune\n",
|
||||
"\n",
|
||||
"# Get the outputs (endpoint, port, credentials)\n",
|
||||
"aws cloudformation describe-stacks --stack-name semantica-neptune \\\n",
|
||||
" --query 'Stacks[0].Outputs' --output table\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"The template creates:\n",
|
||||
"- VPC with public subnets and Internet Gateway\n",
|
||||
"- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n",
|
||||
"- IAM user with least-privilege access for OpenCypher queries\n",
|
||||
"- Security group allowing Bolt protocol (port 8182) access\n",
|
||||
"\n",
|
||||
"> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager.\n",
|
||||
"\n",
|
||||
"**Outputs:**\n",
|
||||
"- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n",
|
||||
"- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n",
|
||||
"- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n",
|
||||
"- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n",
|
||||
"- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n",
|
||||
"\n",
|
||||
"**Cleanup:**\n",
|
||||
"```bash\n",
|
||||
"aws cloudformation delete-stack --stack-name semantica-neptune\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n",
|
||||
"\n",
|
||||
"| Resource | Cost (USD) |\n",
|
||||
"| --- | --- |\n",
|
||||
"| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n",
|
||||
"| Storage (10 GB) | ~1/month |\n",
|
||||
"| I/O requests | ~1-5/month |\n",
|
||||
"| Public IPv4 address | ~3.60/month (0.005/hr) |\n",
|
||||
"| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n",
|
||||
"\n",
|
||||
"> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n",
|
||||
"\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
@@ -70,14 +121,21 @@
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Neptune cluster configuration - REPLACE WITH YOUR VALUES\n",
|
||||
"# (Get these from CloudFormation stack outputs)\n",
|
||||
"os.environ[\"NEPTUNE_ENDPOINT\"] = \"your-cluster.us-east-1.neptune.amazonaws.com\"\n",
|
||||
"os.environ[\"NEPTUNE_PORT\"] = \"8182\"\n",
|
||||
"os.environ[\"AWS_REGION\"] = \"us-east-1\"\n",
|
||||
"\n",
|
||||
"# AWS credentials (if using IAM Auth and not relying on IAM role or ~/.aws/credentials)\n",
|
||||
"# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"your-access-key-id\"\n",
|
||||
"# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"your-secret-access-key\"\n",
|
||||
"# os.environ[\"AWS_SESSION_TOKEN\"] = \"your-session-token\"\n",
|
||||
"# AWS credentials for IAM Authentication\n",
|
||||
"# Option 1: IAM User (static credentials from CloudFormation template)\n",
|
||||
"# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"AKIA...\" # From AwsAccessKeyId output\n",
|
||||
"# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\" # From AwsSecretAccessKey output\n",
|
||||
"# Note: No AWS_SESSION_TOKEN needed for IAM users\n",
|
||||
"\n",
|
||||
"# Option 2: IAM Role / Temporary credentials (e.g., STS AssumeRole, EC2 instance role)\n",
|
||||
"# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"ASIA...\" # Temporary access key\n",
|
||||
"# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\" # Temporary secret key\n",
|
||||
"# os.environ[\"AWS_SESSION_TOKEN\"] = \"...\" # REQUIRED for temporary credentials\n",
|
||||
"\n",
|
||||
"print(f\"Neptune Endpoint: {os.environ.get('NEPTUNE_ENDPOINT')}\")\n",
|
||||
"print(f\"AWS Region: {os.environ.get('AWS_REGION')}\")"
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
AWSTemplateFormatVersion: '2010-09-09'
|
||||
Description: >
|
||||
Amazon Neptune cluster with public endpoint, IAM authentication, and least-privilege
|
||||
IAM user for Semantica cookbook. Uses db.t3.medium (most cost-effective Neptune instance type).
|
||||
|
||||
Parameters:
|
||||
EnvironmentName:
|
||||
Type: String
|
||||
Default: semantica-neptune
|
||||
Description: Environment name prefix for resource naming
|
||||
|
||||
Resources:
|
||||
# =============================================================================
|
||||
# VPC & NETWORKING
|
||||
# =============================================================================
|
||||
|
||||
VPC:
|
||||
Type: AWS::EC2::VPC
|
||||
Properties:
|
||||
CidrBlock: 10.0.0.0/16
|
||||
EnableDnsHostnames: true
|
||||
EnableDnsSupport: true
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-vpc
|
||||
|
||||
InternetGateway:
|
||||
Type: AWS::EC2::InternetGateway
|
||||
Properties:
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-igw
|
||||
|
||||
InternetGatewayAttachment:
|
||||
Type: AWS::EC2::VPCGatewayAttachment
|
||||
Properties:
|
||||
InternetGatewayId: !Ref InternetGateway
|
||||
VpcId: !Ref VPC
|
||||
|
||||
PublicSubnet1:
|
||||
Type: AWS::EC2::Subnet
|
||||
Properties:
|
||||
VpcId: !Ref VPC
|
||||
AvailabilityZone: !Select [0, !GetAZs '']
|
||||
CidrBlock: 10.0.1.0/24
|
||||
MapPublicIpOnLaunch: true
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-public-subnet-1
|
||||
|
||||
PublicSubnet2:
|
||||
Type: AWS::EC2::Subnet
|
||||
Properties:
|
||||
VpcId: !Ref VPC
|
||||
AvailabilityZone: !Select [1, !GetAZs '']
|
||||
CidrBlock: 10.0.2.0/24
|
||||
MapPublicIpOnLaunch: true
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-public-subnet-2
|
||||
|
||||
PublicRouteTable:
|
||||
Type: AWS::EC2::RouteTable
|
||||
Properties:
|
||||
VpcId: !Ref VPC
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-public-rt
|
||||
|
||||
DefaultPublicRoute:
|
||||
Type: AWS::EC2::Route
|
||||
DependsOn: InternetGatewayAttachment
|
||||
Properties:
|
||||
RouteTableId: !Ref PublicRouteTable
|
||||
DestinationCidrBlock: 0.0.0.0/0
|
||||
GatewayId: !Ref InternetGateway
|
||||
|
||||
PublicSubnet1RouteTableAssociation:
|
||||
Type: AWS::EC2::SubnetRouteTableAssociation
|
||||
Properties:
|
||||
RouteTableId: !Ref PublicRouteTable
|
||||
SubnetId: !Ref PublicSubnet1
|
||||
|
||||
PublicSubnet2RouteTableAssociation:
|
||||
Type: AWS::EC2::SubnetRouteTableAssociation
|
||||
Properties:
|
||||
RouteTableId: !Ref PublicRouteTable
|
||||
SubnetId: !Ref PublicSubnet2
|
||||
|
||||
# =============================================================================
|
||||
# SECURITY GROUP
|
||||
# =============================================================================
|
||||
|
||||
NeptuneSecurityGroup:
|
||||
Type: AWS::EC2::SecurityGroup
|
||||
Properties:
|
||||
GroupName: !Sub ${EnvironmentName}-neptune-sg
|
||||
GroupDescription: Security group for Neptune cluster - allows Bolt protocol access
|
||||
VpcId: !Ref VPC
|
||||
SecurityGroupIngress:
|
||||
- IpProtocol: tcp
|
||||
FromPort: 8182
|
||||
ToPort: 8182
|
||||
CidrIp: 0.0.0.0/0
|
||||
Description: Allow Bolt protocol access from anywhere
|
||||
SecurityGroupEgress:
|
||||
- IpProtocol: -1
|
||||
CidrIp: 0.0.0.0/0
|
||||
Description: Allow all outbound traffic
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-neptune-sg
|
||||
|
||||
# =============================================================================
|
||||
# NEPTUNE CLUSTER
|
||||
# =============================================================================
|
||||
|
||||
NeptuneSubnetGroup:
|
||||
Type: AWS::Neptune::DBSubnetGroup
|
||||
Properties:
|
||||
DBSubnetGroupDescription: Subnet group for Neptune cluster
|
||||
DBSubnetGroupName: !Sub ${EnvironmentName}-subnet-group
|
||||
SubnetIds:
|
||||
- !Ref PublicSubnet1
|
||||
- !Ref PublicSubnet2
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-subnet-group
|
||||
|
||||
NeptuneCluster:
|
||||
Type: AWS::Neptune::DBCluster
|
||||
Properties:
|
||||
DBClusterIdentifier: !Sub ${EnvironmentName}-cluster
|
||||
DBSubnetGroupName: !Ref NeptuneSubnetGroup
|
||||
VpcSecurityGroupIds:
|
||||
- !Ref NeptuneSecurityGroup
|
||||
EngineVersion: '1.4.6.3'
|
||||
IamAuthEnabled: true
|
||||
StorageEncrypted: true
|
||||
DeletionProtection: false
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-cluster
|
||||
|
||||
NeptuneInstance:
|
||||
Type: AWS::Neptune::DBInstance
|
||||
Properties:
|
||||
DBInstanceIdentifier: !Sub ${EnvironmentName}-instance
|
||||
DBInstanceClass: db.t3.medium
|
||||
DBClusterIdentifier: !Ref NeptuneCluster
|
||||
PubliclyAccessible: true
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-instance
|
||||
|
||||
# =============================================================================
|
||||
# IAM USER WITH LEAST PRIVILEGES
|
||||
# =============================================================================
|
||||
|
||||
NeptuneUser:
|
||||
Type: AWS::IAM::User
|
||||
Properties:
|
||||
UserName: !Sub ${EnvironmentName}-user
|
||||
Tags:
|
||||
- Key: Name
|
||||
Value: !Sub ${EnvironmentName}-user
|
||||
|
||||
NeptuneUserPolicy:
|
||||
Type: AWS::IAM::Policy
|
||||
Properties:
|
||||
PolicyName: !Sub ${EnvironmentName}-neptune-access
|
||||
Users:
|
||||
- !Ref NeptuneUser
|
||||
PolicyDocument:
|
||||
Version: '2012-10-17'
|
||||
Statement:
|
||||
- Sid: NeptuneDataAccess
|
||||
Effect: Allow
|
||||
Action:
|
||||
- neptune-db:connect
|
||||
- neptune-db:ReadDataViaQuery
|
||||
- neptune-db:WriteDataViaQuery
|
||||
- neptune-db:DeleteDataViaQuery
|
||||
Resource: !Sub
|
||||
- arn:aws:neptune-db:${AWS::Region}:${AWS::AccountId}:${ClusterResourceId}/*
|
||||
- ClusterResourceId: !GetAtt NeptuneCluster.ClusterResourceId
|
||||
|
||||
NeptuneUserAccessKey:
|
||||
Type: AWS::IAM::AccessKey
|
||||
Properties:
|
||||
UserName: !Ref NeptuneUser
|
||||
|
||||
# =============================================================================
|
||||
# OUTPUTS
|
||||
# =============================================================================
|
||||
|
||||
Outputs:
|
||||
NeptuneEndpoint:
|
||||
Description: Neptune cluster endpoint (hostname only) - use as NEPTUNE_ENDPOINT
|
||||
Value: !GetAtt NeptuneCluster.Endpoint
|
||||
|
||||
NeptunePort:
|
||||
Description: Neptune cluster port - use as NEPTUNE_PORT
|
||||
Value: !GetAtt NeptuneCluster.Port
|
||||
|
||||
AwsAccessKeyId:
|
||||
Description: Access key ID for the Neptune IAM user - use as AWS_ACCESS_KEY_ID
|
||||
Value: !Ref NeptuneUserAccessKey
|
||||
|
||||
AwsSecretAccessKey:
|
||||
Description: Secret access key for the Neptune IAM user - use as AWS_SECRET_ACCESS_KEY
|
||||
Value: !GetAtt NeptuneUserAccessKey.SecretAccessKey
|
||||
|
||||
AwsRegion:
|
||||
Description: AWS region where Neptune is deployed - use as AWS_REGION
|
||||
Value: !Ref AWS::Region
|
||||
|
||||
NeptuneClusterResourceId:
|
||||
Description: Neptune cluster resource ID (for IAM policy reference)
|
||||
Value: !GetAtt NeptuneCluster.ClusterResourceId
|
||||
|
||||
VpcId:
|
||||
Description: VPC ID
|
||||
Value: !Ref VPC
|
||||
|
||||
SecurityGroupId:
|
||||
Description: Neptune security group ID
|
||||
Value: !Ref NeptuneSecurityGroup
|
||||
Reference in New Issue
Block a user