-
Notifications
You must be signed in to change notification settings - Fork 879
Addiing in zookeeper example, fixes #159 #1790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rshade
wants to merge
1
commit into
master
Choose a base branch
from
rshade/zookeeper
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
|
||
/bin/ | ||
/node_modules/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
name: aws-ts-zookeeper | ||
runtime: | ||
name: nodejs | ||
options: | ||
packagemanager: yarn | ||
description: Zookeeper on AWS with managed node group, launch template, and load balancer |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
# ZooKeeper cluster on AWS | ||
|
||
## Components and Features | ||
|
||
1. Networking: | ||
- VPC with 3 availability zones | ||
- Private subnets with NAT gateways | ||
- Security groups for ZooKeeper traffic | ||
|
||
2. Compute: | ||
- Auto Scaling Group with 3 nodes | ||
- Launch template with Ubuntu 24.04 | ||
- IAM roles and instance profile | ||
- CloudWatch integration | ||
|
||
3. Load Balancing: | ||
- Internal Application Load Balancer | ||
- Target group with health checks | ||
- Listener configuration | ||
|
||
4. Security: | ||
- Security groups with minimal required ports | ||
- IMDSv2 requirement | ||
- Encrypted EBS volumes | ||
- SSM access for management | ||
|
||
5. Monitoring: | ||
- CloudWatch agent configuration | ||
- CPU utilization alarms | ||
- Memory and disk monitoring | ||
|
||
## Deployment | ||
|
||
1. Set up your AWS credentials | ||
2. Create a Pulumi stack | ||
|
||
3. Configure the environment: | ||
|
||
```bash | ||
|
||
pulumi config set environment production | ||
``` | ||
|
||
4. Deploy: | ||
|
||
```bash | ||
|
||
pulumi up | ||
``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,276 @@ | ||||||
// Copyright 2016-2025, Pulumi Corporation. All rights reserved. | ||||||
|
||||||
import * as aws from "@pulumi/aws"; | ||||||
import * as awsx from "@pulumi/awsx"; | ||||||
import * as pulumi from "@pulumi/pulumi"; | ||||||
|
||||||
|
||||||
const config = new pulumi.Config(); | ||||||
const projectName = "zookeeper"; | ||||||
const environment = config.require("environment"); | ||||||
|
||||||
// VPC Configuration | ||||||
const vpc = new awsx.ec2.Vpc(`${projectName}-vpc`, { | ||||||
numberOfAvailabilityZones: 3, | ||||||
natGateways: { | ||||||
strategy: environment === "production" ? "OnePerAz" : "Single", | ||||||
}, | ||||||
tags: { | ||||||
Name: `${projectName}-vpc`, | ||||||
Environment: environment, | ||||||
}, | ||||||
}); | ||||||
|
||||||
// Security Groups | ||||||
const zookeeperSG = new aws.ec2.SecurityGroup(`${projectName}-sg`, { | ||||||
vpcId: vpc.vpcId, | ||||||
description: "Security group for ZooKeeper nodes", | ||||||
ingress: [ | ||||||
{ protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["10.0.0.0/8"] }, // SSH | ||||||
{ protocol: "tcp", fromPort: 2181, toPort: 2181, cidrBlocks: ["10.0.0.0/8"] }, // Client port | ||||||
{ protocol: "tcp", fromPort: 2888, toPort: 2888, cidrBlocks: ["10.0.0.0/8"] }, // Follower port | ||||||
{ protocol: "tcp", fromPort: 3888, toPort: 3888, cidrBlocks: ["10.0.0.0/8"] }, // Election port | ||||||
], | ||||||
egress: [{ | ||||||
protocol: "-1", | ||||||
fromPort: 0, | ||||||
toPort: 0, | ||||||
cidrBlocks: ["0.0.0.0/0"], | ||||||
}], | ||||||
tags: { | ||||||
Name: `${projectName}-sg`, | ||||||
Environment: environment, | ||||||
}, | ||||||
}); | ||||||
|
||||||
// IAM Role and Instance Profile | ||||||
const zookeeperRole = new aws.iam.Role(`${projectName}-role`, { | ||||||
assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ | ||||||
Service: "ec2.amazonaws.com", | ||||||
}), | ||||||
}); | ||||||
|
||||||
const zookeeperInstanceProfile = new aws.iam.InstanceProfile(`${projectName}-instance-profile`, { | ||||||
role: zookeeperRole.name, | ||||||
}); | ||||||
|
||||||
// SSM Policy Attachment | ||||||
const ssmPolicy = new aws.iam.RolePolicyAttachment(`${projectName}-ssm-policy`, { | ||||||
role: zookeeperRole.name, | ||||||
policyArn: "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore", | ||||||
}); | ||||||
|
||||||
// CloudWatch Policy | ||||||
const cloudwatchPolicy = new aws.iam.RolePolicy(`${projectName}-cloudwatch-policy`, { | ||||||
role: zookeeperRole.name, | ||||||
policy: JSON.stringify({ | ||||||
Version: "2012-10-17", | ||||||
Statement: [{ | ||||||
Effect: "Allow", | ||||||
Action: [ | ||||||
"cloudwatch:PutMetricData", | ||||||
"cloudwatch:GetMetricData", | ||||||
"cloudwatch:ListMetrics", | ||||||
], | ||||||
Resource: "*", | ||||||
}], | ||||||
}), | ||||||
}); | ||||||
|
||||||
// Launch Template User Data Script | ||||||
const getUserData = (id: number) => `#!/bin/bash | ||||||
apt-get update | ||||||
apt-get install -y openjdk-11-jdk | ||||||
|
||||||
# Install ZooKeeper | ||||||
ZOOKEEPER_VERSION="3.9.3" | ||||||
wget https://dlcdn.apache.org/zookeeper/zookeeper-$ZOOKEEPER_VERSION/apache-zookeeper-$ZOOKEEPER_VERSION-bin.tar.gz | ||||||
tar -xzf apache-zookeeper-$ZOOKEEPER_VERSION-bin.tar.gz | ||||||
mv apache-zookeeper-$ZOOKEEPER_VERSION-bin /opt/zookeeper | ||||||
|
||||||
# Configure ZooKeeper | ||||||
cat > /opt/zookeeper/conf/zoo.cfg << EOF | ||||||
tickTime=2000 | ||||||
initLimit=10 | ||||||
syncLimit=5 | ||||||
dataDir=/var/lib/zookeeper | ||||||
clientPort=2181 | ||||||
server.1=zk1.internal:2888:3888 | ||||||
server.2=zk2.internal:2888:3888 | ||||||
server.3=zk3.internal:2888:3888 | ||||||
EOF | ||||||
|
||||||
mkdir -p /var/lib/zookeeper | ||||||
echo "${id}" > /var/lib/zookeeper/myid | ||||||
|
||||||
# Create systemd service | ||||||
cat > /etc/systemd/system/zookeeper.service << EOF | ||||||
[Unit] | ||||||
Description=ZooKeeper Service | ||||||
After=network.target | ||||||
|
||||||
[Service] | ||||||
Type=forking | ||||||
User=root | ||||||
Group=root | ||||||
ExecStart=/opt/zookeeper/bin/zkServer.sh start | ||||||
ExecStop=/opt/zookeeper/bin/zkServer.sh stop | ||||||
ExecReload=/opt/zookeeper/bin/zkServer.sh restart | ||||||
WorkingDirectory=/opt/zookeeper | ||||||
|
||||||
[Install] | ||||||
WantedBy=multi-user.target | ||||||
EOF | ||||||
|
||||||
# Start ZooKeeper | ||||||
systemctl daemon-reload | ||||||
systemctl enable zookeeper | ||||||
systemctl start zookeeper | ||||||
|
||||||
# Install CloudWatch agent | ||||||
wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb | ||||||
dpkg -i amazon-cloudwatch-agent.deb | ||||||
|
||||||
# Configure CloudWatch agent | ||||||
cat > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json << EOF | ||||||
{ | ||||||
"metrics": { | ||||||
"metrics_collected": { | ||||||
"mem": { | ||||||
"measurement": ["mem_used_percent"] | ||||||
}, | ||||||
"disk": { | ||||||
"measurement": ["disk_used_percent"], | ||||||
"resources": ["/"] | ||||||
} | ||||||
} | ||||||
} | ||||||
} | ||||||
EOF | ||||||
|
||||||
systemctl enable amazon-cloudwatch-agent | ||||||
systemctl start amazon-cloudwatch-agent`; | ||||||
|
||||||
// Launch Template | ||||||
const launchTemplate = new aws.ec2.LaunchTemplate(`${projectName}-launch-template`, { | ||||||
namePrefix: `${projectName}-`, | ||||||
imageId: "ami-0c7217cdde317cfec", // Ubuntu 24.04 LTS AMI ID | ||||||
instanceType: "t3.medium", | ||||||
userData: Buffer.from(getUserData(1)).toString("base64"), | ||||||
vpcSecurityGroupIds: [zookeeperSG.id], | ||||||
iamInstanceProfile: { | ||||||
name: zookeeperInstanceProfile.name, | ||||||
}, | ||||||
blockDeviceMappings: [{ | ||||||
deviceName: "/dev/sda1", | ||||||
ebs: { | ||||||
volumeSize: 50, | ||||||
volumeType: "gp3", | ||||||
deleteOnTermination: "true", | ||||||
encrypted: "true", | ||||||
}, | ||||||
}], | ||||||
tags: { | ||||||
Name: `${projectName}-launch-template`, | ||||||
Environment: environment, | ||||||
}, | ||||||
metadataOptions: { | ||||||
httpEndpoint: "enabled", | ||||||
httpTokens: "required", | ||||||
httpPutResponseHopLimit: 1, | ||||||
}, | ||||||
}); | ||||||
|
||||||
// Auto Scaling Group | ||||||
const asg = new aws.autoscaling.Group(`${projectName}-asg`, { | ||||||
vpcZoneIdentifiers: vpc.privateSubnetIds, | ||||||
desiredCapacity: 3, | ||||||
maxSize: 3, | ||||||
minSize: 3, | ||||||
healthCheckType: "ELB", | ||||||
healthCheckGracePeriod: 300, | ||||||
launchTemplate: { | ||||||
id: launchTemplate.id, | ||||||
version: "$Latest", | ||||||
}, | ||||||
tags: [{ | ||||||
key: "Name", | ||||||
value: `${projectName}-node`, | ||||||
propagateAtLaunch: true, | ||||||
}, { | ||||||
key: "Environment", | ||||||
value: environment, | ||||||
propagateAtLaunch: true, | ||||||
}], | ||||||
}); | ||||||
|
||||||
// Application Load Balancer | ||||||
const alb = new aws.lb.LoadBalancer(`${projectName}-alb`, { | ||||||
internal: true, | ||||||
loadBalancerType: "application", | ||||||
securityGroups: [zookeeperSG.id], | ||||||
subnets: vpc.privateSubnetIds, | ||||||
tags: { | ||||||
Name: `${projectName}-alb`, | ||||||
Environment: environment, | ||||||
}, | ||||||
}); | ||||||
|
||||||
// Target Group | ||||||
const targetGroup = new aws.lb.TargetGroup(`${projectName}-tg`, { | ||||||
port: 2181, | ||||||
protocol: "HTTP", | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ZooKeeper typically uses TCP rather than HTTP. If the service does not support HTTP, update the target group protocol to 'TCP' or adjust the configuration accordingly.
Suggested change
Copilot uses AI. Check for mistakes. Positive FeedbackNegative Feedback |
||||||
vpcId: vpc.vpcId, | ||||||
targetType: "instance", | ||||||
healthCheck: { | ||||||
enabled: true, | ||||||
path: "/", | ||||||
port: "2181", | ||||||
protocol: "HTTP", | ||||||
healthyThreshold: 2, | ||||||
unhealthyThreshold: 3, | ||||||
timeout: 5, | ||||||
interval: 30, | ||||||
}, | ||||||
tags: { | ||||||
Name: `${projectName}-tg`, | ||||||
Environment: environment, | ||||||
}, | ||||||
}); | ||||||
|
||||||
// ALB Listener | ||||||
const listener = new aws.lb.Listener(`${projectName}-listener`, { | ||||||
loadBalancerArn: alb.arn, | ||||||
port: 2181, | ||||||
protocol: "HTTP", | ||||||
defaultActions: [{ | ||||||
type: "forward", | ||||||
targetGroupArn: targetGroup.arn, | ||||||
}], | ||||||
}); | ||||||
|
||||||
// Attach ASG to Target Group | ||||||
const asgAttachment = new aws.autoscaling.Attachment(`${projectName}-asg-attachment`, { | ||||||
autoscalingGroupName: asg.name, | ||||||
lbTargetGroupArn: targetGroup.arn, | ||||||
}); | ||||||
|
||||||
// CloudWatch Alarms | ||||||
const cpuAlarm = new aws.cloudwatch.MetricAlarm(`${projectName}-cpu-alarm`, { | ||||||
comparisonOperator: "GreaterThanThreshold", | ||||||
evaluationPeriods: 2, | ||||||
metricName: "CPUUtilization", | ||||||
namespace: "AWS/EC2", | ||||||
period: 300, | ||||||
statistic: "Average", | ||||||
threshold: 80, | ||||||
alarmDescription: "This metric monitors ec2 cpu utilization", | ||||||
dimensions: { | ||||||
AutoScalingGroupName: asg.name, | ||||||
}, | ||||||
}); | ||||||
|
||||||
// Export values | ||||||
export const vpcId = vpc.vpcId; | ||||||
export const albDnsName = alb.dnsName; | ||||||
export const asgName = asg.name; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
{ | ||
"name": "aws-ts-zookeeper", | ||
"description": "Zookeeper on AWS with managed node group, launch template, and load balancer", | ||
"dependencies": { | ||
"@pulumi/pulumi": "3.145.0", | ||
"@pulumi/aws": "6.66.3", | ||
"@pulumi/awsx": "2.19.0" | ||
} | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
{ | ||
"compilerOptions": { | ||
"strict": true, | ||
"outDir": "bin", | ||
"target": "es2016", | ||
"module": "commonjs", | ||
"moduleResolution": "node", | ||
"sourceMap": true, | ||
"experimentalDecorators": true, | ||
"pretty": true, | ||
"noFallthroughCasesInSwitch": true, | ||
"noImplicitReturns": true, | ||
"forceConsistentCasingInFileNames": true | ||
}, | ||
"files": [ | ||
"index.ts" | ||
] | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The 'encrypted' property for the EBS volume is set as a string instead of a boolean. Consider updating it to use a boolean value (true) to match the expected type.
Copilot uses AI. Check for mistakes.