Skip to content

Commit

Permalink
PIP-85 AWS Redis Persistence (#93)
Browse files Browse the repository at this point in the history
* PIP-85 rename template to show dep order

* PIP-85 redis template

* PIP-85 enabled redis in the cf template

* PIP-85 document redis template
  • Loading branch information
milt authored Apr 4, 2023
1 parent 5a2972d commit 7984095
Show file tree
Hide file tree
Showing 3 changed files with 135 additions and 17 deletions.
62 changes: 62 additions & 0 deletions dev-resources/templates/0_redis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
AWSTemplateFormatVersion: '2010-09-09'
Description: Deploy Redis persistence for LRSPipe Jobs

Parameters:
# Networking
VPCId:
Description: VPC in which to run Redis
Type: AWS::EC2::VPC::Id
Subnets:
Description: Subnets in which to run the Redis Cluster
Type: List<AWS::EC2::Subnet::Id>

# Provisioning
CacheNodeType:
Description: Type of Elasticache Cluster Instance
Type: String
Default: cache.t3.micro
NumCacheNodes:
Type: Number
Default: 1
EngineVersion:
Type: String
Default: 5.0.6

Resources:
SubnetGroup:
Type: AWS::ElastiCache::SubnetGroup
Properties:
Description: Subnet group for access to redis
SubnetIds: !Ref Subnets
RedisSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group for Redis cache instances
VpcId: !Ref VPCId
RedisCluster:
Type: AWS::ElastiCache::CacheCluster
Properties:
Engine: redis
EngineVersion: !Ref EngineVersion
CacheNodeType: !Ref CacheNodeType
VpcSecurityGroupIds:
- !Ref RedisSecurityGroup
CacheSubnetGroupName: !Ref SubnetGroup
NumCacheNodes: !Ref NumCacheNodes

Outputs:
EndpointAddress:
Description: The redis endpoint
Value: !GetAtt RedisCluster.RedisEndpoint.Address
Export:
Name: !Sub "${AWS::StackName}:EndpointAddress"
EndpointPort:
Description: The redis endpoint port
Value: !GetAtt RedisCluster.RedisEndpoint.Port
Export:
Name: !Sub "${AWS::StackName}:EndpointPort"
SecurityGroupId:
Description: The security group for the Redis cluser
Value: !Ref RedisSecurityGroup
Export:
Name: !Sub "${AWS::StackName}:SecurityGroup"
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
AWSTemplateFormatVersion: '2010-09-09'
Description: 'LRSPipe Application Template'

Parameters:
# Networking
VPCId:
Expand Down Expand Up @@ -38,20 +39,27 @@ Parameters:
Type: String
Default: v0.0.25

# Logging
LogRetentionInDays:
Description: How long to retain Cloudwatch Logs
Type: Number
Default: 7

# Redis Persistence
RedisStackName:
Description: (Optional) Name of Redis stack to use for persistence. If not provided job state will be stored on the instance
Type: String
Default: ''

Conditions:
InstanceKeyNameProvided:
!Not [!Equals [!Ref InstanceKeyName, '']]
RedisStackNameProvided:
!Not [!Equals [!Ref RedisStackName, '']]

Resources:

# Logs

LogGroup:
Type: AWS::Logs::LogGroup
Properties:
Expand All @@ -72,6 +80,19 @@ Resources:
CidrIp: !Ref InstanceSSHCidr
Description: 'ssh from allowed IP'

RedisIngress:
Condition: RedisStackNameProvided
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId:
Fn::ImportValue: !Sub '${RedisStackName}:SecurityGroup'
IpProtocol: tcp
FromPort:
Fn::ImportValue: !Sub '${RedisStackName}:EndpointPort'
ToPort:
Fn::ImportValue: !Sub '${RedisStackName}:EndpointPort'
SourceSecurityGroupId: !Ref PipeInstanceSG

# Policy
InstancePolicy:
Type: AWS::IAM::ManagedPolicy
Expand Down Expand Up @@ -280,19 +301,29 @@ Resources:
owner: root
group: root
'/lib/systemd/system/xapipe.service':
content: !Sub |
[Unit]
Description=LRSPipe Service
[Service]
User=root
WorkingDirectory=/opt/xapipe
ExecStart=/opt/xapipe/bin/run.sh --json-file /etc/xapipe/job-conf.json --file-store-dir /var/lib/xapipe/db
SuccessExitStatus=143
TimeoutStopSec=10
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
content: !Sub
- |
[Unit]
Description=LRSPipe Service
[Service]
User=root
WorkingDirectory=/opt/xapipe
ExecStart=/opt/xapipe/bin/run.sh -f --json-file /etc/xapipe/job-conf.json ${RunArgs}
SuccessExitStatus=143
TimeoutStopSec=10
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
- RunArgs: !If
- RedisStackNameProvided
- !Sub
- '--storage redis --redis-uri redis://${EndpointAddress}:${EndpointPort}'
- EndpointAddress:
Fn::ImportValue: !Sub '${RedisStackName}:EndpointAddress'
EndpointPort:
Fn::ImportValue: !Sub '${RedisStackName}:EndpointPort'
- '--file-store-dir /var/lib/xapipe/db'
mode: '000755'
owner: root
group: root
Expand Down
31 changes: 28 additions & 3 deletions doc/aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,39 @@ __NOTE:__ *This configuration is not one-size-fits-all and you may require a dif

## Deployment

Deploying the CloudFormation Template only requires a few straightforward steps.
### (Optional) Redis Persistence

By default the LRSPipe CloudFormation template stores job state on the local disk of the EC2 instance running the LRSPipe process. If the instance is terminated all job state is lost. If you would like to persist job state outside of the instance an optional template is provided to do this with AWS ElastiCache Redis.

To deploy the Redis template:

- Go to AWS CloudFormation Service
- Choose Create Stack (New Resources)
- Choose 'Template is Ready' / 'Upload a template file'
- Upload the Template `dev-resources/templates/0_redis.yml`
- Click 'Next'
- Choose a stack name and note it down for use in the `RedisStackName` parameter in the next section.

#### Configuration

Configure the Redis template parameters. Leave settings at the default where provided and set the following:

- `Subnets`: Select the AWS VPC subnet(s) in which to run Redis. Note that these must be reachable from the subnet selected for `LrsPipeSubnet` below.
- `VPCId`: Select the AWS VPC in which to run Redis. Make sure this VPC includes the subnet(s) you just selected and matches the `VPCId` you select in the next section.

Now click 'Next' and proceed to deploy the template. When the template deployment is complete, proceed with the next section.

### LRSPipe Process

Deploy the CloudFormation Template for the LRSPipe process:

- Go to AWS CloudFormation Service
- Choose Create Stack (New Resources)
- Choose 'Template is Ready' / 'Upload a template file'
- Upload the Template `dev-resources/templates/lrspipe_ec2.yml`
- Upload the Template `dev-resources/templates/1_lrspipe_ec2.yml`
- Click 'Next'

### Configuration
#### Configuration

![LRSPipe Template Deployment Options](img/template-options.png)

Expand All @@ -31,6 +55,7 @@ On the next page you will be presented with a number of deployment options. We w
- `LrsPipeConfig`: This is where you will paste in your LRSPipe JSON job configuration. This is where all of the actual job configuration takes place. See the [JSON-based Job Config](json.md) page for details and instructions on this step.
- `LrsPipeSubnet`: This field, in conjunction with the `VPCId` determine where in your network the LRSPipe instance is created. This is important as it may impact access to the source and target LRS'. If your LRS' are hosted in the same AWS account make sure this Subnet has access to them. Alternatively if your LRS' are hosted externally, make sure this Subnet has internet access configured.
- `LrsPipeVersion`: This is the version of LRSPipe software you want to deploy. This can be used to upgrade the version on a running instance as well. For a list of LRSPipe versions visit [releases](https://github.com/yetanalytics/xapipe/releases).
- `RedisStackName`: If you are using Redis persistence, provide the CloudFormation stack name of the Redis stack you set up previously. Otherwise, leave this blank.
- `VPCId`: This field controls which Virtual Private Cloud the Instance resides in. Make sure that the chosen VPC contains the Subnet from that previous step.

Now click 'Next' and proceed to deploy the template.
Expand Down

0 comments on commit 7984095

Please sign in to comment.