-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
MLPAB-1950 - Build index from DynamoDB export (#1279)
* enable ingestion * add permissions for working with exports * add export config * use bucket name turn off stream provessing use correct index name * allow uploads * create policy before pipeline * update feedback * update config * remove prefix * apply a convention to naming resources * Restrict which index the pipeline can write to * add adr for naming and include keys for pipeline * limit what is put into index * turn off pipeline and stream processing * recreate document id as created by the app * recreate document id as created by the app * Disable app indexing Enable stream processing to index * Update field name * enable pipeline for export and stream processing * build without pipeline * enable pipeline for testing * add some notes for working with dev mode * add instructions for reindexing * add instructions for reindexing * fix readme index * add consequences to adr
- Loading branch information
1 parent
c5a472e
commit 00c1168
Showing
10 changed files
with
253 additions
and
16 deletions.
There are no files selected for viewing
72 changes: 72 additions & 0 deletions
72
docs/architecture/decisions/0005-namespacing-resources-in-aws.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,72 @@ | ||
# 3. Namespacing resources in AWS | ||
|
||
Date: 2024-06-19 | ||
|
||
## Status | ||
|
||
Accepted | ||
|
||
## Context | ||
|
||
This decision is in relation to the name attribute of a resource, not the resource name itself. So for the example below, the name attribute is `event-received-${data.aws_default_tags.current.tags.environment-name}` and the resource name is `event-received`. | ||
|
||
```hcl | ||
resource "aws_iam_role" "event_received" { | ||
name = "event-received-${data.aws_default_tags.current.tags.environment-name}" | ||
... | ||
} | ||
``` | ||
|
||
Making resources in AWS unique is important to avoid conflicts across environments, regions, and accounts. This is especially important when using resources that are shared across multiple environments such as encryption keys. | ||
|
||
Granting access to resources is also easier when they are namespaced because it is easier to identify which resources are being accessed. | ||
|
||
To make granting access to resources easier, we should use a consistent naming convention for resources. | ||
|
||
The values currently used in naming resources are: | ||
|
||
- `environment-name` | ||
- `region` | ||
- `resource-name` | ||
- `account-name` | ||
- `application-name` | ||
|
||
Some examples of namespaced IAM role resources are: | ||
|
||
- `event-received-${data.aws_default_tags.current.tags.environment-name}` | ||
- `${data.aws_default_tags.current.tags.environment-name}-execution-role` | ||
- `${data.aws_default_tags.current.tags.environment-name}-execution-role-${data.aws_region.current.name}` | ||
- `batch-manifests-${data.aws_default_tags.current.tags.application}-${data.aws_default_tags.current.tags.account-name}-${data.aws_region.current.name}/*` | ||
|
||
IAM policies support wildcards in the resource name, so we can use wildcards to grant access to all resources that match a pattern. This is useful when granting access to resources that are created dynamically. | ||
|
||
Adopting a consistent naming convention for resources will make it easier to grant access to resources and avoid conflicts. | ||
|
||
## Decision | ||
|
||
Use a consistent naming convention for resources in AWS. The naming convention should include the following values in this order: | ||
|
||
- `resource-name` describing the role/function of the resource (e.g. `event-received`) | ||
- `application-name` which is the product name (e.g. `opg-modernising-lpa`) | ||
- `account-name` which is the AWS account name (e.g. `development`) | ||
- `region-name` which is the AWS region name (e.g. `eu-west-1`) | ||
- `environment-name` which is the environment name (e.g. `production`) | ||
|
||
It isn't necessary to include resource type in the name because the resource type is already specified in the resource definition. | ||
|
||
`application-name` and `account-name` will be used when a resource name must be globally unique, such as an S3 bucket name. They can be omitted if the resource is not globally unique. | ||
|
||
(this leads to the consequence that application name used in aws_kms_alias is not necessary) | ||
|
||
`account-name` should be used for resources shared at account level. | ||
|
||
`region-name` can be omitted if the resource is not region-specific. | ||
|
||
`environment-name` should be used for resources that are environment-specific. | ||
|
||
(this leads to the consequence that resources will have either `environment-name` or `account-name` and not likely to have both) | ||
|
||
## Consequences | ||
|
||
- Resources will be easier to identify and grant access to | ||
- some resources will need renaming |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
# Rebuilding the Opensearch index | ||
|
||
In the event that the Opensearch index needs to be rebuilt, the following steps should be followed. | ||
|
||
These instructions are for the `test` environment with an index called `lpas_v2_test`. The same steps can be followed for the other environments, with the appropriate index name `lpas_v2_<environment name>`. | ||
|
||
1. Delete the existing index | ||
|
||
```shell | ||
DELETE /lpas_v2_test | ||
``` | ||
|
||
1. Create a new index with the correct mapping | ||
|
||
```shell | ||
PUT /lpas_v2_test | ||
{ | ||
"settings": { | ||
"index": { | ||
"number_of_shards": 1, | ||
"number_of_replicas": 1 | ||
} | ||
}, | ||
"mappings": { | ||
"properties": { | ||
"PK": {"type": "keyword"}, | ||
"SK": {"type": "keyword"}, | ||
"Donor.FirstNames": {"type": "keyword"}, | ||
"Donor.LastName": {"type": "keyword"} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
1. recreate the opensearch pipeline | ||
|
||
We do this by using terraform to taint and recreate the pipeline. | ||
|
||
In a shell, navigate to the `terraform/environment` directory and select the correct workspace: | ||
|
||
```shell | ||
tf workspace select <environment name> | ||
``` | ||
(working with preproduction and production environments requires the breakglass role) | ||
Mark the pipeline for recreation: | ||
```shell | ||
tf taint 'aws_osis_pipeline.lpas_stream[0]' | ||
``` | ||
Then apply the changes: | ||
```shell | ||
tf apply | ||
``` | ||
1. Reindexing | ||
When the pipeline is created, it will trigger a dynamoDB export to S3. Once the export is finished, the pipeline will import the data into index. After the export processing is complete, the pipeline will switch to processing DynamoDB stream events if enabled. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,9 @@ | ||
#!/usr/bin/bash | ||
|
||
source ../../scripts/switch-terraform-version.sh | ||
source ../../scripts/pull-av-scan-zip-packages.sh | ||
export TF_CLI_ARGS_init="-backend-config=role_arn=arn:aws:iam::311462405659:role/operator -upgrade -reconfigure" | ||
export TF_VAR_default_role=operator | ||
export TF_VAR_pagerduty_api_key=$(aws-vault exec mlpa-dev -- aws secretsmanager get-secret-value --secret-id "pagerduty_api_key" | jq -r .'SecretString') | ||
export TF_VAR_container_version=$(aws-vault exec management-global -- aws ssm get-parameter --name "/modernising-lpa/container-version/production" --query 'Parameter.Value' --output text) | ||
echo "Deploying Version: $TF_VAR_container_version" | ||
echo "Deploying Modernising LPA version: $TF_VAR_container_version" | ||
source ../../scripts/pull-av-scan-zip-packages.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.