As a user, I want to store a copy of my archive data in AWS Deep Archive / Glacier

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

Archivist

💪 Motivation

...so that I can have a copy of my data available in the event of corruption or disaster recovery

📖 Additional Details

No response

Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

Tightly coupled with cloud requirement: NASA-PDS/planetary-data-cloud#75

[viviant100]

tagup with MCP - we (PDS) will be able to set up glacier ousevles.

[viviant100]

@ramesh-maddegoda At monthly MCP meeting today (7/8/24), we explored the following options instead of using glaciers:

• Enabling S3 bucket versioning
• Utilizing multi-region S3 buckets

Could you please evaluate these approaches with Nucleus?

[jordanpadams]

@viviant100 poking at this a bit online, I think what the folks on the MCP call may have been referring to was Cross-Region Replication. We could then:

• Enable versioning for easy rollback to a past version in an S3 bucket
• Replicate the data from our S3 archive bucket to another region
• Once the data is replicated to the other region, move it to Deep Archive tier

To kick off this effort, since this does not really need Nucleus in the loop, can you help with a couple sub-tasks:

• Provide a cost comparison of our existing planned costs for a copy of the data in S3 and Glacier vs. S3 + versioning + cross-region replication + Deep Archive (we can use lower storage tier since we have versioning enabled)
• Ask around to other data repositories and archives at JPL to determine what their disaster recovery solutions are for data at rest in the cloud, and whether or not this solution seems sensible or if it is overkill. May be worth reaching out to other like MCP folks and/or AWS SMEs to get insights.

After that, we can pilot out what it looks like to set something like this up.

[ramesh-maddegoda]

Now, in MCP it takes more than 2 hours sometimes to create MWAA in MCP with Terraform and it fails with the following error.

module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env: Still creating... [1h56m25s elapsed]
module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env: Still creating... [1h56m35s elapsed]
module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env: Still creating... [1h56m45s elapsed]
module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env: Still creating... [1h56m55s elapsed]
module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env: Still creating... [1h57m5s elapsed]
module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env: Still creating... [1h57m15s elapsed]
module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env: Still creating... [1h57m25s elapsed]
module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env: Still creating... [1h57m35s elapsed]
module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env: Still creating... [1h57m45s elapsed]
╷
│ Error: waiting for MWAA Environment (pds-nucleus-airflow-env) create: operation error MWAA: GetEnvironment, https response error StatusCode: 403, RequestID: 9c1603bb-8845-4eb1-951b-45623079b703, api error ExpiredTokenException: The security token included in the request is expired
│ 
│   with module.mwaa-env.aws_mwaa_environment.pds_nucleus_airflow_env,
│   on terraform-modules/mwaa-env/mwaa_env.tf line 57, in resource "aws_mwaa_environment" "pds_nucleus_airflow_env":
│   57: resource "aws_mwaa_environment" "pds_nucleus_airflow_env" {
│ 
 
 

Created the request https://jaas.gsfc.nasa.gov/servicedesk/customer/portal/2/GSD-3804 to get long term access keys to avoid this problem.

[ramesh-maddegoda]

This ticket was unblocked after finding a solution to #118

[ramesh-maddegoda]

Created pull request: #119

[jordanpadams]

Status: Initially had an issue with pushing to Glacier. Need to set --force flag in order to overwrite the data in Glacier.

[tloubrieu-jpl]

We finally want a replication across regions and we do that using a replication rule configured in terraform (as done for the ODR bucket).

[ramesh-maddegoda]

Create a pull request #120

[tloubrieu-jpl]

The PR is merged and next step is to document the needed configuration, a ticket will be created for that.

[jordanpadams]

This can be closed. Wrap-up of NASA-PDS/planetary-data-cloud#75 will include the necessary documentation.