GREI 5: HDV Task - Large Data Support

[cmbz]

Overview

Support the sharing of very large datasets (>TBs) by integrating the metadata in the repository with the data in the research computing storage" (Source: NIH OTA)

Tasks

• Project: MOC Proof-of-Concept #105
• Proposal: Large Data Support #106
• Service: Harvard Dataverse Repository Large Data Services #268
• Project: Pilot Large Data Support Service #178
  The following original NIH-OTA tasks are part of the Large Data Support proposal and pilot
  • Define a cost recovery model for large dataset support [was Aim 1.1.2]
  • MVP for integration the repository with computation (NERC) [was Aim 2.1.2] (See: Initiative: Computing on Data Support #201 and Project: MOC Proof-of-Concept #105)
  • Expand support for very large datasets through NESE and NERC; implement cost recovery model [was Aim 3.1.1]
  • Assess cost recovery model [was Aim 4.1.1]
• GREI 5 HDV Task: Implement and assess integration and UX/UI of metadata registry for remote storage and computation #243
  • GREI 5 HDV - Task 1: investigate approaches to collect technical/operating requirements for a specific large data/compute use case and configure a large data/compute resources (Year 3) #244
  • GREI 5 HDV - Task 2: Investigate approaches for determining costs of large data/compute services (Year 3)
  • GREI 5 HDV - Task 3: Investigate authentication approaches for specific large data/compute configurations based on use cases (Year 3)
  • GREI 5 HDV - Task 4: Design UI/UX for datasets or collections that use large data/compute resources (Year 3 and/or 4)

Issues

Pending or In Progress

• https://github.com/IQSS/dataverse-HDV-Curation/issues/344

Completed

• Prod. globus infrastructure, next phase: Create documentation for data uploaders dataverse.harvard.edu#260
• Documentation on using Globus for accessing NESE storage in production  dataverse.harvard.edu#268
• Prod. globus infrastructure, next phase: Work out and test the logistics of Globus downloads from NESE dataverse.harvard.edu#261
• IQSS/10450 - Fixes for Globus download (transfer out) and related info messages dataverse#10451

Resources

• Project: NIH GREI - Generalist Repository Ecosystem Initiative #124
• APPROVED - NIH GREI Coopetition Plan Year 3

[cmbz]

Status: January 2024

Several Globus improvements and bug fixes were made to support large data deposits and integration with research computing services such as the Northeast Storage Exchange (NESE).

Completed

• IQSS/10200 - Create dirs needed for Globus transfer to managed stores dataverse#10201
• Globus Transfer in fails for new datasets and non-privileged users dataverse#10200

[cmbz]

Status: February 2024

A containerized Dataverse was deployed on Mass Open Cloud (MOC), using Northeast Storage Exchange (NESE) compute and resources to demonstrate how Dataverse can support computing on large Dataverse datasets stored on NESE tape resources. A demo of the proof-of-concept was presented at the Mass Open Cloud Alliance Conference on 2024/02/28. Work has begun to fully operationalize the strategy in Epic: Operationalize Large Data and Compute Infrastructure.

Completed

• Project: MOC Proof-of-Concept #105

[cmbz]

Status: April 2024

Large Data Support Working Group

• Zulip communications channel for Large Data Support Working Group established
  • Related: Large Data Support Breakout Session, DCM 2024
• Request made for working group page on GDCC website
• Current Working Group Members roster

Large Data Support Pilot

• The Oregon-Massachusetts Mammography Database (OMAMA-DB) was completely uploaded (not yet published). 7.9 TB for the large data portion, and 865GB for the smaller dataset.
  also see https://github.com/IQSS/dataverse-HDV-Curation/issues/443
• In progress is supporting a 16TB project from an MIT affiliate (end of May deadline)
  https://github.com/IQSS/dataverse-HDV-Curation/issues/444

[cmbz]

Status: May 2024

• Followed up on Geos-Chem to inquire about the PDF files in this collection.
• Tested Globus connection and NESE download of one file from OMAMA, successfully
• Demo Large data dataset will be created by @landreev so we can test download by non-Harvard affiliates, to help write instructions for data download
• See other updates and details here: Project: Pilot Large Data Support Service #178 (comment)
• Added two projects to the new section on "under consideration for support" in Pilot Large Data Issue

[cmbz]

Status: July 2024

• Automated reporting of Harvard Dataverse collections that are over their storage limits has been implemented:. See: https://github.com/IQSS/dataverse-HDV-Curation/issues/344

[cmbz]

Status: August 2024

• Harvard Dataverse storage monitoring and reporting has been implemented. Automatic storage usage reports are now available. See: Enable collection storage quota monitoring in HDV  dataverse.harvard.edu#240

[cmbz]

Status: September 2024

• Implemented improved handling of Globus uploads using an experimental asynchronous framework. See:
  • Globus integration: further improvements/refactoring dataverse#10623
  • Improved handling of Globus uploads (experimental async framework) dataverse#10781

[cmbz]

Status: November 2024

• See other issues for updates:
  • Project: Pilot Large Data Support Service #178
  • Service: Harvard Dataverse Repository Large Data Services #268

[cmbz]

Status: December 2024

• Implemented and tested NESE Disk support (for Bertarelli Collection) was completed
  • Test the NESE-disk Globus configuration in production  dataverse.harvard.edu#305