From ce2ccce25d1672546a4c3f440359cba631dd1409 Mon Sep 17 00:00:00 2001 From: tpeterka Date: Mon, 5 Feb 2024 15:04:27 -0600 Subject: [PATCH 1/2] Update extreme_scale_workflows.md --- _projects/extreme_scale_workflows.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/_projects/extreme_scale_workflows.md b/_projects/extreme_scale_workflows.md index 8c23a1a2..5f508b57 100644 --- a/_projects/extreme_scale_workflows.md +++ b/_projects/extreme_scale_workflows.md @@ -239,6 +239,16 @@ Today's science campaigns consist of multiple tasks with wide-ranging data and c We have published our results in 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS) {% cite yildiz2021dynamic --file jlesc.bib %}. In 2022, we will continue to apply our approach, dynamic heterogeneous in situ workflows, to several science use cases including elastic distributed deep learning ones. This is a joint project between Orcun Yildiz, Bogdan Nicolae and Tom Peterka of ANL, and we are actively looking for students to help us with this project. +## Results for 2022/2023 + +### LowFive: In Situ Data Transport for High-Performance Workflows + +* Tom Peterka, Orcun Yildiz, and Bogdan Nicolae of ANL + +We describe LowFive, a new data transport layer based on the HDF5 data model, for in situ workflows. Executables using LowFive can communicate in situ (using in-memory data and MPI message passing), reading and writing traditional HDF5 files to physical storage, and combining the two modes. Minimal and often no source-code modification is needed for programs that already use HDF5. LowFive maintains deep copies or shallow references of datasets, configurable by the user. More than one task can produce (write) data, and more than one task can consume (read) data, accommodating fan-in and fan-out in the workflow task graph. LowFive supports data redistribution from n producer processes to m consumer processes. We demonstrate the above features in a series of experiments featuring both synthetic benchmarks as well as a representative use case from a scientific workflow, and we also compare with other data transport solutions in the literature. + +We have published our results in 2023 International Parallel and Distributed Processing Symposium (IPDPS). + ## Visits and meetings * Pierre-Louis Guhur of ENS 9 months at ANL in 2016 From 8ccfd831dc1a996b29fe9c322847759ceca115f6 Mon Sep 17 00:00:00 2001 From: Robert Speck Date: Tue, 6 Feb 2024 10:32:10 +0100 Subject: [PATCH 2/2] Update extreme_scale_workflows.md --- _projects/extreme_scale_workflows.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_projects/extreme_scale_workflows.md b/_projects/extreme_scale_workflows.md index 5f508b57..976cdfc4 100644 --- a/_projects/extreme_scale_workflows.md +++ b/_projects/extreme_scale_workflows.md @@ -239,7 +239,7 @@ Today's science campaigns consist of multiple tasks with wide-ranging data and c We have published our results in 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS) {% cite yildiz2021dynamic --file jlesc.bib %}. In 2022, we will continue to apply our approach, dynamic heterogeneous in situ workflows, to several science use cases including elastic distributed deep learning ones. This is a joint project between Orcun Yildiz, Bogdan Nicolae and Tom Peterka of ANL, and we are actively looking for students to help us with this project. -## Results for 2022/2023 +## Results for 2023/2024 ### LowFive: In Situ Data Transport for High-Performance Workflows