Project Pythia at AGU

[kmpaul]

Project Pythia will have a presence at AGU 2021! Please provide your input here on how best we can present Project Pythia at AGU.

NS013: Open-source software for near-surface geophysics and its applications

@kmpaul has been invited to present about Project Pythia at the AGU session NS013 ("Open-source software for near-surface geophysics and its applications"). Kevin is currently working on an abstract for that session, but the thinking is that it will be essentially a "Project Pythia Overview" talk with modifications to target it to the solid-earth geoscientists in the audience. Once a draft for that session abstract is developed, it will be shared here.

IN030: Open-Source Software, Notebooks and FAIR Software

In our most recent Project Pythia meeting, we also discussed having @kmpaul (since he will be attending AGU) submit an abstract to session IN030 ("Open-Source Software, Notebooks and FAIR Software"). This would be a Project Pythia overview presentation. Here is a draft of a possible abstract, very similar to the abstract @brian-rose gave for recent Pangeo showcase:

Project Pythia is an NSF-funded EarthCube project to create a "go-to" resource for geoscientists to learn the ins and outs of the myriad technologies that comprise the scientific Python ecosystem. Partnered closely with the Pangeo community, Project Pythia aims to make it easier for geoscientists at all career levels to adopt and contribute back to the Pangeo scientific Python software stack. In this talk, I will give an overview of the current state of the project -- including the Project Pythia Portal and the Project Pythia Foundations Book, plans for the future, and how you can get involved.

What do people think?

[kmpaul]

I received confirmation from Lindsey Heagy that the IN030 session would be a great fit for a Pythia overview talk.

[brian-rose]

Great! I say go ahead with the submission to IN030. The draft abstract looks good to me.

[kmpaul]

Who should be listed as a co-author on this submission? @ProjectPythia/core @ProjectPythia/education @ProjectPythia/infrastructure

[kmpaul]

Incidentally, I've received feedback from Lindsey Heagy about the NS013 session, and it sounds like they would like a little bit of an "introduction to Pangeo", too. So, I will tailor an abstract for that session to give a bit of an introduction to Pangeo and Pythia and how the two fit together.

[kmpaul]

For NS013: "Open-source software for near-surface geophysics and its applications"

Based on feedback from Lindsey Heagy, here's what I now know about the invited talk for the near-surface geophysics community:

• "What are some of the tipping points you have seen in the Pangeo / Pythia communities that have driven the adoption of cloud computing for research and education? Any comments you have on when HPC is well suited for a problem and when cloud is suited for a problem would also be valuable."
  • Possible Answers:
    1. When the data gets "too heavy to move," doing your analysis "next to the data" is key. (Cite extended "distributed computer memory hierarchy" chart) Global climate and high-resolution meteorology data are representative of this problem and require "proximate analysis." Pangeo technology is scalable.
    2. Cloud computing offers a robust solution for both sharing data as well as services, something that local computing (including private clusters and HPC) just can't provide.
    3. Cloud computing (and increasingly HPC as well) is based on containerized technology and workflows, which makes it incredibly easy to deploy and tear down computing instances, such as for training, tutorials, classes, etc. (This also makes it possible for cloud computing environments to be elastic, and scale out only as far as you need when you need it.)
  • Other answers?
• Lindsey also said that the near-surface geophysics community uses Jupyter a lot, but they don't use Xarray, Dask, and Zarr. So, an overview of these technologies and what they are good for would be good.
  • I think the answer to this is straightforward: Xarray is "Pandas for N-dimensional arrays." Dask provides distributed scalability "under the hood" in a way that is flexible for different kinds of analyses. And Zarr provides a storage format that is more cloud friendly than HDF5.
• Lastly, Lindsey said that the number of Python packages in the near-surface geophysics community is growing rapidly, and that lessons on how Pangeo has worked to support and grow development would be appreciated.
  • I think the answer to this is to describe how Pangeo advocates and supports the community move to *open development." And so a slide on the "What/Why/How" of open development" would be good.

What do people think about these answers?

[dcamron]

Maybe less relevant for the abstract in particular, but I really like the initial question and am curious to hear about people's experiences around "tipping points" they've encountered. Are there any large-scale problems, projects, or needs where any of us had to bring in a new technology or capability to be able to move forward? Or where a leap in technology was required to even begin a new undertaking? I think a collection of experiences like this could be valuable to some communities or people looking to advocate for these sorts of shifts in technology.

[kmpaul]

Considering the above, here's a...

Draft Abstract for NS013

Title: Pangeo and Project Pythia: Helping Geoscientists Navigate the Scientific Python Ecosystem

Abstract:

Pangeo is a community of academic scientists and industry software engineers partnered to "foster collaboration around the open source scientific Python ecosystem for ocean, atmosphere, land, and climate science." While Pangeo is probably best known for its scalable software stack built on Jupyter, Xarray, and Dask, the Pangeo community's (arguably) greatest contribution is its support for and advocacy of open development practices and reproducible science. To aid in Pangeo's efforts to educate scientists and advance open science, Project Pythia was formed to created a "go-to" resource to help geoscientists navigate and learn the complex scientific Python ecosystem and to help train scientists in the best practices necessary to foster open science and open development. One of Project Pythia's most recent efforts is to build a Foundations Book, covering the basic technology and knowledge necessary to wade into the complex scientific Python ecosystem. In this presentation, I will introduce the audience to Pangeo, its software stack, and the current state of Project Pythia. I will attempt to cover how Pangeo can be useful to other communities, as well as how you can get involved.

Funding for Pangeo and Project Pythia both come from NSF EarthCube awards 1740633, 1740648, 2026863, and 2026899.

[kmpaul]

Who wants to be a co-author on this? Please just give a 👍 reaction to this post if you do. @ProjectPythia/core @ProjectPythia/education @ProjectPythia/infrastructure

[kmpaul]

@ProjectPythia/core @ProjectPythia/education @ProjectPythia/infrastructure

Just a reminder to everyone that I need to submit these abstracts by tomorrow (Aug 4, 2021), end of the business day (mountain time). I am assuming that the co-authors on these presentations will be:

• Anderson Banihirwe
• Drew Camron
• John Clyne
• Orhan Eroglu
• Max Grover
• Julia Kent
• Ryan May
• Brian Rose
• Kevin Tyle

However, I don't know if anyone wants someone else listed, too. Please let me know before tomorrow!

[jukent]

Similarly I will be addressing Project Pythia in an AGU poster that is ESDS educational efforts centric. Here is the abstract: https://docs.google.com/document/d/1_n0_-cV5l7U1fjV5BRSAWxOZP9t1hOKdq1VnFVN04GA/edit?usp=sharing

I will use the same co-authors list as @kmpaul. Let me know if you have any comments or do/don't want to be listed.

---

The NCAR Earth System Data Science Initiative (ESDS) aims to build a socio-technical network to support geoscientists by addressing common data storage or analysis problems. ESDS has four main goals: to facilitate the development of diagnostic frameworks, to improve coordination on analysis workflows, to explore science at scale, and to develop and curate educational materials. This poster will detail the components of ESDS’s educational initiative.

Two teams within ESDS, Xdev and GeoCAT, provide development and software engineering support to the community. Xdev’s goal is to make curiosity-driven analysis of big scientific data a day-to-day reality. This involves prototyping, promoting, and working with scientists to develop new technologies. GeoCAT (Geoscience Community Analysis Toolkit) was born from NCL’s Pivot-to-Python with the goal of supporting scientists making the transition to the Python Scientific Ecosystem from NCL. GeoCAT provides NCL related tools such as an example plotting gallery of the caliber of the former NCL gallery. Both teams share a common vision through their collaboration within ESDS and work closely together and with scientists in the Pangeo community to provide learning resources.

Synchronous learning resources include live events such as the Xdev Python Tutorial Seminar Series and the Xdev Python Office Hours. The Seminar Series covers topics from Python foundations to modules such as Xarray and Dask. Office Hours provide 1-on-1 assistance to geoscientists that may evolve into example Jupyter notebooks of common analysis workflows.

Asynchronous learning resources include the GeoCAT examples gallery and Project Pythia. Project Pythia is designed to be the destination for Python learning materials for geoscientists, consisting of the Pythia Portal which includes links to selected Python Learning Resources, and the Pythia Foundations Book which is a Geoscience-centric Python course. The Tutorial Seminar Series is also designed to be followed asynchronously through tutorial recordings, content hosting, and F.A.Q.s..

This poster will expand upon these resources including information on their metrics, how you could contribute to these efforts, and how your team could use them to the benefit of furthering your curiosity-driven scientific analysis.

[erogluorhan]

We have also referred to Project Pythia within our overarching GeoCAT abstract as follows:

"The GeoCAT team is a part of Project Pythia, a National Science Foundation Earth Cube grant awardee. Project Pythia focuses on providing a public, web-accessible training resource for earth scientists to more effectively use both the Scientific Python Ecosystem and cloud computing to process huge volumes of numerical scientific data. Additionally, the GeoCAT team provides support through GitHub and through cross-organizational collaborations."

It's submitted as a presentation abstract, and if it gets accepted, we will be getting inputs about how to shortly but effectively present Project Pythia in our slides.

[kmpaul]

I have submitted both the IN030 and the NS013 abstracts.

[kmpaul]

Here is a link to the content for the AGU "Open Source Software, Notebooks, and FAIR Software" session poster.

https://docs.google.com/document/d/1sXX0EZxtrqFQqcPeEywF5RgOsXJWJix4-XrDNQjbAWo/edit?usp=sharing

Let me know what you think. I will add this content into one of AGU's iPoster templates to create an "interactive poster" for the conference session.

[clyne]

Love it! Great work!

[brian-rose]

Looks terrific!

Just one niggling comment: can you list my (and @ktyle's) affiliation as "University at Albany (SUNY)". And, if possible at this stage, I think it would be appropriate to add @r-ford to the author list.

[kmpaul]

Thanks. Sorry I keep getting the institution name wrong.

[kmpaul]

Here is a link to the AGU "Open-Source Software for Near-Surface Geophysics and Its Applications" session poster on Pangeo and Pythia:

https://docs.google.com/document/d/1j9CShwdsAaNi-iAzO3wDoSWFbCAhSjFaBsRmwTuEUM8/edit?usp=sharing

Let me know what you think. (Same deal with the AGU iPosters.)

[ktyle]

@kmpaul regarding the Pangeo/Pythia iPoster, can you add the Pythia Twitter link since you have it for Pangeo?

[ktyle]

For the Project Pythia iPoster:

Introduction panel:

1. "parent's" --> "parents'"
2. "have scares" --> "have scared"

The Platform panel:

1. "we our goal" --> "our goal"

Get Involved panel:

1. As with the Pangeo/Pythia iPoster, add the Pythia Twitter link.

Great job on both iPosters ... love the changing graphics!

[kmpaul]

Thank you!

[kmpaul]

@ktyle: All changes have been made.

[brian-rose]

@kmpaul both posters look great and read well! I really have nothing to add. Great use of the i-poster format (my device keeps autocorrecting this to imposter, ha ha)..

[kmpaul]

Thanks for the feedback, everyone! The posters have now been "published." I can make changes until Dec 31 (I think), but the posters are published in preparation for the conference.

[clyne]

Hi All,

I've been asked to give a ~4 minute talk on Project Pythia at the EarthCube Town Hall. I'd appreciate any feedback on my slides. I'll have to upload them late tomorrow. Sorry for the short fuse!

EarthCube Town Hall at AGU
Friday, 17 December 2021
11:15 - 12:15 CST
Convention Center - Room 352

[brian-rose]

Looks great for a pithy talk! I had the same nits and @ktyle about the bullets on slide 2.

[jukent]

Very pithy Pythia slides! I think these look great John, good luck.

…

On Wed, Dec 8, 2021 at 3:11 PM clyne ***@***.***> wrote: Hi All, I've been asked to give a ~4 minute talk on Project Pythia at the EarthCube Town Hall. I'd appreciate any feedback on my slides <https://docs.google.com/presentation/d/1xUPo2gsIGASAEdi4J5satTWWtu-E6ZFqQgf9Ml8GhHc/edit?usp=sharing>. I'll have to upload them late tomorrow. Sorry for the short fuse! EarthCube Town Hall at AGU Friday, 17 December 2021 11:15 - 12:15 CST Convention Center - Room 352 — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALEGIOYUZH3YD5ZHRSZVLGTUP7JZ3ANCNFSM5A2S3BRA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- *Julia Kent* Visualization and Analysis Software Team (VAST) Experimental Development (Xdev) Software Engineer II Phone: x2488 Office: ML494B

[ktyle]

@clyne On slide 2, is there a reason why the final bullet, Using Cloud resources is in bold? Also I see a speaker icon at the bottom right of that slide. Finally, if only because I'm pretty focused on Git / GitHub these days, the "H" in GitHub is typically capitalized. Other than these minor nitpicks it all looks good!

[clyne]

Thanks for catching these. I've addressed them.