Introduction of package dependency on Parsl

[christinaholtNOAA]

Parsl is a 3rd-party Python package that NOAA GSL via the SENA project has been investigating for use in distributed workflows. It is primarily a workflow manager, but has many subpackages that can be used in standalone ways consistent with our plans for UW. To be clear, I am not suggesting we adopt Parsl as a replacement to our workflow managers.

As we have become more familiar with it at GSL, it seems like a clear winner for several of the tools we already had in our pipeline, but with a significant head start. Here are the key tools I think we can accomplish full support for more quickly:

• Data movement tool that supports disk transfer, ftp, globus, http, and rsync
• Scheduler interface support, including slurm, torque, pbspro, and lsf
• Launcher interfaces that support mpirun, srun, mpiexec, etc.

We've had several chats with the maintainers of this package, and they seem to be checking the lists for what we'd like to see when we adopt a 3rd party package:

• Well-established community using the package, and an annual user workshop
• Funding secured for long-term maintenance and support (10 years out at this point)
• Community help via email and open Slack
• Encouragement of adoption and of community contributions to the code base

Let's discuss the adoption of this 3rd party package here. Here are some questions to get the ball rolling:

• Are there additional requirements that Parsl, as a project, should meet?
• What are the next steps?
• When do we involve WCOSS admins on the vetting process?

[WeirAE]

I'm excited, I think Parsl will provide a lot of flexibility. The extra requirement is that if we incorporate it without making it a replacement, we'll still need alternative tools, as well as some kind of manager selector that populates the various tools with the preferred manager. I don't think we need to bring WCOSS in until we've at least tested it ourselves and have decided to really incorporate it.

[jprestop]

@WeirAE and @christinaholtNOAA

I don't think we need to bring WCOSS in until we've at least tested it ourselves and have decided to really incorporate it.

Our METplus team has recently been talking a lot with NCO about the Python packages required by our products. Because we want our products to be used operationally on WCOSS2, we develop our code with the version of Python that is implemented on WCOSS2. We have been asked to remove Python packages that we put in because, in the end, they weren't accepted on WCOSS2. Thankfully, we're down to one library that is being discussed for approval at this time and the others we were able to successfully remove. In our recent Release Retrospective with EMC, for example, they stated they would like our team to not develop with any packages until they are approve for operational use on WCOSS2. We weren't incredibly familiar with the approval process, so we recently (this week) wrote to NCO to ask more formally about what the process actually is.

Here's what we were told (summarized):
To make a request, you would fill out this form and then forward the google bounce back with the answers to the wcoss2 helpdesk email: https://docs.google.com/forms/d/1k4ynHAZ07_H6QPlV2JDT3wsq_LavyPDDmmH1J7_IJVQ From there, it goes into Steven Earle's teams queue. The timing, unfortunately, is variable. It depends on how long the queue is and depends how busy they are with failures (WCOSS2 hasn't been very stable for the last couple months and there have been a lot of failures). All this adds up to variability that they can't really quantify. After Steven Earle's team, it gets reviewed by other branches in NCO and finally by the NCEP security team. Typically, with a small backlog like they have now, they say 1-2 months for final approval. A lot of this also depends on the back and forth they have with the submitter; clear implementation instructions, for example, can speed up the process.

That being said, I highly encourage you to start the WCOSS2 approval process as soon as possible, if having your code on WCOSS2 is important to you, so that you don't end up in a similar situation that we have been in, where you potentially have to redo things in a different way. I'm happy to talk further with folks about our experience if you'd like and happy to answer any questions you have.

[arunchawla-NOAA]

WCOSS2 should be brought in earlier than later. But first we need to address what value the package brings to workflow development

[christinaholtNOAA]

For workflow development, I think that parsl is a strong package that alleviates our need to build and support code that provides the interfaces between a variety of platforms and commonly used schedulers and parallelization tools. It provides a nice way to construct a workflow using tools that we had planned to develop anyway, but removes that need. And it has been developed by a team of highly knowledgeable and experienced software teams to meet a variety of workflow needs -- it isn't specific to NWP or a given UFS App.

These are exactly the types of benefits I hope to see when choosing a 3rd party library. Especially the part about how close it is to the tools we need and had already planned to build. I'd always prefer to adopt a well-supported, well-tested library over building and supporting a product that does the same thing in a much smaller scope. This leaves room for expansion and growth in the research community over time. For example, I'd never even considered an interface for Globus transfers, and this capability is already available.

[arunchawla-NOAA]

Does Parsl itself have other packages that are dependent on it?

[christinaholtNOAA]

Yes, of course. There is a list distributed with the code base here: https://github.com/Parsl/parsl/blob/master/requirements.txt

They've also included a mechanism in their setup.py to have additional package dependencies for certain platforms. I believe the requirements.txt are the minimum requirements, though.

[arunchawla-NOAA]

An important question is that if we are not using Parsl for its workflow manager capabilities, which is its predominant role then why would NCO agree to have yet another workflow manager added to the list of supported libraries ?

[christinaholtNOAA]

I think its predominant role for UW will be in its capabilities to help us do the workflow tasks that are very common, but that we do not have supported and unified tools to handle. It will be the package that allows us to more universally handle job card creation and launcher interfaces.

I believe these can both be extremely useful for NCO normalize how these tasks are handled across Apps, but also help with the learning curve during future transitions to new platforms where schedulers and MPI tools may differ.

File movement across platforms being done with the same interface that files move around on disk could also be helpful for NCO...but I'm not entirely clear on how data delivery outside of WCOSS is linked to a given workflow in operations right now. Is that a part of the workflow for an App, or is it handled by a different group and workflow that picks up data when it arrives where it's supposed to on disk?

[christinaholtNOAA]

We decided, in the end, to stick with our own simple interfaces to batch systems and file movement.