'Generic Ensemble Forecast Reader' in LIS not working

[prakrutkansara]

Hello,

I am trying to use the 'GenEnsFcst' met forcing source to process downscaled GEOS5 forecast data for running LIS in retrospective mode on Discover. It stops running after about 10 seconds. The lis log does not show any error at the end, but the slurm error file shows 'application called MPI_abort'. I am not able to figure out what's wrong with my config files. I am attaching my LIS config file, lis log file, and slurm error file.

lis_config_template.txt
lislog_2023_0000.txt
error_25985700.txt

Any help would be appreciated! Thank you.

[karsenau]

Hi @prakrutkansara:

I tried to look at the files that you provided and noticed a few things. However, you reported that your run stopped with an MPI_Abort message in your slurm output file.

A couple of questions:

1. This is your first time running LIS?
2. If so, have you had any success prior with the same ./LIS command line with other LIS config files and runs on Slurm?

BTW - Would you mind sharing what line you submit to slurm for running the LIS executable?

Thanks!

[prakrutkansara]

Hi @karsenau:

1. No, I have been running LIS for over an year now.
2. Yes, I can successfully run LIS with the same config file if change the met source to "GDAS".

Attached is the batch job script I'm using to submit to slurm.

lis_template.txt

[karsenau]

Ok, thank you for your responses in relation to running LIS and that you can successfully run with the same LIS config, if you switch to "GDAS".

One other follow-up question: Has anyone else in your group had success with reading in the following GEOS forcing file:
[INFO] Reading in GenEnsFcst Forcing File:
./outputs/forecast/downscaled_metforcing/FORCING//2023/nov01/ens1/GEOS5.202311.
nc4

Try the following options in your lis.config file:

1. Change your start date from 2023-11-05 to 2023-11-01;
2. Change the Spatial interpolation method for the forcing from "bilinear" to "neighbor"

And see if one of those options get you further along.

BTW - Are you writing the SLURM "log" output files, or just the "error" logs?

[prakrutkansara]

Thank you for your prompt response @karsenau!

One of our previous group member had success in reading GEOS forcing file before for a different domain but the person who did it mentioned that he used the GEOS5 reader instead of Generic Ensemble Forecast Reader. When I tried using the GEOS5 forecast reader, it does not read the customized sub-daily forecasts that I have generated by myself, rather it only looks for the daily forecast files (which I presume is setup for the raw GEOS5 S2S forecasts directly). I'm not sure if GEOS5 reader had changed to Generic Ensemble Forecast Reader, hence I switched to using Generic Ensemble Forecast Reader. Also, the way his scripts were setup and file naming conventions were setup, it seems that they match with the code inside the 'get_genEnsFcst_filename.F90' inside Generic Ensemble Forecast Reader.

I'm not sure if this would be helpful or not but I have also looked at the source code files to find out where lis is stopping. It stops inside the subroutine 'genEnsFcst_Variables_read' in 'genEnsFcst_VariablesMod.F90' at Line 333.

I did try your suggestions of changing the start date to 2023-11-01 and 'neighbor'. Unfortunately, it leads to the same errors.

For SLURM, I am writing both output and error files.

Thank you,
Prakrut

[karsenau]

Hi @prakrutkansara:

Yes, that is correct -- there is another GEOS5 (version 1, hourly data) reader that was used awhile ago but the data are no longer produced by NASA's GMAO. That reader in LIS can be found in our documentation, if you look in our lis.config.adoc file:

lis/configs/lis.config.adoc
https://github.com/NASA-LIS/LISF/blob/master/lis/configs/lis.config.adoc

Under 'Met forcing sources':
You will see the option: "GEOS5 forecast"

For the "GenEnsFcst" met forcing option, there is a particular directory path and netcdf file convention that is expected. When I looked at the order of the directory structure in your lislog file, for example:
[INFO] Reading in GenEnsFcst Forcing File:
./outputs/forecast/downscaled_metforcing/FORCING//2023/nov01/ens1/GEOS5.202311.nc4

I believe the initial month, "nov01" subdirectory should be in front of the year, "2023".

Also, make sure you know that the forecast input forcing domain covers your run domain and the initial time point in that above netcdf file is recognized by the reader.

Thank you for sharing what routine that you may see stopping during the run. You could put a print statement in that routine to make sure that you are getting past the "opening" of the file.

Try to run your current setup within an interactive node, and not to a slurm queue, and see if you see any of the "LIS_verify" error messages show up at as printed to the screen. Do you see any of those messages?

[prakrutkansara]

Hi @karsenau:

Thank you for all your help on troubleshooting the issue. I have resolved it now. The issue was that my LIS run spatial resolution (0.05°) was different from the metforcing resolution (0.125°) which is why I needed a buffer of met forcing data outside the domain boundaries.

For the "GenEnsFcst" forcing option, the correct directory structure still was: './outputs/forecast/downscaled_metforcing/FORCING//2023/nov01/ens1/GEOS5.202311.nc4' with '2023' in front of 'nov01'. In addition to it, one other thing to note is even though the forecast initialization was on Nov 2, the directory name had to be 'nov01' in order for the Generic Ensemble forecast to be able to read it. I looked at the source code for Generic Ensemble Forecast Reader and I could confirm that the month directory name has to be 'mmm01'. I could still use the start day of my LIS run as Nov 2 in lis config file.

There was one more error I faced. Here's more information for anyone else who is trying to use Generic Ensemble Forecast Reader: I was following the template 'lis.config' from the GitHub master branch: 'lis/testcases/metforcing/geos5forecast/lis.config'. But for the 'Met forcing sources', I used 'GenEnsFcst' instead of 'GEOS5 forecast'. It gave me error - 'forrtl : severe(151) allocatable array is already allocated'. In order to fix it, I had to change the 'Blending method for forcings' from 'ensemble' to 'overlay'. @karsenau Please correct if I'm wrong.

[karsenau]

Hi @prakrutkansara:

I'm glad you were able to figure out some of the issues that you encountered last month with the ensemble forcing reader options.

1. Yes, the resolution will matter when reading in your generated ensemble forecast forcing files.

2. In later versions of the LISF code, the order has now become initial forecast month, "mmm01", then year, "yyyy". So note to other users using the latest version of the code. Yes, we currently have the "01" date in the "mmm01" convention, but that feature has also been recently updated in recent years to allow the user to specify the initial year-month-date of the initial forecast, e.g.,:

Generic ensemble forecast initial date: 20230501

3). That is correct - you can run the "Blending method for forcings" with "overlay" vs. "ensemble" to allow for different precipitation or other forecast variables (e.g., bias-corrected SWdown) to be overlaid on a "base" set of forcings (e.g., GEOS or other models).