Feature/AquaCropv7.2 integration #1633

lbusschaert · 2024-11-05T15:36:48Z

Description

This PR introduces the implementation of a crop model (AquaCropv7.2) in LIS, along with the LDT parameter processing. The implementation was done within the KU Leuven team of Prof. Gabriëlle De Lannoy, mainly by Michel Bechtold and myself.
Research has been done with a preliminary version of AquaCropv7.0 in LIS:

The objective was now to develop a final working version in LIS.

I was visiting the NASA-LIS team in early 2024 and prepared this implementation in collaboration with Sujay Kumar and David Mocko. The implementation is now finalized and has been carefully tested and compared to AquaCrop reference output.

AquaCropv7.2 (ac72) is now part of the surfacemodels/land/. The implementation is documented in
LIS_AC_documentation.pdf

We acknowledge that this is a large PR and we greatly appreciate the time you spend on it. Do not hesitate to contact me in case of any questions ([email protected]).

Testcase

Test cases for LDT and LIS have been added under the ldt/testcases and lis/testcases folders. The data for the test cases are stored in a Zenodo repository organized in the requested format: https://zenodo.org/records/14035685

Testing was done for the generic crop having a 365-day cycle for 4 restart times (1 Feb, 1 Aug, 1 Dec, 1 Jan next sim), as well as for the AquaCrop Maize file planted on March 22nd. Minor differences are supposably introduced by precision differences (saving restart variables as single precision while AC uses doubles) but have no impact on the final biomass, yield, Tact, Eact. Note that the restart is not working for - perennial crops - GDD crops (will be fixed later) - salinity - (may need a more elaborate list somewhere) No irrigation nor groundwater files were used as it has been decided that those options will be forced from LIS. Irrigation will be introduced in the next developments. Groundwater will not be introduced for the time being.

Switched to AquaCrop sp. ac_modules are an exact copy of AC src stored in branch https://github.com/lbusschaert/AquaCrop/tree/LIS/sp_2. WIP: restarting is technically working but needs to be checked via test cases

…tion

dmocko · 2024-11-05T16:39:29Z

Hi @lbusschaert

Congratulations to you and your team for submitting this PR to get AquaCrop fully merged into the LIS code!

I did a quick review, and I have a few initial requests:

Can you please add all of the new config entries, and a brief explanation on them, into the ldt/configs/ldt.config.adoc and lis/configs/lis.config.adoc files?

For example, "AquaCrop reference year for climatology:" for the ldt.config.adoc file, and "AquaCrop.7.2 model timestep:" for the lis.config.adoc file.

Similarly, can you please add the new possible LIS outputs into the lis/configs/MODEL_OUTPUT_LIST.TBL.adoc file?

I see that you have them in your testcase, but it would be great to include them in the primary model output doc.

You checked in an empty user.cfg file, but we actually prefer to not have any user.cfg file in our main branch. Can you please remove this file from the commits?

Thank you.

lbusschaert · 2024-11-06T10:09:32Z

Hi @lbusschaert

Congratulations to you and your team for submitting this PR to get AquaCrop fully merged into the LIS code!

I did a quick review, and I have a few initial requests:

Can you please add all of the new config entries, and a brief explanation on them, into the ldt/configs/ldt.config.adoc and lis/configs/lis.config.adoc files?

For example, "AquaCrop reference year for climatology:" for the ldt.config.adoc file, and "AquaCrop.7.2 model timestep:" for the lis.config.adoc file.

Similarly, can you please add the new possible LIS outputs into the lis/configs/MODEL_OUTPUT_LIST.TBL.adoc file?

I see that you have them in your testcase, but it would be great to include them in the primary model output doc.

You checked in an empty user.cfg file, but we actually prefer to not have any user.cfg file in our main branch. Can you please remove this file from the commits?

Thank you.

Hi @dmocko,

Thank you for noticing these points. They should be addressed now.

dmocko · 2024-11-08T16:45:50Z

Both LDT and LIS compiled on the Discover Milan nodes.

Will next work on running the testcase.

dmocko · 2024-12-23T16:33:46Z

FYI for others on the LIS team: The config files and READMEs for the testcases associated with this PR are located in:

LISF/ldt/testcases/Param_AquaCrop
LISF/lis/testcases/surfacemodels/land/ac.7.2

The LDT testcase ran as expected, and produced identical output to the provided target.

dmocko · 2024-12-23T16:52:44Z

Hi @lbusschaert

The LIS testcase crashed for me. It ran to this point, and then stopped:

 [INFO] LIS cycle completed
[INFO] LIS cycle time: 01/01/2018 01:00:00
  [INFO] ************************************************
  [INFO] Start date (ymd tod):             2017           1           1
  [INFO] Stop date (ymd tod):              2018           1           1
  [INFO] Current date (ymd tod):           2017           1           1
  [INFO] ************************************************
  [INFO] ************************************************
  [INFO] Start date (ymd tod):             2017           1           1
  [INFO] Stop date (ymd tod):              2018           1           1
  [INFO] Current date (ymd tod):           2017           1           1
  [INFO] ************************************************
 [INFO] Using NLDAS-2 forcing
 [INFO] NLDAS-2 forcing directory : ./INPUT/NLDAS2.FORCING
 [INFO] AC72: new simulation period, reading of temperature record... Done!

The target lislog file (for this section) looks like this:

 [INFO] LIS cycle completed
[INFO] LIS cycle time: 01/01/2018 01:00:00
  [INFO] ************************************************
  [INFO] Start date (ymd tod):             2017           1           1
  [INFO] Stop date (ymd tod):              2018           1           1
  [INFO] Current date (ymd tod):           2017           1           1
  [INFO] ************************************************
  [INFO] ************************************************
  [INFO] Start date (ymd tod):             2017           1           1
  [INFO] Stop date (ymd tod):              2018           1           1
  [INFO] Current date (ymd tod):           2017           1           1
  [INFO] ************************************************
 [INFO] Using NLDAS-2 forcing
 [INFO] NLDAS-2 forcing directory : ./INPUT/NLDAS2.FORCING
 [INFO] AC72: new simulation period, reading of temperature record... Done!
 MSG: AC72_coldstart -- cold-starting AC72
 MSG: AC72_coldstart -- Using the specified start time    2017.00000000000     
 [INFO]: LIS timestep    3600.000    
[INFO] LIS cycle time: 01/01/2017 01:00:00
 [INFO] getting file1a.. 
 ./INPUT/NLDAS2.FORCING/2017/001/NLDAS_FORA0125_H.A20170101.0000.002.grb
 [INFO] getting file2a.. 
 ./INPUT/NLDAS2.FORCING/2017/001/NLDAS_FORA0125_H.A20170101.0100.002.grb
[INFO] LIS cycle time: 01/01/2017 02:00:00

The target simulation then continues again to 01/01/2018 and ends successfully.

Is this the expected behavior of Aquacrop? That it runs through the simulation period two times?

The job standard output gave this information:

forrtl: severe (122): invalid attempt to assign into a pointer that is not associated
Image              PC                Routine            Line        Source             
LIS                00000000015CEE4F  Unknown               Unknown  Unknown
LIS                000000000098EAF4  ac72_setup_               725  ac72_setup.F90
LIS                000000000069B501  lis_lsmmod_mp_lis         579  LIS_lsmMod.F90
LIS                000000000075F695  lis_surfacemodelm         180  LIS_surfaceModelMod.F90
LIS                000000000141E2C0  retrospective_run          97  retrospective_runMod.F90
LIS                0000000000E2870A  MAIN__                     67  lisdrv.F90
LIS                000000000041A60D  Unknown               Unknown  Unknown
libc-2.31.so       0000150AF863E24D  __libc_start_main     Unknown  Unknown
LIS                000000000041A53A  Unknown               Unknown  Unknown

Line 725 of this routine in your PR is here:
https://github.com/lbusschaert/LISF/blob/feature/ac.7.2_integration/lis/surfacemodels/land/ac.7.2/ac72_setup.F90#L725

I am not sure what this error means and how to resolve it in your code. Do you have any suggestions/ideas?

No rush on a response. I know it's a holiday week, and I will be taking time off starting tomorrow. Happy Holidays!

lbusschaert · 2024-12-28T13:51:16Z

Hi @lbusschaert

The LIS testcase crashed for me. It ran to this point, and then stopped:

 [INFO] LIS cycle completed
[INFO] LIS cycle time: 01/01/2018 01:00:00
  [INFO] ************************************************
  [INFO] Start date (ymd tod):             2017           1           1
  [INFO] Stop date (ymd tod):              2018           1           1
  [INFO] Current date (ymd tod):           2017           1           1
  [INFO] ************************************************
  [INFO] ************************************************
  [INFO] Start date (ymd tod):             2017           1           1
  [INFO] Stop date (ymd tod):              2018           1           1
  [INFO] Current date (ymd tod):           2017           1           1
  [INFO] ************************************************
 [INFO] Using NLDAS-2 forcing
 [INFO] NLDAS-2 forcing directory : ./INPUT/NLDAS2.FORCING
 [INFO] AC72: new simulation period, reading of temperature record... Done!

The target lislog file (for this section) looks like this:

 [INFO] LIS cycle completed
[INFO] LIS cycle time: 01/01/2018 01:00:00
  [INFO] ************************************************
  [INFO] Start date (ymd tod):             2017           1           1
  [INFO] Stop date (ymd tod):              2018           1           1
  [INFO] Current date (ymd tod):           2017           1           1
  [INFO] ************************************************
  [INFO] ************************************************
  [INFO] Start date (ymd tod):             2017           1           1
  [INFO] Stop date (ymd tod):              2018           1           1
  [INFO] Current date (ymd tod):           2017           1           1
  [INFO] ************************************************
 [INFO] Using NLDAS-2 forcing
 [INFO] NLDAS-2 forcing directory : ./INPUT/NLDAS2.FORCING
 [INFO] AC72: new simulation period, reading of temperature record... Done!
 MSG: AC72_coldstart -- cold-starting AC72
 MSG: AC72_coldstart -- Using the specified start time    2017.00000000000     
 [INFO]: LIS timestep    3600.000    
[INFO] LIS cycle time: 01/01/2017 01:00:00
 [INFO] getting file1a.. 
 ./INPUT/NLDAS2.FORCING/2017/001/NLDAS_FORA0125_H.A20170101.0000.002.grb
 [INFO] getting file2a.. 
 ./INPUT/NLDAS2.FORCING/2017/001/NLDAS_FORA0125_H.A20170101.0100.002.grb
[INFO] LIS cycle time: 01/01/2017 02:00:00

The target simulation then continues again to 01/01/2018 and ends successfully.

Is this the expected behavior of Aquacrop? That it runs through the simulation period two times?

The job standard output gave this information:

forrtl: severe (122): invalid attempt to assign into a pointer that is not associated
Image              PC                Routine            Line        Source             
LIS                00000000015CEE4F  Unknown               Unknown  Unknown
LIS                000000000098EAF4  ac72_setup_               725  ac72_setup.F90
LIS                000000000069B501  lis_lsmmod_mp_lis         579  LIS_lsmMod.F90
LIS                000000000075F695  lis_surfacemodelm         180  LIS_surfaceModelMod.F90
LIS                000000000141E2C0  retrospective_run          97  retrospective_runMod.F90
LIS                0000000000E2870A  MAIN__                     67  lisdrv.F90
LIS                000000000041A60D  Unknown               Unknown  Unknown
libc-2.31.so       0000150AF863E24D  __libc_start_main     Unknown  Unknown
LIS                000000000041A53A  Unknown               Unknown  Unknown

Line 725 of this routine in your PR is here: https://github.com/lbusschaert/LISF/blob/feature/ac.7.2_integration/lis/surfacemodels/land/ac.7.2/ac72_setup.F90#L725

I am not sure what this error means and how to resolve it in your code. Do you have any suggestions/ideas?

No rush on a response. I know it's a holiday week, and I will be taking time off starting tomorrow. Happy Holidays!

Hi @dmocko,

Good to see that LDT ran without any problems on your machine.

About LIS, it may be good to provide additional information on the reading of the meteorological forcings.

When AquaCrop is running a crop using stages calibrated in "growing degree days (GDDs)" (or heat accumulation units), the temperatures of the simulation period before the actual run need to be known. AquaCrop computes the lengths of the stages (in days) along with other variables, before the simulation (here set to a period of 365 days) starts.
Therefore, when the simulation gets initialized, the whole temperature record is read in ac72_read_Trecord: https://github.com/lbusschaert/LISF/blob/ac19e784e1aca729a9a9fa145947a3cc31a2629e/lis/surfacemodels/land/ac.7.2/ac72_prep_f.F90#L113

Once the temperatures for the full year are read in, the actual run starts. This happens for each simulation period (i.e. each year).

We know this is slowing down the simulations, and we foresee to get rid of this additional forcing reading in next (long-term) AquaCrop versions. Nevertheless, we found that running times are still acceptable, as it takes 10-20 min to read in the forwings for each year.

Can you clarify whether the simulation runs through before crashes? Do you get all the target output? Or does it stop before? I'll see if I can reproduce the error on my end. I suspect it might be related to an uninitialized variable.

Also on my side, there is no rush on a response. Happy Holidays, David!

dmocko · 2024-12-29T20:22:28Z

Hi @lbusschaert

Thanks for the explanation about the reading of the temperatures. This part of the LIS run took 17 minutes 59 seconds
on our Discover system using a single processor. It stopped right after that, and did not produce any target output.
Only the lislog.0000 file was written, and I shared the end of this file in my previous comment.

The run crashed on our system with this error on line 725 of ac72_setup.F90:
forrtl: severe (122): invalid attempt to assign into a pointer that is not associated

The line that caused the crash is linked in my previous comment, and is this line here:
call SetTemperatureRecord(GetClimRecord())

lbusschaert · 2025-01-08T11:01:23Z

Hi @dmocko

Best wishes for 2025!

I tried to see if I can reproduce the error on our systems, but all runs went through.
Could you specify which compiler and modules you are using? This way I can investigate further.

Best,
Louise

dmocko · 2025-01-08T13:15:39Z

Hi @lbusschaert

We are using these modules:

comp/gcc/11.2.0 2) comp/intel/2021.4.0 3) mpi/impi/2021.4.0

As part of this module file:

https://github.com/NASA-LIS/LISF/blob/master/env/discover/sles15/lisf_7.5_intel_2023.2.1

Here is our LISF/lis/make/configure.lis file:

FC              = mpiifort
FC77            = mpiifort
LD              = mpiifort
CC              = mpicc
AR              = ar
MOD_ESMF        = /discover/nobackup/projects/lis/libs/sles-15.4/esmf/8.5.0_intel-2023.2.1_impi-2021.11/mod/modO/Linux.intel.64.intelmpi.default
LIB_ESMF        = /discover/nobackup/projects/lis/libs/sles-15.4/esmf/8.5.0_intel-2023.2.1_impi-2021.11/lib/libO/Linux.intel.64.intelmpi.default
INC_JPEG2000    = /discover/nobackup/projects/lis/libs/sles-15.4/openjpeg/2.3.0-150000.3.13.1.x86_64/usr/include/
LIB_JPEG2000    = /discover/nobackup/projects/lis/libs/sles-15.4/openjpeg/2.3.0-150000.3.13.1.x86_64/usr/lib/
INC_ECCODES     = /discover/nobackup/projects/lis/libs/sles-15.4/eccodes/2.32.0_intel-2023.2.1/include/
LIB_ECCODES     = /discover/nobackup/projects/lis/libs/sles-15.4/eccodes/2.32.0_intel-2023.2.1/lib/
INC_NETCDF      = /discover/nobackup/projects/lis/libs/sles-15.4/netcdf/4.9.2_intel-2023.2.1/include/
LIB_NETCDF      = /discover/nobackup/projects/lis/libs/sles-15.4/netcdf/4.9.2_intel-2023.2.1/lib/
INC_HDF4        = /discover/nobackup/projects/lis/libs/sles-15.4/hdf4/4.2.16-2_intel-2023.2.1/include/
LIB_HDF4        = /discover/nobackup/projects/lis/libs/sles-15.4/hdf4/4.2.16-2_intel-2023.2.1/lib/
INC_HDF5        = /discover/nobackup/projects/lis/libs/sles-15.4/hdf5/1.14.2_intel-2023.2.1/include/
LIB_HDF5        = /discover/nobackup/projects/lis/libs/sles-15.4/hdf5/1.14.2_intel-2023.2.1/lib/
INC_HDFEOS      = /discover/nobackup/projects/lis/libs/sles-15.4/hdfeos2/3.0_intel-2023.2.1/include/
LIB_HDFEOS      = /discover/nobackup/projects/lis/libs/sles-15.4/hdfeos2/3.0_intel-2023.2.1/lib/
LIB_MINPACK     =
INC_CRTM        =
LIB_CRTM        =
INC_PROF_UTIL   =
LIB_PROF_UTIL   =
INC_CMEM        =
LIB_CMEM        =
LIB_LAPACK      =
INC_PETSC       =
LIB_PETSC       =
CFLAGS          = -c  -fp-model precise -traceback -DIFC -DLINUX -DLIS_JULES
FFLAGS77        = -c  -O2  -fp-model precise -traceback -nomixed-str-len-arg -names lowercase -convert big_endian -assume byterecl -DSPMD -DHIDE_SHR_MSG -DNO_SHR_VMATH -DIFC -DLINUX -I$(MOD_ESMF) -DUSE_INCLUDE_MPI -I$(INC_ECCODES) -I$(INC_NETCDF) -I$(INC_HDFEOS) -I$(INC_HDF4) -I$(INC_HDF5) -DLIS_JULES
FFLAGS          = -c  -O2  -fp-model precise -u -traceback -fpe0 -nomixed-str-len-arg -names lowercase -convert big_endian -assume byterecl -DSPMD -DHIDE_SHR_MSG -DNO_SHR_VMATH -DIFC -DLINUX -I$(MOD_ESMF) -DUSE_INCLUDE_MPI -I$(INC_ECCODES) -I$(INC_NETCDF) -I$(INC_HDFEOS) -I$(INC_HDF4) -I$(INC_HDF5) -DLIS_JULES
LDFLAGS         = -L$(LIB_ESMF) -lesmf -lstdc++ -limf -lrt -lmpi -L$(LIB_ECCODES) -leccodes_f90 -leccodes -L$(LIB_JPEG2000) -lopenjp2 -L$(LIB_NETCDF) -lnetcdff -lnetcdf -L$(LIB_HDFEOS) -lhdfeos -lGctp -L$(LIB_HDF4) -lmfhdf -ldf -ljpeg -lz /usr/lib64/libtirpc.so -L$(LIB_HDF5) -lhdf5_fortran -lhdf5_hl -lhdf5 -lz
LIS_LIB_FLAGS   = -lesmf -lstdc++ -limf -lrt -leccodes_f90 -leccodes -lopenjp2 -lnetcdff -lnetcdf -lhdfeos -lGctp -lmfhdf -ldf -ljpeg -lz -lhdf5_fortran -lhdf5_hl -lhdf5
LIS_LIB_PATHS   = -L$(LIB_ESMF) -L$(LIB_ECCODES) -L$(LIB_JPEG2000) -L$(LIB_NETCDF) -L$(LIB_HDFEOS) -L$(LIB_HDF4) -L$(LIB_HDF5)

@jvgeiger and @emkemp - Do you have any insight on why line 725 of ac72_setup.f90 is giving the error about assigning into a pointer that is not associated?

It's also not clear to me why this line only throws the error at the start of the second run through (after preparing the temperature data).

lbusschaert · 2025-01-08T13:22:06Z

Hi @dmocko

Thank you for providing the specifics of your setup. I'll investigate more.

Note that this is the first time that the line is read. The temperature data is prepared via the call to ac72_read_Trecord(n) in L678 of ac72_setup.f90.

lbusschaert · 2025-01-10T13:05:47Z

Hi @dmocko,

I've been in contact with the IT team here to try to understand the problem. It is likely due to a difference in compiler. We are currently updating our compilation (we are still using intel 2021) but this may still take a couple of weeks. I can then test the code.

In the meantime, would you be able to also try to run the LIS test case with an executable compiled in DEBUG mode to see if it reacts differently to another level of optimization?

Thank you

lbusschaert added 30 commits January 22, 2024 03:50

DEV: initial implementation AC71

9c5e022

final fixes for original v7.1

31bfd5d

ENH: first major clean-up

2264d31

"ENH: major clean-up 2, change of SOILPARM.TBL"

cc29df7

ENH: WCM removal

ca84a2e

ENH: simplified lis.config

572e117

DEV: run AC with LIS forcings

fa7207c

ENH: corrections for forcing ref height

217202c

removal of MERRA2_AC

74594ac

ENH: ac71 reads crop information from LDT

65ee56d

DEV: spatial crops for AC71

6bdaf62

ENH: minor edits

e927640

ENH: cleaning-up of vars before restart

8f8dbc3

ENH: Nrsoillayers and initial cdts in lis.config

893b7c7

ENH: crop and sim period defined in lis.config

697deb6

ENH: coupling to LIS irrigation module (sprinkler)

abc73f7

ENH: cleaning up of output vars and units

4f89f84

ENH: clean up

81852ae

ENH: final clean-up before LDT

6feb0eb

DEV: AC71 in LDT

56af948

ENH: AC crop constant crop assignment

80ebd88

ENH: add check for model timestep

c3e8074

BUG: check for daily time step AC fixed

b5e0107

DEV: implementation of Trecord for GDD

d250313

BUG: typo fix

987f8cb

DOC: better comments

9d3109d

ENH: added MPI_barrier -- requires testing on larger domain

01705a9

ENH: fixed 365-6 sim period

10489d3

BUG: fixed if statement

b3fdb13

lbusschaert added 13 commits October 23, 2024 13:52

ENH: transition from aquacrop in dp to sp

b8a57cf

Switched to AquaCrop sp. ac_modules are an exact copy of AC src stored in branch https://github.com/lbusschaert/AquaCrop/tree/LIS/sp_2. WIP: restarting is technically working but needs to be checked via test cases

BUG: LIS clock adjustment for restart

410b844

WIP: towards a working restart

363c9d4

ENH/STY: added restart vars and header update

81b3d61

Merge branch 'master' of github.com:NASA-LIS/LISF into ac.7.2_integra…

f4731f5

…tion

ENH: working restart

a94fb31

ENH: consistency fixes

020e9dd

ENH: minor fixes

252a5e5

ENH: minor fixes

6f0437e

TST: added test ldt and lis test cases

1673435

ENH: minor ldt fix

9ff66ea

ENH: undid change in metforc

628678b

ENH: minor changes after comments on PR

5c7fb18

dmocko added enhancement New feature or request Documentation Update to documentation (User's Guide or within the code) labels Nov 5, 2024

lbusschaert added 3 commits November 6, 2024 11:03

delete user.cfg

1638147

DOC: added AquaCrop config entries to config.adoc files

62317a7

DOC: added AquaCrop LIS outputs to .adoc file

ac19e78

lbusschaert mentioned this pull request Nov 6, 2024

DOC: added documentation as requested by NASA-LISF KUL-RSDA/LISF#56

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/AquaCropv7.2 integration #1633

Feature/AquaCropv7.2 integration #1633

lbusschaert commented Nov 5, 2024

dmocko commented Nov 5, 2024

lbusschaert commented Nov 6, 2024

dmocko commented Nov 8, 2024

dmocko commented Dec 23, 2024

dmocko commented Dec 23, 2024

lbusschaert commented Dec 28, 2024 •

edited

Loading

dmocko commented Dec 29, 2024

lbusschaert commented Jan 8, 2025

dmocko commented Jan 8, 2025

lbusschaert commented Jan 8, 2025

lbusschaert commented Jan 10, 2025

Feature/AquaCropv7.2 integration #1633

Are you sure you want to change the base?

Feature/AquaCropv7.2 integration #1633

Conversation

lbusschaert commented Nov 5, 2024

Description

Testcase

dmocko commented Nov 5, 2024

lbusschaert commented Nov 6, 2024

dmocko commented Nov 8, 2024

dmocko commented Dec 23, 2024

dmocko commented Dec 23, 2024

lbusschaert commented Dec 28, 2024 • edited Loading

dmocko commented Dec 29, 2024

lbusschaert commented Jan 8, 2025

dmocko commented Jan 8, 2025

lbusschaert commented Jan 8, 2025

lbusschaert commented Jan 10, 2025

lbusschaert commented Dec 28, 2024 •

edited

Loading