Ensemble restart file not generating (Step 3 in DA)

[GR-Sooraj]

Hello All,

I am trying to assimilate SMAP soil moisture. I have successfully run the OL run. Unfortunately, during the ensemble restart processing, the file is not generating, and even it is not showing any error (a portion of the log file shown below). After showing the total number of grid points, it is ending abruptly. As per the target log file, next is ensemble restart file generation.

It will be great if I can get some help to move forward. Attaching the config file and log file.
ldt_ens_config.txt
ldtlog.0000.txt

[dmocko]

Hi @GR-Sooraj

Can you please try it without the quotations around this?:

Input restart filename: /home/user6/sudarsan/LIS/TOPLEVELDIR/sooraj/test_da/OUTPUT/SURFACEMODEL/202201/LIS_RST_NOAHMP36_202201311800.d01.nc

[dmocko]

Can you please attach these two files, so we can try to run this?

Processed LSM parameter filename: /home/user6/sudarsan/LIS/TOPLEVELDIR/sooraj/ldt_config/lis_input1.d01.nc # Final output file read by LIS-7

Input restart filename: "/home/user6/sudarsan/LIS/TOPLEVELDIR/sooraj/test_da/OUTPUT/SURFACEMODEL/202201/LIS_RST_NOAHMP36_202201311800.d01.nc"

[GR-Sooraj]

Hi @dmocko ,

Attaching those two files in a zip folder
two_ncfiles.zip

Thank You

[dmocko]

Hi @GR-Sooraj

Thanks for sharing these files. I just ran with your ldt.config file and these files, and LDT successfully ran:

  - - - - - - - - - Noah-MP Parameters - - - - - - - - - - - -
 Total number of surface tiles          35
 Total number of tiles for LSM          35
 Total number of grid points:          35
  Generating restart file: ./LIS_EnRST_NOAHMP36_202201311800.d01.nc
  Successfully generated restart file: ./LIS_EnRST_NOAHMP36_202201311800.d01.nc
 --------------------------------
  Finished LDT run 
 --------------------------------

I used the latest code in the master branch. What code/branch/version are you using?

When you run, is there any message given to the screen or your standard job output about the code stopping?

Another thing to try is this: cd LISF/ldt/make, type "gmake realclean", then when you configure the code, choose "-1" for this:

Optimization level (-3=strict checks with warnings, -2=strict checks, -1=debug, 0,1,2,3, default=2):

Then the rest of your configure and compile as before. Hopefully, it will provide more info for us when LDT stops.

[GR-Sooraj]

Hi @dmocko

I have used the version 7.5.0

The screenshot of the outerr file after running / running got stopped is shown below:

[dmocko]

Please ensure that MPI is installed correctly. Please see this:
#836 (comment)

Also, you may be out of memory. Probably not, because this is a very small domain, but you may try increasing available memory.