Query about dp --fine-tune taking too long without writing any file

[rajnichahal]

Hi,

I am trying dp --finetune with newest version of deepmd. The model I am trying to finetune with new data was generated using 'se_e2_a' descriptor. The training has been running for a day, but no file is written. Usually, my regular training is done in a day or two.

I am not sure if it is taking that long to load the previous frozen model, or if something else is going on. Anyway, I am wondering if this is normal with the finetune option. I would appreciate any thoughts on it.

Thanks!

[njzjz]

Is there any message on the screen?

[rajnichahal]

Hi, I am not seeing any message.

[njzjz]

Was it killed? I guess you could use top to check the CPU usage.

[rajnichahal]

No, it wasn't killed right away. It continued running without writing anything and was eventually killed after 24 hr wall time.

[rajnichahal]

Here is my command for finetune job submission:
dp train input.json --finetune previous-frozen-model.pb