The new Deepmdkit v2.2.11 and v3 is much slower and occupies much more GPU memory than before

[Pdeedee]

Did any body encounters the problem?With the same param.json and machine.json, the newer version of Deepmd is slow and occupies more memory.

[wanghan-iapcm]

Could you please provide an example and the version of "faster deepmd-kit" so that we can reproduce the efficiency decrease?

[Pdeedee]

of course! The fast version I use was 2.2.7.
This is the speed I was using 2.2.7 with se_atten_v2,

This is the speed I was using 3.0 with se_atten_v2
.
Moreover, the version of both 2.2.11 and 3.0 occupies 98% memory of 4090, but 2.2.7 only occupies about 2% while training.

[Pdeedee]

An interesting thing I found was that when the GPU is alread with another task, the occupation would decrease to 50%. I observe the occupation decrease when the train.log decrease the neighborlist batch size. Would it probably be the neighborlist build bug.

[Pdeedee]

This is the param.json automatically generated by deepmd 3.0
{
"model": {
"type_map": [
"H",
"C",
"N",
"O"
],
"descriptor": {
"type": "se_atten_v2",
"sel": 120,
"rcut_smth": 0.5,
"rcut": 6.0,
"neuron": [
25,
50,
100
],
"resnet_dt": false,
"axis_neuron": 16,
"seed": 1024089868,
"attn": 128,
"attn_layer": 0,
"attn_dotr": true,
"attn_mask": false,
"precision": "float64",
"_comment2": " that's all"
},
"fitting_net": {
"neuron": [
200,
200,
200
],
"resnet_dt": true,
"seed": 2744276785,
"_comment": " that's all"
},
"_comment": " that's all"
},
"learning_rate": {
"type": "exp",
"decay_steps": 5000,
"start_lr": 0.001,
"stop_lr": 3.51e-08,
"_comment": "that's all"
},
"loss": {
"type": "ener",
"start_pref_e": 0.1,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0.1,
"limit_pref_v": 1,
"_comment": " that's all"
},
"training": {
"numb_steps": 800000,
"seed": 2674951016,
"disp_file": "lcurve.out",
"disp_freq": 2000,
"save_freq": 10000,
"_comment": "that's all",
"training_data": {
"systems": [
"../data.init/cl20nptvasp/deepmdl/H24C24N48O48",
"../data.init/big1/cl20iter7test_set/H24C24N48O48",
"../data.iters/iter.000000/02.fp/data.001",
"../data.iters/iter.000000/02.fp/data.000",
"../data.iters/iter.000001/02.fp/data.001",
"../data.iters/iter.000001/02.fp/data.000",
"../data.iters/iter.000002/02.fp/data.001",
"../data.iters/iter.000002/02.fp/data.000",
"../data.iters/iter.000004/02.fp/data.001",
"../data.iters/iter.000004/02.fp/data.000",
"../data.iters/iter.000005/02.fp/data.001",
"../data.iters/iter.000005/02.fp/data.000",
"../data.iters/iter.000006/02.fp/data.001",
"../data.iters/iter.000006/02.fp/data.000",
"../data.iters/iter.000007/02.fp/data.001",
"../data.iters/iter.000007/02.fp/data.000",
"../data.iters/iter.000008/02.fp/data.001",
"../data.iters/iter.000008/02.fp/data.000",
"../data.iters/iter.000009/02.fp/data.001",
"../data.iters/iter.000009/02.fp/data.000",
"../data.iters/iter.000010/02.fp/data.001",
"../data.iters/iter.000010/02.fp/data.000",
"../data.iters/iter.000011/02.fp/data.001",
"../data.iters/iter.000011/02.fp/data.000"
],
"batch_size": [
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1
]
}
}
}

[njzjz]

when the GPU is alread with another task

Did you test the performance in this situation? Normally a GPU can only do one task at the same time.

[Pdeedee]

With 2.2.7, I can train 2 models in only 1 4090,but with 2.2.11, I can only train 1.

[njzjz]

The memory behavior is desired. See #3699.

[wanghan-iapcm]

did you use the same input for 2.2.7 and 2.2.11? please provide the input files.
it would be grateful if you provide the minimal dataset that reproduce the issue.

[njzjz]

In addtion, the log files are also helpful to provide information.

[iProzd]

@Pdeedee I have executed the example training script examples/water/se_atten_compressible/input.json using version 3.0.0, and the training speed seems all right. Could you also perform training with this example, ensuring that the environment installation is correct (preferably isolated from the 2.2.7 environment) and correctlly utilizing the GPU?