[DONT MERGE] revert to PR 10351

[marigoold]

No description provided.

[github-actions]

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10379/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.7ms (= 4372.0ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.7ms (= 5771.2ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.32 (= 57.7ms / 43.7ms)

OneFlow resnet50 time: 26.1ms (= 2615.0ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.7ms (= 3765.3ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.44 (= 37.7ms / 26.1ms)

OneFlow resnet50 time: 18.5ms (= 3708.2ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 37.8ms (= 7556.9ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 2.04 (= 37.8ms / 18.5ms)

OneFlow resnet50 time: 17.0ms (= 3406.9ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 33.2ms (= 6643.7ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.95 (= 33.2ms / 17.0ms)

OneFlow resnet50 time: 17.1ms (= 3423.6ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 28.8ms (= 5755.6ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.68 (= 28.8ms / 17.1ms)

OneFlow swin dataloader time: 0.200s (= 39.961s / 200, num_workers=1)
PyTorch swin dataloader time: 0.128s (= 25.523s / 200, num_workers=1)
Relative speed: 0.639 (= 0.128s / 0.200s)

OneFlow swin dataloader time: 0.054s (= 10.860s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.662s / 200, num_workers=4)
Relative speed: 0.613 (= 0.033s / 0.054s)

OneFlow swin dataloader time: 0.030s (= 6.093s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.409s / 200, num_workers=8)
Relative speed: 0.559 (= 0.017s / 0.030s)

❌ OneFlow resnet50 time: 48.9ms (= 4893.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.8ms (= 6479.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 64.8ms / 48.9ms)

OneFlow resnet50 time: 37.1ms (= 3712.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 46.9ms (= 4686.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.26 (= 46.9ms / 37.1ms)

OneFlow resnet50 time: 28.3ms (= 5663.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 40.6ms (= 8120.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.43 (= 40.6ms / 28.3ms)

OneFlow resnet50 time: 25.0ms (= 5002.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.6ms (= 7920.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.58 (= 39.6ms / 25.0ms)

OneFlow resnet50 time: 24.2ms (= 4840.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.2ms (= 7241.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.50 (= 36.2ms / 24.2ms)