add fuse_layer_norm_grad functor and op

[crazy-JiangDongHua]

No description provided.

[github-actions]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

[github-actions]

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10612/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.9ms (= 4392.4ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.6ms (= 5758.7ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.31 (= 57.6ms / 43.9ms)

OneFlow resnet50 time: 26.4ms (= 2642.9ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.1ms (= 3713.9ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.41 (= 37.1ms / 26.4ms)

OneFlow resnet50 time: 18.5ms (= 3693.1ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 35.4ms (= 7083.0ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.92 (= 35.4ms / 18.5ms)

OneFlow resnet50 time: 17.3ms (= 3469.9ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 34.0ms (= 6800.5ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.96 (= 34.0ms / 17.3ms)

OneFlow resnet50 time: 16.8ms (= 3350.3ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 29.9ms (= 5985.4ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.79 (= 29.9ms / 16.8ms)

OneFlow swin dataloader time: 0.200s (= 40.006s / 200, num_workers=1)
PyTorch swin dataloader time: 0.128s (= 25.541s / 200, num_workers=1)
Relative speed: 0.638 (= 0.128s / 0.200s)

OneFlow swin dataloader time: 0.056s (= 11.249s / 200, num_workers=4)
PyTorch swin dataloader time: 0.032s (= 6.491s / 200, num_workers=4)
Relative speed: 0.577 (= 0.032s / 0.056s)

OneFlow swin dataloader time: 0.031s (= 6.203s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.308s / 200, num_workers=8)
Relative speed: 0.533 (= 0.017s / 0.031s)

❌ OneFlow resnet50 time: 49.2ms (= 4919.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.3ms (= 6533.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.33 (= 65.3ms / 49.2ms)

OneFlow resnet50 time: 36.2ms (= 3619.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 45.5ms (= 4552.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.26 (= 45.5ms / 36.2ms)

OneFlow resnet50 time: 27.9ms (= 5576.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 42.1ms (= 8422.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.51 (= 42.1ms / 27.9ms)

OneFlow resnet50 time: 25.3ms (= 5063.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.5ms (= 7693.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.52 (= 38.5ms / 25.3ms)

OneFlow resnet50 time: 24.8ms (= 4965.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 35.8ms (= 7163.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.44 (= 35.8ms / 24.8ms)

[ShawnXuan]

是否需要一个空的 FuseLayerNormGradCpuKernel 在 oneflow/user/kernels/layer_norm_cpu_kernel.cpp 文件中

[github-actions]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.