References and discussions

[AdityaKane2001]

@sayakpaul

Following some important links for reference:

• WandB project: https://wandb.ai/compyle/keras-regnet-training?workspace=user-compyle
• GH PR: Adding RegNets to tf.keras.applications keras-team/keras#15702

If possible let's keep the discussions and updates regarding training here. As discussed I'll create an issue for each experiment we want to run.

[AdityaKane2001]

What I've tried so far:

• Fixed padding in Blocks and Stem (run)
• Normalization to ImgNet mean and std (I know it was decided to avoid that, but since it was in the original implementation I thought it would be best to eliminate that first) (run1, run2). It did give a boost of 0.24%, but it is likely that it is a fluke. Thus dropping it henceforth.

Things to try:

• Color jitter and/or Mixup
• Correct initializers (he_normal)
• EMA **

The main issue I am facing is the number of models, and their difference in flop regimes. It is difficult to generalize the findings for more than 2 or 3 models at a time. So we might need to start from scratch with every model.

Please share your thoughts on the same.

[AdityaKane2001]

	Paper	Ours	Diff	Location	Comments
X002	68.9	67.15	1.75	gs://ak-us-train/models/10_24_2021_10h29m42s/all_model_epoch_96	adamw, area_factor=0.25
X004	72.6	71.22	1.38	gs://ak-us-train/models/10_26_2021_08h41m16s/all_model_epoch_96	adamw, area_factor=0.08
X006	74.1	72.37	1.73	gs://ak-us-train/models/10_26_2021_08h43m17s/all_model_epoch_96	adamw, area_factor=0.08
X008	75.2	73.45	1.75	gs://ak-us-train/models/10_26_2021_08h46m10s/all_model_epoch_97	adamw, area_factor=0.08
X016	77	75.55	1.45	gs://ak-us-train/models/10_28_2021_12h59m13s/all_model_epoch_94	adamw, area_factor=0.08, mixup=0.2
X032	78.3	77.09	1.21	gs://ak-us-train/models/10_28_2021_17h54m03s/all_model_epoch_96	adamw, area_factor=0.08, mixup=0.2
X040	78.6	77.87	0.73	gs://ak-us-train/models/10_30_2021_09h13m50s/all_model_epoch_94	adamw, area_factor=0.08, mixup=0.2
X064	79.2	78.22	0.98	gs://ak-us-train/models/11_05_2021_05h00m49s/all_model_epoch_97	adamw, area_factor=0.08, mixup=0.3
X080	79.3	78.41	0.89	gs://ak-us-train/models/11_08_2021_13h07m18s/all_model_epoch_94	adamw, area_factor=0.08, mixup=0.3
X120	79.7	79.09	0.61	gs://ak-us-train/models/11_08_2021_08h31m23s/all_model_epoch_97	adamw, area_factor=0.08, mixup=0.4
X160	80	79.53	0.47	gs://ak-us-train/models/11_10_2021_11h05m01s/all_model_epoch_96	adamw, area_factor=0.08, mixup=0.4
X320	80.5	80.35	0.15	gs://ak-us-train/models/11_09_2021_13h39m30s/all_model_epoch_97	adamw, area_factor=0.08, mixup=0.4

[AdityaKane2001]

	Paper	Ours	Diff	Location	Comments
Y002	70.3	68.51	1.79	gs://ak-us-train/models/11_16_2021_03h54m11s/all_model_epoch_94	adamw, WD=1e-5, area_factor=0.16 mixup=0.2
Y004	74.1	72.11	1.99	gs://ak-us-train/models/12_10_2021_09h36m01s/all_model_epoch_96	adamw, area_factor=0.16, mixup=0.2,WD=1e-5
Y006	75.5	73.52	1.98	gs://ak-us-train/models/12_05_2021_13h26m36s/all_model_epoch_94	adamw, area_factor=0.16, mixup=0.2
Y008	76.3	74.48	1.82	gs://ak-us-train/models/12_01_2021_07h24m52s/all_model_epoch_94	adamw, area_factor=0.16, mixup=0.2
Y016	77.9	76.95	0.95	gs://ak-us-train/models/11_28_2021_10h11m52s/all_model_epoch_94	adamw, area_factor=0.08, mixup=0.2
Y032	78.9	78.05	0.85	gs://ak-us-train/models/11_27_2021_03h48m42s/all_model_epoch_94	adamw, area_factor=0.08, mixup=0.2
Y040	79.4	78.2	1.2	gs://ak-us-train/models/11_26_2021_11h30m09s/all_model_epoch_95	adamw, area_factor=0.08, mixup=0.2
Y064	79.9	78.95	0.95	gs://ak-us-train/models/11_26_2021_03h36m02s/all_model_epoch_95	adamw, area_factor=0.08, mixup=0.3
Y080	79.9	79.11	0.69	gs://ak-us-train/models/11_24_2021_09h10m48s/all_model_epoch_96	adamw, area_factor=0.08, mixup=0.3
Y120	80.3	79.45	0.85	gs://ak-us-train/models/11_22_2021_08h43m05s/all_model_epoch_96	adamw, area_factor=0.08, mixup=0.4
Y160	80.4	79.71	0.69	gs://ak-us-train/models/11_23_2021_13h33m06s/all_model_epoch_95	adamw, area_factor=0.08, mixup=0.4
Y320	80.9	80.12	0.78	gs://ak-us-train/models/11_26_2021_03h14m49s/all_model_epoch_96	adamw, area_factor=0.08, mixup=0.4