Difficulty Reproducing Results for Grounded3DLLM #10
Comments
|
Thanks for your interest!
2/3. The training should ensure that the pre-trained checkpoints (the path of "general.checkpoint" in the script) are correct. Your results show that the pre-trained clasp detector (step 2) is ~0, which must be incorrect. I am not sure which part of the loading process is wrong; perhaps you can test the results of step 2 (CLASP detector) and ensure the pre-trained detector is correct. Let me know more information during reproduction. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Dear authors,
Thank you for your excellent work and for making your resources available to the community!
I have been attempting to reproduce the results reported in your paper but have encountered some difficulties. I would greatly appreciate your guidance on a few points:
1. Evaluation of Stage 3 Checkpoint
I attempted to evaluate the provided Stage 3 checkpoint. However, the required scene features and detection folders were not included within the uploaded checkpoint, preventing me from running the evaluation. Could you provide instructions or the necessary resources to evaluate the Stage 3 checkpoint?
2. Retraining Stage 3 Model
I tried retraining the Stage 3 model using the provided Stage 1 and Stage 2 checkpoints but achieved results significantly lower than those reported in the paper, particularly on ScanRefer, Multi3DRef, Scan2Cap, and ScanNet200.
To retrain, I placed the original Stage 1 and Stage 2 checkpoints (from the repository) into the
savedfolder and ran the following command:I conducted this training on a machine with 4 A100 80GB GPUs.
3. Evaluation Results
After running the evaluation script using:
final_scripts/eval_llm.sh ./saved/step3_mask3d_lang_4GPUS/01-03-14-19-41
I obtained the following results (summarized below for key metrics):
m3drefer official evaluator:
IoU 0.25: Overall = 12.4
IoU 0.50: Overall = 11.5
ScanRefer Evaluation:
Overall BBox Top-1 IoU 0.25: 0.1088
Overall BBox Top-1 IoU 0.50: 0.0955
Scan2Cap:
CIDEr: 0.308 (IoU 0.25), 0.258 (IoU 0.50)
Despite following the provided steps and ensuring a similar training environment, the outcomes are not aligning with the reported results.
Could you confirm if there are additional steps or configurations I might have missed? Alternatively, are there any known issues or dependencies that could affect the reproducibility?
Thank you for your time and assistance!
These are the complete reported results log upon running the following command:
bash final_scripts/eval_llm.sh ./saved/step3_mask3d_lang_4GPUS/01-03-14-19-41The text was updated successfully, but these errors were encountered: