Unable to reproduce claimed results for VoxFormer on KITTI360-SSCBench

[ahayler]

Dear all,

I am trying to replicate the results of VoxFormer on your KITTI360 benchmark by using your checkpoint. So far I have achieved sensible results, but with an occupancy IoU of about 34.5 instead of your 38.6. Here is how I proceeded to generate the needed query proposals for stage 2:

1. Predict the depth from matched stereo images using the provided script with --dataset kitti360 --baseline 331.53255659999996 .(the baseline number was indicated in the script. Outputs look sensible)
2. Estimate LIDAR from the depth maps using the provided script.
3. Accumulate the LIDAR scans using your script a sequence length 10 (tried 1 alternatively with similar results). KITTI360 poses get inverted during loading (and matched to the correct image). Again the results look like a sensible accumulation of depth maps/LIDAR scans.
4. Use Stage1 with the provided checkpoint to predict the query proposals.
5. Use them with the images themselves as input for stage2 to get our final predictions (loading your stage2 checkpoint).

Note: As stereo images were not provided in SSCBench and the pose files in the dataset also do not necessarily match the frame Ids provided, I created a mapping of the SSCBench frame Ids to the KITTI360 frame ids.

Is my process correct? Why might my results be worse? Could you either provide detailed instructions on how to replicate your results or the predictions?

Kind regards,
Adrian

[npurson]

#9 could be the possible cause, you may check the IoU scores for classes beyond 10.

[ahayler]

Thank you for the suggestion! Sadly I already accounted for that!

[npurson]

1. Predict the depth from matched stereo images using the provided script with --dataset kitti360 --baseline 331.53255659999996 .(the baseline number was indicated in the script. Outputs look sensible)

@ahayler I'm wondering where is the provided script indicating the baseline number? I've searched the SSCBench and VoxFormer repos for that specific number but couldn't locate it. I want to make sure I'm using the right script.

[ahayler]

@npurson

In the mobilestereonet prediction.py script you can find on top:

Mathmatical Formula to convert the disparity to depth:

depth = baseline * focal / disparity
For KITTI the baseline is 0.54m and the focal ~721 pixels.
The final formula is:
depth = 0.54 * 721 / disp

For KITTI-360, depth = 0.6 * 552.554261 / disp

With 0.54 * 721 = 388.1823 being the value for semantic KITTI it seemed like 0.6 * 552.554261 = 331.53255659999996 is the baseline value for KITTI360. This was only an informed guess, so correct me if I am wrong.

Let me know, whether or not you manage to reproduce the results from the paper :)

[npurson]

@ahayler Thank you for your response. I have an additional question regarding:

Note: As stereo images were not provided in SSCBench and the pose files in the dataset also do not necessarily match the frame Ids provided, I created a mapping of the SSCBench frame Ids to the KITTI360 frame ids.

How could I match the frames between SSCBench and KITTI360 to get stereo image pairs? I found the frame ID ranges do not align between the 2 datasets (e.g. sequence 0 in SSCBench contains frames 0 - 10,482, while sequence 0 in KITTI360 contains frames 0 - 11,517).
Besides, the frames with the same ID across datasets do not appear to visually match (e.g., frame 0 matches across the above mentioned sequences, but frame 10.482 of them are obviously not from the same scene).