Cannot reproduce the results reported in the Paper (CD=2.723)

[AlphaPav]

You need to train the whole network with Chamfer Distance. It reaches CD ~0.40 on ShapeNet.
Then, you need to fine-tune the network with Gridding Loss + Chamfer Distance on the Coarse Point Cloud.
Finally, you fine-tune the network with Chamfer Distance. Chamfer Distance is taken as a metric, therefore, you cannot get lower CD without using Chamfer Distance as a loss.

Originally posted by @hzxie in #3 (comment)

[hzxie]

So, what's the problem you are facing now?
Please provide more details.

[AlphaPav]

Hi author, thanks for the amazing work.

On your released pre-trained model, I can get 0.7082 F-score, 2.722 CD.
However, when I train from scratch, I had some problems listed below:

"You need to train the whole network with Chamfer Distance." --- It reaches 4.588 CD, 0.6133 F-score, which is similar with Table 7&Not Used&CD&Complete = 4.460 in your paper.

"Then .. fine-tune the network with Gridding Loss + Chamfer Distance on the Coarse Point Cloud." ---- It reaches 4.536 CD, 0.6255 F-score. It was supposed to be about ~2.7, right?

sparse_loss = chamfer_dist(sparse_ptcloud, data['gtcloud'])
dense_loss = chamfer_dist(dense_ptcloud, data['gtcloud'])
grid_loss = gridding_loss(sparse_ptcloud, data['gtcloud'])
_loss = sparse_loss + dense_loss + grid_loss

__C.NETWORK.GRIDDING_LOSS_SCALES = [128]
__C.NETWORK.GRIDDING_LOSS_ALPHAS = [0.1]

"Finally, you fine-tune the network with Chamfer Distance." --- the CD didn't decrease below 4.536.

I'm wondering what steps am I making mistakes? (like learning rate/loss weight of gridding loss)

[AlphaPav]

your processed ShapeNet dataset has 28974 training data samples
while the PCN dataset has 231792 training data samples

is it because your provided dataset is not completed?

[hzxie]

@AlphaPav
Sorry for the late reply. I don't have time to check this issue these days.
But I'm sure that there is nothing wrong with the released dataset. 231792 / 28974 = 8, which indicates that there are 8 partial input point cloud for each model in ShapeNet.

[AlphaPav]

@AlphaPav
Sorry for the late reply. I don't have time to check this issue these days.
But I'm sure that there is nothing wrong with the released dataset. 231792 / 28974 = 8, which indicates that there are 8 partial input point cloud for each model in ShapeNet.

The PCN dataset is about 48 GB, while the released dataset is about 10 GB. Do you mean that you randomly augment each point cloud 8 times during training?

[hzxie]

No. I think the difference may be caused by different compression ratios.
You can also generate the ShapeNet dataset from PCN with this script.

[SarahChane98]

Hi! I also cannot reproduce the results. The highest CD I got after training three times was 5.2. May I know how many epochs you've trained for each round respectively? (i.e. CD only, CD + gridding loss, CD only)

[hzxie]

@SarahChane98
I cannot report the exact numbers of epochs for each round.
For each round, I train several times until the loss does not decrease.
Try to fine-tune the network again with the previous weights (from last training).

[paulwong16]

Hi there, I just tested your pretrained model on test set, and the result is close to the value reported in paper. However, when I tested on validation dataset, it reported a dense CD around 7.177. I was wondering why there is a hugh gap between CDs on val set and test set?

---

and a dense cd around 5.087 for training set reported with pretrained model (should be the same as training dense loss if i understand correctly)

[hzxie]

@paulwong16
If the reported results are correct, one possible reason why the pretrained model performs worse in the validation and training set is that we choose the best model for the test set instead of the validation set and training set.

[paulwong16]

@paulwong16

Because we choose the best model for the test set instead of the validation set.

but why CD on test set could be even much lower than on training set?

[hzxie]

@paulwong16
Because the pretrained model is best for fitting distribution of the testing set.
Instead, the distribution of the training and validation set may be different from the testing set.

[paulwong16]

@paulwong16

Because the pretrained model is best for fitting distribution of the testing set.

Instead, the distribution of the training and validation set may be different from the testing set.

well...i believe the best model should not be chosen according to the test result (instead, should be the validation result). And from the best results I could reproduce, the training loss was a little lower than val loss and test loss, and test loss was close to the val loss.

Anyway, thanks for your kind reply, I will try to reproduce the result.