nan values during training

[BehzadBozorgtabar]

Hello,

When I am trying your training code for my data, I will get nan values for the total_loss and sample_energy.

Any suggestion?

[LuyuanLi97]

Hi！Have you solved this problem?

[chiachen-chang]

If your data does not include NaN or Inf values, the problem may occur due to the calculation of the relative Euclidean distance during the reconstruction process.

    relative_euclidean_distance = (x-x_hat).norm(2, dim=1) / x.norm(2, dim=1)
        cosine_similarity = F.cosine_similarity(x, x_hat, dim=1)
        return relative_euclidean_distance, cosine_similarity

The reconstructed output x_hat may contain all zero values, which can occur because we have limited control over the decoding process. To address this issue, we can consider adding a minimum value to the output. The code snippet for this approach is provided below：

epsilon = 1e-10
euclidean_distance = (x_reshaped - x_hat_reshaped).norm(2,dim=1) / (x_reshaped.norm(2,dim=1)+epsilon)