Training LatentInpaintDiffusion model on my datasets, but the results are poor, how can i solve it?

[LM7XAw]

Thank you for your great work on this model! I'm training it on my own defect dataset, but the results are not coming out well. What might be the cause of this? How can I improve it?

I'm using the configuration file configs/stable-diffusion/v2-inpainting-inference.yaml with the SD 2.0-inpainting checkpoint as the pre-trained model, but the output is still not satisfactory.

input_gs:

masked_image:

sample_gs:

these images are scaled and padded to $512\times512$, and I randomly mask either the defect areas, the entire object, or a random region in the image.

During training, the loss initially decreases (see sample_gs), but then it starts increases again, and the generated images end up as pure noise.

I'm training on two 3090 with a batch size of 2. I've tried changing the area of the defect mask, prompt, and learning rate, but the results are still not very satisfactory. Any tips for improving the results?