GPU VRAM load slowly increases

[GiannisPikoulis]

Hello.

Thank you for your code. I am executing clip_finetune() on CelebA-HQ-256x256 and I am monitoring my GPU VRAM usage. I am noticing a gradual increase in VRAM while precomputing the latents from the given dataset. Is this normal? Also which part of the code is responsible for this behavior. I thought VRAM usage should remain steady throughout the training procedure and reach its peak from the beginning. Also given this behavior, for a large enough number of n_precomp_img this will eventually lead in memory overflow which is definitely not desired.

Thanks in advance.

[hniksoleimani]

it happened to me and the fineuning process stopped, also using --clip_finetine_eff eventually leads to another error:

"ValueError: Expected tensor to be a tensor image of size (C, H, W). Got tensor.size() = torch.Size([1, 3, 256, 256])."