No token padding for train_network

[gesen2egee]

When training with Illustrious-xl 0.1, it appears that token padding was disabled during the training process.
https://arxiv.org/pdf/2409.19946
Illustrious: an Open Advanced Illustration Model A.6

Testing with token padding enabled seems to result in worse outcomes.
To address this issue, I've added the --no_token_padding option to the training network,
allowing users to disable token padding during training for potentially better results.

prompt 1girl, solo
no neg

[kohya-ss]

A.6 states the following:

During training, text encoder outputs must be padded to be packed in batch.

According to this, token padding is performed during training. I believe A.6 is a report that it is better not to perform padding on CFG.

Also disable_token_padding is going to be dropped in the sd3 branch because it requires complex processing.

[feffy380]

As far as I can tell, the Illustrious report is saying the padding tokens unintentionally learned information about the images, which causes adverse effects when they're included during inference especially since most UIs will optimize CFG by inserting padding so cond and uncond can be batched together.

As mentioned above, everything must be padded to the same length for batching to work, so as a fix they suggest masking out the padding tokens during training.
@kohya-ss when masking is implemented for SD3, could this also be extended to SDXL since they use the same CLIP encoders?

[kohya-ss]

@kohya-ss when masking is implemented for SD3, could this also be extended to SDXL since they use the same CLIP encoders?

I hadn't thought of that, but it's certainly possible. However, unlike SD3, there is an option to extend the token length to 150/225 in a bit hacky way. So it seems that the implementation cannot be copied as is...