Task does not use GPU after a period of time (Apple Sillicon)

[Sora-dcliu]

Note: For general usage questions and help, please use either our FaceSwap Forum
or FaceSwap Discord server. General usage questions are liable to be closed without
response.

Crash reports MUST be included when reporting bugs.

Describe the bug
In GUI, after training the model for about 1 hour,  The GPU usage becomes 0%, and the iteration is slow. But at the beginning of training, the GPU usage can reach about 90%.
The computer didn't sleep and the monitor wasn't turned off.

To Reproduce
Steps to reproduce the behavior:

1. python faceswap.py gui
2. choose input A B and model dir
3. Click "Generate" and click "Train"

Expected behavior
GPU can always work for training

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOS M1 MAX
• Python Version 3.9
• Conda Version [4.12.0]
• Commit ID [e.g. e83819f]

Additional context
Add any other context about the problem here.

Crash Report
The crash report generated in the root of your Faceswap folder

[torzdf]

I don't use Apple Silicon, but will leave this open in case any Apple Silicon users have any insights

[Sora-dcliu]

Correction: In fact, when running for about 3 hours and about 66500 iterations, the change stops using the GPU. And the program starts to get very slow.

[geewiz94]

I have observed this as well, for there's a 50/50 chance for the GPU to stop during a training session. Sometimes it happens after 1 hour, sometimes I can train for 24 hours without issue, using the same model/settings. I'm not sure what the cause is, or how to find out..

[torzdf]

If I were to guess, I would suspect that this is some kind of internal throttling. This is most likely to be a Tensorflow/M1 issue rather than specifically Faceswap though. Possibly related: https://developer.apple.com/forums/thread/708154