Huge performance degradation on Py 3.12

[MRigal]

Hi guys,

Maybe due to the changes introduced in v1.3.0, switching to Python 3.12 (from 3.11) almost doubled our test suite time in CI. Reporting just so that other are also informed by the issue.
We switched to time-machine and it limited to extra-time by less than 10%.

Best and thanks for this library which helped me a lot many years!