Pandas datetime to timedelta conversion will now raise

[Casper-Guo]

Following our discussion in #674, Pandas is now changing the behavior to raising an error (ref: pandas-dev/pandas#60882).

I am guessing that we are relying on similar behaviors elsewhere in the code too and we should go through and fix them.

Update: Merged at pandas-dev/pandas#60882 and scheduled for release in Pandas v3.0.0

[theOehrly]

Given that this is merged, I think it is sufficient to run all tests locally with a nightly build of pandas. This will catch anything but weird edge cases (like the one in #674, I know...), but manually checking this seems like a lot of work.