Terminate and remove stale MLFlow runs

[ryanjdillon]

Currently, if MLFlow logging fails, the newly generated run is left in an active status. Since these will have either missing logs entirely or partial logs, we could perhaps add removal of these in some error handling.

At the very least these should be put into a terminated states; however being left "active" appears to have no bearing on anything other than being possibly leading to some confusion.

[ryanjdillon]

For example, here are start and end timestamps for the mlflow experiment (gordo machine) 25c8dd76-73f0-4a23-a2ab-7710492e5a39-9999:

2020-01-07 09:46:37.099000 2020-01-07 09:46:37.288000
2020-01-15 08:27:05.488000 2020-01-15 08:27:06.992000
2020-01-16 09:38:57.141000 nada
2020-01-16 09:50:02.722000 nada
2020-01-23 05:00:55.899000 2020-01-23 05:00:58.580000
2020-01-23 05:31:51.467000 2020-01-23 05:31:54.821000
2020-01-23 05:38:03.344000 2020-01-23 05:38:07.931000
2020-01-23 05:48:13.310000 2020-01-23 05:48:16.649000
2020-01-23 05:57:46.336000 2020-01-23 05:57:51.378000
2020-01-23 06:30:02.369000 2020-01-23 06:30:06.674000
2020-01-23 06:49:35.559000 2020-01-23 06:49:38.986000
2020-01-24 12:01:19.015000 2020-01-24 12:01:22.637000
2020-01-28 09:43:50.239000 2020-01-28 09:43:52.902000
2020-01-28 10:00:00.554000 2020-01-28 10:00:02.990000
2020-01-31 20:27:39.926000 2020-01-31 20:27:45.795000
2020-01-31 21:11:49.262000 2020-01-31 21:11:52.282000
2020-02-04 08:52:20.377000 2020-02-04 08:52:29.920000
2020-02-04 09:35:30.304000 2020-02-04 09:35:35.726000
2020-02-07 08:00:59.088000 2020-02-07 08:01:34.463000
2020-02-10 13:39:09.111000 2020-02-10 13:39:11.118000
2020-02-13 16:53:37.565000 nada
2020-02-13 18:23:06.276000 nada
2020-02-14 06:26:54.376000 nada
2020-02-14 12:55:22.771000 nada
2020-02-14 13:13:44.181000 nada
2020-02-14 13:27:33.930000 nada
2020-02-14 13:34:15.251000 nada
2020-02-14 13:36:36.870000 nada
2020-02-14 13:41:42.174000 nada
2020-02-14 13:42:07.875000 nada
2020-02-14 13:42:30.290000 nada
2020-02-14 13:42:47.386000 nada
2020-02-14 15:27:18.512000 nada
2020-02-14 15:57:31.498000 nada
2020-02-17 07:21:51.337000 nada
2020-02-17 11:31:59.025000 nada
2020-02-17 11:32:22.872000 nada
2020-02-17 11:32:37.991000 nada
2020-02-17 11:32:48.059000 nada
2020-02-17 11:33:01.208000 nada
2020-02-17 11:33:19.283000 nada
2020-02-18 08:28:12.018000 2020-02-18 08:28:14.314000