HIRSBestLineFilter.finalise fails on numpy 1.14 with invalid type promotion

[gerritholl]

Realised through FCDR_HIRS but the problem lies in typhon. On numpy 1.13:

$ plot_hirs_field_timeseries --plot_noise_with_other --without_gain --without_rself --temp_fields --count_fields space --with_corr above --corr_count 6 --corr_timeres 30min --sat noaa15 --channel 1 --from_date 1999-01-01 --to_date 1999-01-2
WARNING  2017-12-18 11:00:34,532 registry._define_single_adder:324: Redefining 'micro' (<class 
'pint.definitions.PrefixDefinition'>)
WARNING  2017-12-18 11:00:34,532 registry._define_single_adder:324: Redefining 'µ' (<class 'pin
t.definitions.PrefixDefinition'>)
INFO     2017-12-18 11:00:36,346 dataset.read_period:373: Reading fcdr_hirs {'satname': 'noaa15
'} for period 1999-01-01 00:00:00  – 1999-01-02 00:00:00
[                                                                     ] N/A% (ETA:  --:--:--) /
dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/ma/core.py:3008: FutureWarning
: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a structured
array. This code will likely break in a future numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).
  if (obj.__array_interface__["data"][0]
/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/ma/core.py:3009: FutureWarnin
g: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a structured
array. This code will likely break in a future numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).
  != self.__array_interface__["data"][0]):
[===================                                                   ]  27% (ETA:  0:00:11) W
ARNING  2017-12-18 11:00:41,122 filters.filter:151: 1 scanlines are out of order, resorting
WARNING  2017-12-18 11:00:41,132 filters.filter:159: 24 duplicate scanlines (judging from scanl
ine number), removing
[======================================================================] 100% (Time: 0:00:13) 

Continues happily beyond this point. But on numpy 1.14, it fails at this point with:

Traceback (most recent call last):
  File "/dev/shm/gerrit/venv/stable-3.6/bin/plot_hirs_field_timeseries", line 11, in <module>
    load_entry_point('FCDR-HIRS==0.1.15+dev23.g827117e', 'console_scripts', 'plot_hirs_field_ti
meseries')()
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/FCDR_HIRS/analysis/timeseri
es.py", line 1370, in main
    temp_fields=p.temp_fields)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/FCDR_HIRS/analysis/timeseri
es.py", line 337, in __init__
    NO_CACHE=True, **hrsargs)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/utils/cache.py", lin
e 54, in wrapper
    return user_function(*args, **kwds)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/dataset.py"
, line 477, in read_period
    arr = of.finalise(arr)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/filters.py"
, line 509, in finalise
    fields_notclose = {nm for nm in rep.dtype.names
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/filters.py"
, line 513, in <setcomp>
    else numpy.isclose(rep[nm][0, ...], rep[nm])
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/core/numeric.py", lin
e 2329, in isclose
    dt = multiarray.result_type(y, 1.)
TypeError: invalid type promotion

There may be other places where typhon functionality fails on numpy 1.14. I suspect this is due to me doing something inappropriate to which numpy has become less tolerant, but this remains to be seen.

[gerritholl]

Numpy 1.14 was released yesterday so this may become more pressing now. Until this is fixed, anyone relying on this functionality will have to downgrade to numpy 1.13.3.