[ENH] Parallelized cross validation & other evaluation methods

[kernc]

Parallel multiprocess execution of model evaluation methods.

Assumptions:

• there is some n_cpu * (data.nbytes + C) extra RAM available,
• passed arguments (learners, data) are picklable, else it fallbacks to one single thread.

[kernc]

16cc875#commitcomment-15862760

[codecov-io]

Current coverage is 87.59%

Merging #1004 into master will decrease coverage by 0.12%

@@             master      #1004   diff @@
==========================================
  Files            75         75          
  Lines          7419       7458    +39   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           6508       6533    +25   
- Misses          911        925    +14   
  Partials          0          0          

1. 3 files (not in diff) in Orange/data were modified. more
   • Misses +2
   • Hits -26
2. 2 files (not in diff) in ...range/classification were modified. more
   • Hits +9
3. 5 files (not in diff) in Orange were modified. more
   • Misses +13
   • Hits -16

Powered by Codecov. Last updated by ae82d28...66531a4

[kernc]

Should get some testing on Macintosh OS.

[BlazZupan]

Please add JOBLIB_START_METHOD=forkserver for MacOS X.

[kernc]

There's a lot room for improvement in terms of condensing that code. But for now ...

Please test thoroughly. Someone (@ajdapretnar, @thocevar) should try it on Windows too.

[ajdapretnar]

I get this on Windows (zoo.tab, SVM classifier).

File "d:\orange\orange3\Orange\widgets\evaluate\owtestlearners.py", line 573, in apply
self._update_results()
File "d:\orange\orange3\Orange\widgets\evaluate\owtestlearners.py", line 391, in _update_results
self.data, learners, **common_args)
File "d:\orange\orange3\Orange\evaluation\testing.py", line 458, in init
with joblib.Parallel(n_jobs=n_jobs, backend=ctx) as parallel:
AttributeError: exit

[kernc]

Should work now.

[thocevar]

The reason for "AttributeError: exit" was an old version of joblib (0.8 instead of 0.9). Otherwise, it works for me.

[kernc]

Blocked on #1114.

[kernc]

@BlazZupan, @astaric This is ready for another review.

Not sure why these lines don't cover. An appropriate test has been added.

[astaric]

For me, the gui still freezes while the cross valudation is being executed. Would it be possible to do it without blocking the gui thread?

(They way widget that download data, like https://github.com/biolab/orange-bio/blob/master/orangecontrib/bio/widgets3/OWBioMart.py#L710, do it?)

[BlazZupan]

Causes a RuntimeError

Can't run multiprocessing code without a __main__ guard.

It works with n_jobs=1, or with the main guard, but I guess the intended behaviour is that CrossValidation should work as before, unchanged, without main guard.

[kernc]

The default is now n_jobs=1 so it should work as before.

@astaric joblib complains: 😬

UserWarning: Multiprocessing backed parallel loops cannot be nested below threads, setting n_jobs=1

FWIW, it doesn't stall at all with 'fork' backend e.g. on Linux. 😃

[kernc]

@ajdapretnar can you confirm this works well on Windos and just merge it if it does. Thanks. 😄