Bump NAML version, fix evaluation sparse target

This update bumps NAML from pointing to a commit on a fork to the stable NAML release. This stable NAML release addresses several issues, most importantly an issue that introduced a memory leak which lead to high failure rate, and the ability to get stuck in an infinite loop. Without these fixes, NAML was too unstable to evaluate.

This also includes a small patch to address a bug where if the target was provided in sparse format and returned as such in the integration script, the evaluation script would crash.

v2.1.6 - Fixes

Fixes:

• Set task type explicitly for naive automl
• Unsparsify target variable for naive automl (required to work wit sparse data)
• Use numpy data for autosklearn if the pandas dataframe is sparse, as sparse dataframes are not supported (yet).

Use different NAML version instead

The previous one had a fixed based on master, but that contains a bug for regression datasets. So instead we point to a version, as far as I can tell, has neither bug.

v2.1.4

Adds NaiveAutoML (https://github.com/fmohr/naiveautoml) as new integrated framework!
The framework isn't really designed to run for a long time, and currently may encountered segmentation faults.

v2.1.3

• Store more artifacts by default
• Retry on AWS VolumeLimitExceeded
• Fix issues with storing autosklearn artifacts and add print logging to work around their logging configuration.

v2.1.2

What's Changed

• Add an option to keep columns with all missing data around when using impute_array
• Fix a bug where inference batches were not generated correctly if the data contained columns with all missing values.
• Fix a bug where the arff header of the split arff files incorrectly could label booleans as numeric when they should be treated as categorical (this only affected frameworks that depend on ARFF).

Full Changelog: v2.1.1...v2.1.2

v2.1.1

What's Changed

AWS:

• start, stop, and log time to failures.csv log

Docker:

• No longer assign user and user permissions when creating docker
• Introduce docker.run_as configuration option, which lets you specify under which user the docker container should execute the benchmark script.
• Further cut down on the files included in the docker image

Frameworks:

• Add additional logging to framework integration scripts

AutoGluon:

• reduce maximum runtime for good_quality and high_quality presets, which otherwise exceed the runtime by design
• allow larger models to persist in memory, this matches an upcoming default

GAMA:

• update for 23.0.0 release

Full Changelog: v2.1.0...v2.1.1

v2.1.0

Highlights:

• The benchmark now requires Py3.9+ and its dependencies are updated.
• AMIs and Docker images now use Ubuntu 22.04
• Upgrades support for newer versions of the various frameworks.
• Support for uploading results to OpenML and connecting to the OpenML test server
• Experimental support for time series with AutoGluon
• Results can now be stored incrementally
• Add option to measure inference time in more standardized fashion for most frameworks.

Note that sharing built docker images currently has some complications due to permission issues, as a work around patch start as root (see also: #495 (comment))
GAMA integration is currently broken, as the goal parameter was incorrectly removed in the last release, this will be fixed next GAMA release.

Thanks to all contributors!

Full Changelog: v2.0.6...v2.1.0

v2.0.5

What's Changed

• Signal to encode predictions as proba now works by @PGijsbers in #447
• Monkeypatch openml to keep whitespace in features by @PGijsbers in #446
• fix for mlr3automl installation by @Coorsaa in #443

Full Changelog: v2.0.4...v2.0.5

v2.0.4

What's Changed

• Fix a bug which could prevent building docker images by @sebhrusen in #437
• Avoid querying terminated instance with CloudWatch by @PGijsbers in #438
• Add precision to runtimes in results.csv by @ledell in #433
• Iteratively build the forest to honor constraints by @PGijsbers in #439
• Iterative fit for TunedRandomForest to meet memory and time constraints by @PGijsbers in #441

Full Changelog: v2.0.3...v2.0.4