Baal 2.0: Experiment API

Quite a large update with a renewed experience for researchers: Experiment API.

Nonetheless, production users will benefit from EPIG, the new SotA in Active Learning!

Experiment API
The new experiment object will simplify your experiments by reducing it to a 1-liner.

EPIG
Contribution from @reeshipaul @Dref360 and @fbickfordsmith, EPIG is the new state-of-the-art in Bayesian Active Learning.
It better estimates the predictive uncertainty than PowerBALD and considers the labeled training set.

Stopping Criterion
A new object that helps you stop an experiment when reaching your labeling budget or a plateau in uncertainty/performance.

Breaking Changes

• Arguments in ModelWrapper.*_on_dataset have been for the most part moved to TrainingArguments and are included in ModelWrapper constructor. Visit the documentation for more details.

What's Changed

• Add Stopping Criteria for loop by @Dref360 in #286
• Add EPIG by @reeshipaul @Dref360 @fbickfordsmith in #293
• Add Args to ModelWrapper to simplify common API by @Dref360 in #294
• Add documentation for criterion by @Dref360 in #297
• Experiment API v2 by @Dref360 in #296

Full Changelog: v1.9.2...v2.0.0

v1.9.2

What's Changed

• Allow HF to not replicate in memory by @Dref360 in #281
• Run poetry.lock by @Dref360 in #282
• Fix typo in logging and bump structlog by @Dref360 in #285

Full Changelog: v1.9.1...v1.9.2

v1.9.1 Heuristic Aliases!

Full Changelog: v1.9.0...v1.9.1

v1.9.0

What's Changed

• Handle NER usecase by @Dref360 in #263
• Add warnings when using replicate_in_memory=True and MCCachingModule by @Dref360 in #273
• weights=None in vgg models by @arthur-thuy in #276
• fix: no pool preds for random heuristic by @arthur-thuy in #277

Full Changelog: v1.8.0...v1.9.0

V1.8.0

New features

• MCCachingModule: Cache computation between Monte Carlo iteration link. Special thanks to @pieterblok for introducing this capability!
• We can now evaluation every x epochs in ModelWrapper.train_and_test_on_datasets.

What's Changed

• Fix doc for NLP example by @Dref360 in #241
• #246 Fix issue where training was not kept consistent by @Dref360 in #249
• Baal in Production Notebook | Classification | NLP | Hugging Face by @nitish1295 in #245
• Add code walkthrough for CIFAR10 by @Dref360 in #253
• Deprecate shuffle prop in favor of PowerSampling by @Dref360 in #260
• docs: use correct link to experiment script by @arthur-thuy in #267
• added setting to skip x epochs when using train_and_test_on_datasets by @studentWUR in #258
• Add caching mechanism for MCDropout by @Dref360 in #268
• Improve get_indices_for_active_step by caching the result according t… by @Dref360 in #269

New Contributors

• @nitish1295 made their first contribution in #245
• @arthur-thuy made their first contribution in #267
• @studentWUR made their first contribution in #258

Deprecation

We will deprecate shuffle_prop in favor of PowerSampling. Look here to know more about PowerBALD!

Full Changelog: v1.7.0...v1.8.0

v1.7.0

Large release where we added support for torchmetrics, moved to mkdocs and made the app more robust!

We also updated our visual identity, huge thanks to @Donasolo for the new logos!

What's Changed

• Move to Github Actions by @Dref360 in #212
• Added None to act as max for np.clip by @GeorgePearse in #213
• Assign version to pytorch-lightning by @Dref360 in #226
• Update README and CONTRIBUTING by @Dref360 in #222
• Add metric logging by @Dref360 in #223
• Throw away value for custom dataloaders by @bresilla in #228
• #232 Allow for Python <4 by @Dref360 in #233
• #156 Support torchmetrics by @Dref360 in #230
• Add warning when data augmentation is applied on the pool by @Dref360 in #229
• Add structure for mkdocs instead of Sphinx by @Dref360 in #225

New Contributors

• @GeorgePearse made their first contribution in #213
• @bresilla made their first contribution in #228

Full Changelog: v1.6.0...v1.7.0

v1.6.0

What's Change

• Update faq.md by @Dref360 in #203
• #205 Add stochastic heuristics from Kirsch et al. by @Dref360 in #206

Full Changelog: v1.5.2...v1.6.0

V1.5.2

What's Changed

• Better indexing support arrow dataset by @parmidaatg in #183
• Raises an error instead of a warning when label has no label by @Dref360 in #187
• #192 Use configure instead of configure_once to remove warnings by @Dref360 in #193
• #190 Fix MRO for Lightning examples and deprecate said example by @Dref360 in #191
• Can easily unpatch modules and use as a context manager #198 #194 :

mc_dropout_model = MCDropoutModule(your_model)
# this is stochastic
predictions = [mc_dropout_model(input) for _ in range(ITERATIONS)]

model = mc_dropout_model.unpatch()
# this is deterministic
output = model(input)


# Context manager
with MCDropoutModule(your_model) as model:
    # this is stochastic
    predictions = [model(input) for _ in range(ITERATIONS)]
# this is deterministic
output = model(input)

Full Changelog: 1.5.1...v1.5.2

v1.5.1

What's Changed

• Solve bug where a dataset wouldnt be able to label by @Dref360 in #178

Full Changelog: 1.5.0...1.5.1

v1.5.0

What's Changed

• Split API documentation in multiple files by @Dref360 in #158
• Lightning Flash example by @parmidaatg in #154
• Add bandit to CircleCI by @Dref360 in #164
• Add Label function to HFDataset @Dref360 in #165
• #161 update query size by @Dref360 in #166
• Same fix in dropconnect and consistent dropout as in dropout by @Dref360 in #172
• Add last_active_step iteration to iterate over the last N active steps. by @Dref360 in #174

Deprecated

1. We now deprecate our PL integration in favor of Lightning Flash. More information to come.
2. We renamed n_data_to_label to query_size to match academic papers notation.

Full Changelog: 1.4.0...1.5.0