NEWS.md

Flux Release Notes

v0.13

• After a deprecations cycle, the datasets in Flux.Data have been removed in favour of MLDatasets.jl.
• params is not exported anymore since it is a common name and is also exported by Distributions.jl
• flatten is not exported anymore due to clash with Iterators.flatten.
• Remove Juno.jl progress bar support as it is now obsolete.
• Dropout gained improved compatibility with Int and Complex arrays and is now twice-differentiable.
• Notation Dense(2 => 3, σ) for channels matches Conv; the equivalent Dense(2, 3, σ) still works.
• Many utily functions and the DataLoader are now provided by MLUtils.jl.
• The DataLoader is now compatible with generic dataset types implementing MLUtils.numobs and MLUtils.getobs.
• Added truncated normal initialisation of weights.
• The Flux.Diagonal layer is now called Scale, and accepts an activation function.
• loadparams! is replaced by loadmodel! which copies trainable + non-trainable parameters and performs more thorough structural checking

v0.12.10

• Dropout/AlphaDropout now supports user-specified RNGs

v0.12.9

• Fixed incorrect output and added GPU compatibility for AlphaDropout.
• Add trilinear Upsample layer.
• Improved performance of RNNs
• Optimisers now accept an ϵ argument
• Improved handling of complex values inputs while training
• Fixed AlphaDropout

v0.12.8

• Optimized inference and gradient calculation of OneHotMatrixpr

v0.12.7

• Added support for GRUv3
• The layers within Chain and Parallel may now have names.

v0.12.5

• Added option to configure groups in Conv.
• REPL printing via show displays parameter counts.

v0.12.4

• Implemented an Embedding layer based on NNlib.gather and NNlib.scatter.

v0.12.1 - v0.12.3

• CUDA.jl 3.0 support
• Bug fixes and optimizations.

v0.12.0

• Add identity_init.
• Add Orthogonal Matrix initialization as described in Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.
• Added Focal Loss function to Losses module
• The Dense layer now supports inputs with multiple batch dimensions.
• Dense and Conv layers no longer perform implicit type conversion.
• The keyword initW is of Dense layers is now init, to agree with convolutional layers.
• Excise datasets in favour of other providers in the julia ecosystem.
• Added option to set bias to false to eliminating bias from being trained.
• Add CTC loss function to Losses module
• Removed kwarg only constructors for convolutional layers.
• Add sparse initialization as described in Deep learning via Hessian-free optimization.
• Moved GPU CI to use buildkite instead of GitLab
• New Parallel layer adds inception module-like building blocks.
• Feature additions and bug fixes for BatchNorm, LayerNorm, InstanceNorm, and GroupNorm normalization layers
• Added Upsample and PixelShuffle layers
• End of deprecation cycle: loss functions cannot be accessed directly from Flux anymore, they live in the Flux.Losses module. All loss functions perform mean aggregation by default.

v0.11.2

• Adds the AdaBelief optimiser.
• Other new features and bug fixes (see GitHub releases page)

v0.11

• Moved CUDA compatibility to use CUDA.jl instead of CuArrays.jl
• Add kaiming initialization methods: kaiming_uniform and kaiming_normal
• Use DataLoader with NamedTuples, so that tensors can be accessed by name.
• Error if Dense layers weights and biases are not arrays.
• Add Adaptive Pooling in Flux layers.
• Change to DataLoader's constructor
• Uniform loss interface
• Loss functions now live in the Flux.Losses module
• Optimistic ADAM (OADAM) optimizer for adversarial training.
• Add option for same padding to conv and pooling layers by setting pad=SamePad().
• Added option to set bias to Flux.Zeros to eliminating bias from being trained.
• Added GlobalMaxPool and GlobalMeanPool layers for performing global pooling operations.
• Added ClipValue and ClipNorm in this pr to Flux.Optimise to provide a cleaner API for gradient clipping.
• Added new kwarg-only constructors for the various convolutional layers.
• Documented the convolutional layer constructors accepting weight and bias keyword arguments to supply custom arrays for those fields.
• Testing suite improvements now test for gradients of all layers along with GPU support.
• Functors have now moved to Functors.jl to allow for their use outside of Flux.
• Added helper functions Flux.convfilter and Flux.depthwiseconvfilter to construct weight arrays for convolutions outside of layer constructors so as to not have to depend on the default layers for custom implementations.
• dropout function now has a mandatory active keyword argument. The Dropout struct *whose behavior is left unchanged) is the recommended choice for common usage.
• and many more fixes and additions...

v0.10.1 - v0.10.4

See GitHub's releases.

v0.10.0

• The default AD engine has switched from Tracker to Zygote.jl
  • The dependency on Tracker.jl has been removed.
  • This means Flux now does not depend on using a specialised TrackedArray type, and can be used with normal Array implementations directly.
  • Tracker compatibility is maintained in most common cases, but Zygote will be the preferred AD backend for Flux from now on.
• The CUDNN wrappers have been moved from Flux into CuArrays, to allow for better supporting the CUDA backend, and improve user experience, not to mention making Flux lean.
• *crossentropy functions now work as expected with CuArrays. PR for binarycrossentropy.
• Added clearer docs around training and the Optimiser interface.
• Layer initialisations have been improved with a clearer API on how to extend it for other purposes.
• Better messaging around CUDA availability, with hooks to initialize the GPU as default where possible.
• @treelike has been formalised as a functor, with an effective deprecation.
• testmode! is deprecated in favour of istraining

v0.9.0

• Depthwise convolutional layer API changes from in => mult channel specification to in => out channel specification, and deprecates implicit out constructor.
• New SkipConnection, which can be used to train residual neural network architectures.
• New RADAM optimiser.

v0.8.0

• Dropout now has a dims argument for specifying the unbroadcast dimensions.
• New ConvTranspose layer.
• New Maxout layer
• Datasets are now hash verified on download to avoid corruption.
• We now zero the initial state for RNNs.
• Normalisation can now work on arbitrary dims.
• Many docs and bugfixes thanks to @KristofferC and others.
• NamedTuples now work like Tuples when doing mapleaves.
• New "performance tips" section of the docs.
• The training loop is now more readable and better shows how to use the lower-level APIs.
• New AlphaDropout.
• Data.Iris makes Fisher's Iris dataset available with Iris.labels and Iris.features.
• New InstanceNorm, as popularized by Instance Normalization: The Missing Ingredient for Fast Stylization.
• New GroupNorm, as described in Group Normalization.
• New CrossCor.

AD Changes:

• det, logdet and logabsdet now have adjoints.
• Support for PermuteDimsArray.
• Flux.Tracker is now its own package, in preparation for replacing it with Zygote.

v0.7.0

Despite the heroic efforts of scholars and archeologists, pre-0.7 history is lost to the sands of time.