[Feature request/question] Partial interpolation / cut-outs + batching transformations

[choltz95]

Hi, Thank you so much for this!

I was having trouble following the interpolation functionality. I am interested in interpolating/blending two images, but restricting the blending only to certain (maybe randomly selected) regions.

Given a specified volume/area, is it possible to generate a mask with the position of the "unmasked" region chosen randomly? Kind of similar in spirit to random_crop.

A simplified use-case is doing cut-outs for augmentation.

Is this in the scope of this project / possible to do currently?

Separately, is there an example or easy way to apply a sequence of [random] transformations - e.g. on to batches of inputs.

Thanks!

[claudiofantacci]

Hi @choltz95, sorry for the delay in getting back to you, this has somewhat fallen under my radar 🙃

Let's go in order:

• re interpolation/blending: the interpolation function that pix provides follows scipy.map_coordinates (but it's more friendly on accelerators in JAX-world). This means that it is not really though-out for blending to images out of the box (a la cv.addWeighted). If all you want is blending, that should be quite easy to implement as also described in the above OpenCV function description. Otherwise, am I missing something?
• re mask position/coordinates: I might again miss the point here, but random mask at image coordinates can be generated drawing random numbers in a way that indices are not out of bound. Then, assuming you are in JAX-land, to mask portion of an image you can use jax.lax.dynamic_slice. We are working right now at an implementation that replicates tf.image.random_crop so that will probably come to help here, but we don't have an ETA for that (maybe a few weeks?).
• re scope: blending could be in the scope of this library, but I've not seen use cases to allocate time implementing that, given that the basic operation should be as simple as a weighted sum (see OpenCV implementation above). Interpolation is already implemented, while random crops are coming soon :-) Hope this makes sense!
• re sequence of random transformations: there are no examples for this. We acknowledge that other frameworks provide APIs to apply transformations in sequence, but we have not decided yet whether to provide this functionality or not. The debate is about the added benefit of such a functionality, when applying sequence of transformations now simply implies to call functions one after another 🤔

[CloudyDory]

Hi, is there a way to apply transformation to a batch of images that the OP has also mentioned? Specifically, about apply the same type of transformation, but with random parameters to each sample in a batch?

[claudiofantacci]

Hi! Depends on the transformation, but in principle what I would do is to setup a transformation with random parameters to work on a single image, and then I would jax.vmap it over a batch, making sure to pass different random seeds in so that each element in the batch will have a different set of random parameters. Hope this makes sense 😄