Non-local and Total variance minimization

[le000043]

Hello,

Thank you for building and maintaining such a wonderful software package. I am using Non-local (NL) and Total variance minimization (TVM) as preprocessors for some Tensorflow 2 classifiers on the 224x224x3 dataset. These preprocessors prove to be very slow and I suspect this is due to the lack of GPU support for both preprocessors. Are there any solutions to speed these two defenders?

Thanks a lot

[beat-buesser]

Hi @le000043 Thank you very much! Can you share how you have implemented the preprocessors?

[le000043]

this is my current integration for Non-local Mean to ART. This is for the gray-box threat model; therefore, it is not necessary for component MyNonLocalDenoiser to extend ART's Preprocessor component

class MyNonLocalDenoiser():
  def __init__(self, resolution, patch_size=7, patch_distance=11, h=0.1,
    fast_mode=True, channel_axis=None):
    self.patch_size = patch_size
    self.patch_distance = patch_distance
    self.h = h
    self.fast_mode = fast_mode
    self.channel_axis = channel_axis
    self.resolution = resolution
  def __call__(self, x: np.ndarray, y: Optional[np.ndarray] = None) -> Tuple[np.ndarray, Optional[np.ndarray]]:
    small_x = tf.image.resize(x, (32, 32), method = 'bilinear')
    denoised_small_img = denoise_nl_means(
      image = small_x, 
      fast_mode = self.fast_mode, 
      patch_distance = self.patch_distance, 
      h = self.h,
      channel_axis = self.channel_axis)
    denoised_x = (tf.image.resize(denoised_small_img, (self.resolution[0], self.resolution[1]), method = 'bilinear'))
    return denoised_x, y
    ```

[beat-buesser]

It might work, but it would be better to inherit from Preprocessor and implement the methods that your scenario requires. ART is checking the type of preprocessors at various locations. To make your preprocessor framework-specific to see if it runs faster try inheriting and implementing PreprocessorTensorFlowV2 which is a TensorFlow-specific extension of Preprocessor.

[le000043]

Awesome, thanks! Are there any ways we can speed up TVM or enabling GPU support for it ?

[beat-buesser]

I think it should be possible. You would have to transform the numpy-based TVM code in the existing Preprocessor implementation into a TensorFlow-based TVM implementation inheriting from PreprocessorTensorFlowV2.

[le000043]

thanks!