SCALE + ASH methods

README.md

@@ -177,6 +177,8 @@ Currently, **oodeel** includes the following baselines:
 | NMD | [Neural Mean Discrepancy for Efficient Out-of-Distribution Detection](https://openaccess.thecvf.com/content/CVPR2022/html/Dong_Neural_Mean_Discrepancy_for_Efficient_Out-of-Distribution_Detection_CVPR_2022_paper.html) | CVPR 2022 | planned |
 | Gram | [Detecting Out-of-Distribution Examples with Gram Matrices](https://proceedings.mlr.press/v119/sastry20a.html) | ICML 2020 | avail [tensorflow](docs/notebooks/tensorflow/demo_gram_tf.ipynb) or  [torch](docs/notebooks/torch/demo_gram_torch.ipynb) |
 | GEN | [GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection](https://openaccess.thecvf.com/content/CVPR2023/html/Liu_GEN_Pushing_the_Limits_of_Softmax-Based_Out-of-Distribution_Detection_CVPR_2023_paper.html) | CVPR 2023 | avail [tensorflow](docs/notebooks/tensorflow/demo_gen_tf.ipynb) or [torch](docs/notebooks/torch/demo_gen_torch.ipynb) |
+| ASH | [Extremely Simple Activation Shaping for Out-of-Distribution Detection](http://arxiv.org/abs/2209.09858) | ICLR 2023 | avail [tensorflow](docs/notebooks/tensorflow/demo_ash_tf.ipynb) or [torch](docs/notebooks/torch/demo_ash_torch.ipynb) |
+| SCALE | [Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement](https://arxiv.org/abs/2310.00227) | ICLR 2024 | avail [tensorflow](docs/notebooks/tensorflow/demo_scale_tf.ipynb) or [torch](docs/notebooks/torch/demo_scale_torch.ipynb) |
 | RMDS | [A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection](https://arxiv.org/abs/2106.09022) | preprint | avail [tensorflow](docs/notebooks/tensorflow/demo_rmds_tf.ipynb) or [torch](docs/notebooks/torch/demo_rmds_torch.ipynb) |
 | SHE | [Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy](https://openreview.net/forum?id=KkazG4lgKL) | ICLR 2023 | avail [tensorflow](docs/notebooks/tensorflow/demo_she_tf.ipynb) or [torch](docs/notebooks/torch/demo_she_torch.ipynb) |
 

docs/index.md

@@ -176,6 +176,9 @@ Currently, **oodeel** includes the following baselines:
 | NMD | [Neural Mean Discrepancy for Efficient Out-of-Distribution Detection](https://openaccess.thecvf.com/content/CVPR2022/html/Dong_Neural_Mean_Discrepancy_for_Efficient_Out-of-Distribution_Detection_CVPR_2022_paper.html) | CVPR 2022 | planned |
 | Gram | [Detecting Out-of-Distribution Examples with Gram Matrices](https://proceedings.mlr.press/v119/sastry20a.html) | ICML 2020 | avail [tensorflow](./notebooks/tensorflow/demo_gram_tf.ipynb) or  [torch](./notebooks/torch/demo_gram_torch.ipynb) |
 | GEN | [GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection](https://openaccess.thecvf.com/content/CVPR2023/html/Liu_GEN_Pushing_the_Limits_of_Softmax-Based_Out-of-Distribution_Detection_CVPR_2023_paper.html) | CVPR 2023 | avail [tensorflow](./notebooks/tensorflow/demo_gen_tf.ipynb) or [torch](./notebooks/torch/demo_gen_torch.ipynb) |
+| ASH | [Extremely Simple Activation Shaping for Out-of-Distribution Detection](http://arxiv.org/abs/2209.09858) | ICLR 2023 | avail [tensorflow](docs/notebooks/tensorflow/demo_ash_tf.ipynb) or [torch](docs/notebooks/torch/demo_ash_torch.ipynb) |
+| SCALE | [Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement](https://arxiv.org/abs/2310.00227) | ICLR 2024 | avail [tensorflow](docs/notebooks/tensorflow/demo_scale_tf.ipynb) or [torch](docs/notebooks/torch/demo_scale_torch.ipynb) |
+| RMDS | [A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection](https://arxiv.org/abs/2106.09022) | preprint | avail [tensorflow](docs/notebooks/tensorflow/demo_rmds_tf.ipynb) or [torch](docs/notebooks/torch/demo_rmds_torch.ipynb) |
 | SHE | [Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy](https://openreview.net/forum?id=KkazG4lgKL) | ICLR 2023 | avail [tensorflow](docs/notebooks/tensorflow/demo_she_tf.ipynb) or [torch](docs/notebooks/torch/demo_she_torch.ipynb) |
 
 

mkdocs.yml

@@ -17,6 +17,8 @@ nav:
       - React: notebooks/tensorflow/demo_react_tf.ipynb
       - Gram: notebooks/tensorflow/demo_gram_tf.ipynb
       - GEN: notebooks/tensorflow/demo_gen_tf.ipynb
+      - SCALE: notebooks/tensorflow/demo_scale_tf.ipynb
+      - ASH: notebooks/tensorflow/demo_ash_tf.ipynb
       - RMDS: notebooks/tensorflow/demo_rmds_tf.ipynb
       - SHE: notebooks/tensorflow/demo_she_tf.ipynb
     - OOD Baselines (Torch):
@@ -30,6 +32,8 @@ nav:
       - React: notebooks/torch/demo_react_torch.ipynb
       - Gram: notebooks/torch/demo_gram_torch.ipynb
       - GEN: notebooks/torch/demo_gen_torch.ipynb
+      - SCALE: notebooks/torch/demo_scale_torch.ipynb
+      - ASH: notebooks/torch/demo_ash_torch.ipynb
       - RMDS: notebooks/torch/demo_rmds_torch.ipynb
       - SHE: notebooks/torch/demo_she_torch.ipynb
     - Advanced Topics:

oodeel/extractor/feature_extractor.py

@@ -52,6 +52,11 @@ class FeatureExtractor(ABC):
             Defaults to None.
         react_threshold: if not None, penultimate layer activations are clipped under
             this threshold value (useful for ReAct). Defaults to None.
+        scale_percentile: if not None, the features are scaled
+            following the method of Xu et al., ICLR 2024.
+            Defaults to None.
+        ash_percentile: if not None, the features are scaled following
+            the method of Djurisic et al., ICLR 2023.
     """
 
     def __init__(
@@ -60,13 +65,17 @@ def __init__(
         feature_layers_id: List[Union[int, str]] = [-1],
         input_layer_id: Union[int, str] = [0],
         react_threshold: Optional[float] = None,
+        scale_percentile: Optional[float] = None,
+        ash_percentile: Optional[float] = None,
     ):
         if not isinstance(feature_layers_id, list):
             feature_layers_id = [feature_layers_id]
 
         self.feature_layers_id = feature_layers_id
         self.input_layer_id = input_layer_id
         self.react_threshold = react_threshold
+        self.scale_percentile = scale_percentile
+        self.ash_percentile = ash_percentile
         self.model = model
         self.extractor = self.prepare_extractor()
 

oodeel/extractor/keras_feature_extractor.py

@@ -24,6 +24,7 @@
 from typing import Optional
 
 import tensorflow as tf
+import tensorflow_probability as tfp
 from tqdm import tqdm
 
 from ..datasets.tf_data_handler import TFDataHandler
@@ -55,6 +56,11 @@ class KerasFeatureExtractor(FeatureExtractor):
             Defaults to None.
         react_threshold: if not None, penultimate layer activations are clipped under
             this threshold value (useful for ReAct). Defaults to None.
+        scale_percentile: if not None, the features are scaled
+            following the method of Xu et al., ICLR 2024.
+            Defaults to None.
+        ash_percentile: if not None, the features are scaled following
+            the method of Djurisic et al., ICLR 2023.
     """
 
     def __init__(
@@ -63,6 +69,8 @@ def __init__(
         feature_layers_id: List[Union[int, str]] = [-1],
         input_layer_id: Optional[Union[int, str]] = None,
         react_threshold: Optional[float] = None,
+        scale_percentile: Optional[float] = None,
+        ash_percentile: Optional[float] = None,
     ):
         if input_layer_id is None:
             input_layer_id = 0
@@ -71,6 +79,8 @@ def __init__(
             feature_layers_id=feature_layers_id,
             input_layer_id=input_layer_id,
             react_threshold=react_threshold,
+            scale_percentile=scale_percentile,
+            ash_percentile=ash_percentile,
         )
 
         self.backend = "tensorflow"
@@ -144,6 +154,43 @@ def prepare_extractor(self) -> tf.keras.models.Model:
             )
             # apply ultimate layer on clipped activations
             output_tensors.append(last_layer(x))
+
+        # === If SCALE method, scale activations from penultimate layer ===
+        # === If ASH method, scale and prune activations from penultimate layer ===
+        elif (self.scale_percentile is not None) or (self.ash_percentile is not None):
+            penultimate_layer = self.find_layer(self.model, -2)
+            penult_extractor = tf.keras.models.Model(
+                new_input, penultimate_layer.output
+            )
+            last_layer = self.find_layer(self.model, -1)
+
+            # apply scaling on penultimate activations
+            penultimate = penult_extractor(new_input)
+            if self.scale_percentile is not None:
+                output_percentile = tfp.stats.percentile(
+                    penultimate, 100 * self.scale_percentile, axis=1
+                )
+            else:
+                output_percentile = tfp.stats.percentile(
+                    penultimate, 100 * self.ash_percentile, axis=1
+                )
+
+            mask = penultimate > tf.reshape(output_percentile, (-1, 1))
+            filtered_penultimate = tf.where(
+                mask, penultimate, tf.zeros_like(penultimate)
+            )
+            s = tf.math.exp(
+                tf.reduce_sum(penultimate, axis=1)
+                / tf.reduce_sum(filtered_penultimate, axis=1)
+            )
+
+            if self.scale_percentile is not None:
+                x = penultimate * tf.expand_dims(s, 1)
+            else:
+                x = filtered_penultimate * tf.expand_dims(s, 1)
+            # apply ultimate layer on scaled activations
+            output_tensors.append(last_layer(x))
+
         else:
             output_tensors.append(self.find_layer(self.model, -1).output)
 

oodeel/extractor/torch_feature_extractor.py

@@ -58,6 +58,12 @@ class TorchFeatureExtractor(FeatureExtractor):
             Defaults to None.
         react_threshold: if not None, penultimate layer activations are clipped under
             this threshold value (useful for ReAct). Defaults to None.
+        scale_percentile: if not None, the features are scaled
+            following the method of Xu et al., ICLR 2024.
+            Defaults to None.
+        ash_percentile: if not None, the features are scaled following
+            the method of Djurisic et al., ICLR 2023.
+
     """
 
     def __init__(
@@ -66,13 +72,17 @@ def __init__(
         feature_layers_id: List[Union[int, str]] = [],
         input_layer_id: Optional[Union[int, str]] = None,
         react_threshold: Optional[float] = None,
+        scale_percentile: Optional[float] = None,
+        ash_percentile: Optional[float] = None,
     ):
         model = model.eval()
         super().__init__(
             model=model,
             feature_layers_id=feature_layers_id,
             input_layer_id=input_layer_id,
             react_threshold=react_threshold,
+            scale_percentile=scale_percentile,
+            ash_percentile=ash_percentile,
         )
         self._device = next(model.parameters()).device
         self._features = {layer: torch.empty(0) for layer in self._hook_layers_id}
@@ -149,6 +159,16 @@ def prepare_extractor(self) -> None:
             pen_layer = self.find_layer(self.model, -2)
             pen_layer.register_forward_hook(self._get_clip_hook(self.react_threshold))
 
+        # === If SCALE method, scale activations from penultimate layer ===
+        if self.scale_percentile is not None:
+            pen_layer = self.find_layer(self.model, -2)
+            pen_layer.register_forward_hook(self._get_scale_hook(self.scale_percentile))
+
+        # === If ASH method, scale and prune activations from penultimate layer ===
+        if self.ash_percentile is not None:
+            pen_layer = self.find_layer(self.model, -2)
+            pen_layer.register_forward_hook(self._get_ash_hook(self.ash_percentile))
+
         # Register a hook to store feature values for each considered layer + last layer
         for layer_id in self._hook_layers_id:
             layer = self.find_layer(self.model, layer_id)
@@ -317,6 +337,50 @@ def hook(_, __, output):
 
         return hook
 
+    def _get_scale_hook(self, percentile: float) -> Callable:
+        """
+        Hook that scales activation features.
+
+        Args:
+            threshold (float): threshold value
+
+        Returns:
+            Callable: hook function
+        """
+
+        def hook(_, __, output):
+            output_percentile = torch.quantile(output, percentile, dim=1)
+            mask = output > output_percentile[:, None]
+            output_masked = output * mask
+            s = torch.exp(torch.sum(output, dim=1) / torch.sum(output_masked, dim=1))
+            s = torch.unsqueeze(s, 1)
+            output = output * s
+            return output
+
+        return hook
+
+    def _get_ash_hook(self, percentile: float) -> Callable:
+        """
+        Hook that scales and prunes activation features under a threshold value
+
+        Args:
+            threshold (float): threshold value
+
+        Returns:
+            Callable: hook function
+        """
+
+        def hook(_, __, output):
+            output_percentile = torch.quantile(output, percentile, dim=1)
+            mask = output > output_percentile[:, None]
+            output_masked = output * mask
+            s = torch.exp(torch.sum(output, dim=1) / torch.sum(output_masked, dim=1))
+            s = torch.unsqueeze(s, 1)
+            output = output_masked * s
+            return output
+
+        return hook
+
     def _clean_forward_hooks(self) -> None:
         """
         Remove all the forward hook attached to the model's layers. This function should

oodeel/methods/base.py

@@ -52,16 +52,31 @@ class OODBaseDetector(ABC):
 
     def __init__(
         self,
-        use_react: bool = False,
-        react_quantile: float = 0.8,
-        postproc_fns: List[Callable] = None,
+        use_react: Optional[bool] = False,
+        use_scale: Optional[bool] = False,
+        use_ash: Optional[bool] = False,
+        react_quantile: Optional[float] = None,
+        scale_percentile: Optional[float] = None,
+        ash_percentile: Optional[float] = None,
+        postproc_fns: Optional[List[Callable]] = None,
     ):
         self.feature_extractor: FeatureExtractor = None
         self.use_react = use_react
+        self.use_scale = use_scale
+        self.use_ash = use_ash
         self.react_quantile = react_quantile
+        self.scale_percentile = scale_percentile
+        self.ash_percentile = ash_percentile
         self.react_threshold = None
         self.postproc_fns = self._sanitize_posproc_fns(postproc_fns)
 
+        if use_scale and use_react:
+            raise ValueError("Cannot use both ReAct and scale at the same time")
+        if use_scale and use_ash:
+            raise ValueError("Cannot use both ASH and scale at the same time")
+        if use_ash and use_react:
+            raise ValueError("Cannot use both ReAct and ASH at the same time")
+
     @abstractmethod
     def _score_tensor(self, inputs: TensorType) -> np.ndarray:
         """Computes an OOD score for input samples "inputs".
@@ -191,11 +206,18 @@ def _load_feature_extractor(
         Returns:
             FeatureExtractor: a feature extractor instance
         """
+        if not self.use_ash:
+            self.ash_percentile = None
+        if not self.use_scale:
+            self.scale_percentile = None
+
         feature_extractor = self.FeatureExtractorClass(
             model,
             feature_layers_id=feature_layers_id,
             input_layer_id=input_layer_id,
             react_threshold=self.react_threshold,
+            scale_percentile=self.scale_percentile,
+            ash_percentile=self.ash_percentile,
         )
         return feature_extractor
 

oodeel/methods/energy.py

@@ -59,11 +59,19 @@ class Energy(OODBaseDetector):
     def __init__(
         self,
         use_react: bool = False,
+        use_scale: bool = False,
+        use_ash: bool = False,
         react_quantile: float = 0.8,
+        scale_percentile: float = 0.85,
+        ash_percentile: float = 0.90,
     ):
         super().__init__(
             use_react=use_react,
+            use_scale=use_scale,
+            use_ash=use_ash,
             react_quantile=react_quantile,
+            scale_percentile=scale_percentile,
+            ash_percentile=ash_percentile,
         )
 
     def _score_tensor(self, inputs: TensorType) -> Tuple[np.ndarray]:

oodeel/methods/entropy.py

@@ -52,11 +52,19 @@ class Entropy(OODBaseDetector):
     def __init__(
         self,
         use_react: bool = False,
+        use_scale: bool = False,
+        use_ash: bool = False,
         react_quantile: float = 0.8,
+        scale_percentile: float = 0.85,
+        ash_percentile: float = 0.90,
     ):
         super().__init__(
             use_react=use_react,
+            use_scale=use_scale,
+            use_ash=use_ash,
             react_quantile=react_quantile,
+            scale_percentile=scale_percentile,
+            ash_percentile=ash_percentile,
         )
 
     def _score_tensor(self, inputs: TensorType) -> Tuple[np.ndarray]:

oodeel/methods/gen.py

@@ -51,11 +51,19 @@ def __init__(
         gamma: float = 0.1,
         k: int = 100,
         use_react: bool = False,
+        use_scale: bool = False,
+        use_ash: bool = False,
         react_quantile: float = 0.8,
+        scale_percentile: float = 0.85,
+        ash_percentile: float = 0.90,
     ):
         super().__init__(
             use_react=use_react,
+            use_scale=use_scale,
+            use_ash=use_ash,
             react_quantile=react_quantile,
+            scale_percentile=scale_percentile,
+            ash_percentile=ash_percentile,
         )
         self.gamma = gamma
         self.k = k

oodeel/methods/mls.py

@@ -53,11 +53,19 @@ def __init__(
         self,
         output_activation: str = "linear",
         use_react: bool = False,
+        use_scale: bool = False,
+        use_ash: bool = False,
         react_quantile: float = 0.8,
+        scale_percentile: float = 0.85,
+        ash_percentile: float = 0.90,
     ):
         super().__init__(
             use_react=use_react,
+            use_scale=use_scale,
+            use_ash=use_ash,
             react_quantile=react_quantile,
+            scale_percentile=scale_percentile,
+            ash_percentile=ash_percentile,
         )
         self.output_activation = output_activation