Add Advanced user guide section with Model Wrappers doc (#219)

docs/index.md

@@ -39,7 +39,7 @@ Support for additional modalities and tasks will be added in future releases.
 
 ### 1. Evaluate your own FMs on public benchmark datasets
 
-With a trained FM as input, you can run ***eva*** on several publicly available datasets & tasks for which ***eva*** provides out-of the box support. One ***eva*** run will automatically download and preprocess the relevant data, compute embeddings with the trained FM, fit and evaluate a classification head and report the mean and standard deviation of the relevant performance metrics the selected task.
+With a trained FM as input, you can run ***eva*** on several publicly available datasets & tasks for which ***eva*** provides out-of the box support. One ***eva*** run will automatically download and preprocess the relevant data, compute embeddings with the trained FM, fit and evaluate a downstream head and report the mean and standard deviation of the relevant performance metrics the selected task.
 
 Supported datasets & tasks include:
 
@@ -80,8 +80,8 @@ The runs used the default setup described in the section below. The table shows
 
 ***eva*** trains the decoder on the "train" split and uses the "validation" split for monitoring, early stopping and checkpoint selection. Evaluation results are reported on the "validation" split and, if available, on the "test" split.
 
-For more details on the FM-backbones and instructions to replicate those results with ***eva***, refer to the [Replicate results section](user-guide/replicate_evaluations.md) 
-in the [User Guide](user-guide/index.md).
+For more details on the FM-backbones and instructions to replicate those results with ***eva***, refer to the [Replicate results section](user-guide/advanced/replicate_evaluations.md) 
+in the [Advanced user guide](user-guide/advanced/index.md).
 
 ## Evaluation setup
 

docs/reference/models/networks.md

@@ -5,6 +5,7 @@ Reference information for the model `Networks` API.
 ::: eva.models.networks.MLP
 
 ## Wrappers
+::: eva.models.networks.wrappers.BaseModel
 ::: eva.models.networks.wrappers.ModelFromFunction
 ::: eva.models.networks.wrappers.HuggingFaceModel
 ::: eva.models.networks.wrappers.ONNXModel

docs/user-guide/advanced/index.md

@@ -0,0 +1,7 @@
+# Advanced user guide
+
+### [Replicate Evaluations](replicate_evaluations.md)
+Find the instructions to completely replicate the evaluation results presented [on the ***eva*** main page](../../index.md)
+
+### [Model Wrappers](model_wrappers.md)
+Explains how to use **eva**'s Model Wrapper API to load models from different formats and sources.

docs/user-guide/advanced/model_wrappers.md

@@ -0,0 +1,83 @@
+# Model Wrappers
+
+
+This document shows how to use **eva**'s [Model Wrapper API](../../../reference/models/networks/#wrappers) (`eva.models.networks.wrappers`) to load different model formats from a series of sources such as PyTorch Hub, HuggingFace Model Hub and ONNX. 
+
+## Loading PyTorch models
+The **eva** framework is built on top of PyTorch Lightning and thus naturally supports loading PyTorch models.
+You just need to specify the class path of your model in the backbone section of the `.yaml` config file.
+
+```
+backbone:
+  class_path: path.to.your.ModelClass
+  init_args:
+    arg_1: ...
+    arg_2: ...
+```
+
+Note that your `ModelClass` should subclass `torch.nn.Module` and implement the `forward()` method to return embedding tensors of shape `[embedding_dim]`.
+
+### PyTorch Hub
+To load models from PyTorch Hub or other torch model providers, the easiest way is to use the `ModelFromFunction` wrapper class:
+
+```
+backbone:
+  class_path: eva.models.networks.wrappers.ModelFromFunction
+  init_args:
+    path: torch.hub.load
+    arguments:
+      repo_or_dir: facebookresearch/dino:main
+      model: dino_vits16
+      pretrained: false
+    checkpoint_path: path/to/your/checkpoint.torch
+```
+
+
+Note that if a `checkpoint_path` is provided, `ModelFromFunction` will automatically initialize the specified model using the provided weights from that checkpoint file.
+
+
+### timm
+Similar to the above example, we can easily load models using the common vision library `timm`:
+```
+backbone:
+  class_path: eva.models.networks.wrappers.ModelFromFunction
+  init_args:
+    path: timm.create_model
+    arguments:
+      model_name: resnet18
+      pretrained: true
+```
+
+
+## Loading models from HuggingFace Hub
+For loading models from HuggingFace Hub, **eva** provides a custom wrapper class `HuggingFaceModel` which can be used as follows:
+
+```
+backbone:
+  class_path: eva.models.networks.wrappers.HuggingFaceModel
+  init_args:
+    model_name_or_path: owkin/phikon
+    tensor_transforms: 
+      class_path: eva.vision.data.transforms.model_output.ExtractCLSFeatures
+```
+
+In the above example, the forward pass implemented by the `owkin/phikon` model returns an output tensor containing the hidden states of all input tokens. In order to extract the state corresponding to the CLS token only, we can specify a transformation via the `tensor_transforms` argument which will be applied to the model output.
+
+## Loading ONNX models
+`.onnx` model checkpoints can be loaded using the `ONNXModel` wrapper class as follows:
+
+```
+class_path: eva.models.networks.wrappers.ONNXModel
+init_args:
+  path: path/to/model.onnx
+  device: cuda
+```
+
+## Implementing custom model wrappers
+
+You can also implement your own model wrapper classes, in case your model format is not supported by the wrapper classes that **eva** already provides. To do so, you need to subclass `eva.models.networks.wrappers.BaseModel` and implement the following abstract methods: 
+
+- `load_model`: Returns an instantiated model object & loads pre-trained model weights from a checkpoint if available. 
+- `model_forward`: Implements the forward pass of the model and returns the output as a `torch.Tensor` of shape `[embedding_dim]`
+
+You can take the implementations of `ModelFromFunction`, `HuggingFaceModel` and `ONNXModel` wrappers as a reference.

docs/user-guide/replicate_evaluations.md → docs/user-guide/advanced/replicate_evaluations.md

@@ -12,7 +12,7 @@ to an FM that produces embeddings without any prior learning on image tasks. To
 ```
 # set environment variables:
 export PRETRAINED=false
-export EMBEDDINGS_DIR="./data/embeddings/dino_vits16_random/<task>"
+export EMBEDDINGS_ROOT="./data/embeddings/dino_vits16_random"
 export DINO_BACKBONE=dino_vits16
 export CHECKPOINT_PATH=null
 export NORMALIZE_MEAN=[0.485,0.456,0.406]
@@ -29,7 +29,7 @@ The next baseline model, uses a pretrained ViT-S16 backbone with ImageNet weight
 ```
 # set environment variables:
 export PRETRAINED=true
-export EMBEDDINGS_DIR="./data/embeddings/dino_vits16_imagenet/<task>"
+export EMBEDDINGS_ROOT="./data/embeddings/dino_vits16_imagenet"
 export DINO_BACKBONE=dino_vits16
 export CHECKPOINT_PATH=null
 export NORMALIZE_MEAN=[0.485,0.456,0.406]
@@ -45,7 +45,7 @@ To evaluate performance on the larger ViT-B8 backbone pretrained on ImageNet, ru
 ```
 # set environment variables:
 export PRETRAINED=true
-export EMBEDDINGS_DIR="./data/embeddings/dino_vitb8_imagenet/<task>"
+export EMBEDDINGS_ROOT="./data/embeddings/dino_vitb8_imagenet"
 export DINO_BACKBONE=dino_vitb8
 export CHECKPOINT_PATH=null
 export NORMALIZE_MEAN=[0.485,0.456,0.406]
@@ -63,7 +63,7 @@ on [GitHub](https://github.com/lunit-io/benchmark-ssl-pathology/releases/). To e
 ```
 # set environment variables:
 export PRETRAINED=false
-export EMBEDDINGS_DIR="./data/embeddings/dino_vits16_lunit/<task>"
+export EMBEDDINGS_ROOT="./data/embeddings/dino_vits16_lunit"
 export DINO_BACKBONE=dino_vits16
 export CHECKPOINT_PATH="https://github.com/lunit-io/benchmark-ssl-pathology/releases/download/pretrained-weights/dino_vit_small_patch16_ep200.torch"
 export NORMALIZE_MEAN=[0.70322989,0.53606487,0.66096631]
@@ -80,7 +80,7 @@ python -m eva predict_fit --config configs/vision/dino_vit/offline/<task>.yaml
 
 ```
 # set environment variables:
-export EMBEDDINGS_DIR="./data/embeddings/dino_vitb16_owkin/<task>
+export EMBEDDINGS_ROOT="./data/embeddings/dino_vitb16_owkin"
 
 # run eva:
 python -m eva predict_fit --config configs/vision/owkin/phikon/offline/<task>.yaml
@@ -98,7 +98,7 @@ on [GitHub](https://github.com/lunit-io/benchmark-ssl-pathology/releases/), run:
 ```
 # set environment variables:
 export PRETRAINED=false
-export EMBEDDINGS_DIR="./data/embeddings/dino_vits16_kaiko/<task>"
+export EMBEDDINGS_ROOT="./data/embeddings/dino_vits16_kaiko"
 export DINO_BACKBONE=dino_vits16
 export CHECKPOINT_PATH=[TBD*]
 export NORMALIZE_MEAN=[0.5,0.5,0.5]
@@ -121,7 +121,7 @@ on [GitHub](https://github.com/lunit-io/benchmark-ssl-pathology/releases/), run:
 ```
 # set environment variables:
 export PRETRAINED=false
-export EMBEDDINGS_DIR="./data/embeddings/dino_vitb8_kaiko/<task>"
+export EMBEDDINGS_ROOT="./data/embeddings/dino_vitb8_kaiko"
 export DINO_BACKBONE=dino_vitb8
 export CHECKPOINT_PATH=[TBD*]
 export NORMALIZE_MEAN=[0.5,0.5,0.5]

docs/user-guide/getting_started.md

@@ -32,4 +32,4 @@ This will:
  - Fit a complete model consisting of the frozen FM-backbone (a pretrained `dino_vits16`) and a downstream head (single layer MLP) on the BACH-train split.
  - Evaluate the trained model on the val split and report the results
 
-To learn more about how to run ***eva*** and customize your runs, familiarize yourself with [How to use ***eva***](how_to_use.md) and get started with [Tutorials](tutorials.md) 
+To learn more about how to run ***eva*** and customize your runs, familiarize yourself with [How to use ***eva***](how_to_use.md) and get started with [Tutorials](tutorials/index.md) 

docs/user-guide/index.md

@@ -8,10 +8,10 @@ Install ***eva*** on your machine.
 
 Get familiar with the different workflows, subcommands and configurations.
 
-### [Tutorials](tutorials.md)
+### [Tutorials](tutorials/index.md)
 
 Run ***eva*** with the different subcommands, replicate our evaluation results and train a supervised ResNet18 from scratch.
 
-### [Replicate evaluations](replicate_evaluations.md)
+### [Advanced user guide](advanced/index.md)
 
-Find the instructions to completely replicate the evaluation results presented [on the ***eva*** main page](../index.md)
+Get to know **eva** in more depth by studying our advanced user guides.

docs/user-guide/tutorials/evaluate_resnet.md

@@ -1,6 +1,6 @@
 # Train and evaluate a ResNet
 
-If you read [How to use eva](../how_to_use.md) and followed the [Tutorials](../tutorials.md) to this point, you might ask yourself why you would not always use the *offline* workflow to run a complete evaluation. An *offline*-run stores the computed embeddings and runs faster than the *online*-workflow which computes a backbone-forward pass in every epoch.
+If you read [How to use eva](../how_to_use.md) and followed the [Tutorials](../tutorials/index.md) to this point, you might ask yourself why you would not always use the *offline* workflow to run a complete evaluation. An *offline*-run stores the computed embeddings and runs faster than the *online*-workflow which computes a backbone-forward pass in every epoch.
 
 One use case for the *online*-workflow is the evaluation of a supervised ML model that does not rely on an backbone/head architecture. To demonstrate this, lets train a ResNet 18 from [Pytoch Image Models (timm)](https://timm.fast.ai/).
 

mkdocs.yml

@@ -36,7 +36,10 @@ nav:
         - user-guide/tutorials/index.md
         - user-guide/tutorials/offline_vs_online.md
         - user-guide/tutorials/evaluate_resnet.md
-      - Replicate evaluations: user-guide/replicate_evaluations.md
+      - Advanced user guide:
+        - user-guide/advanced/index.md
+        - user-guide/advanced/replicate_evaluations.md
+        - user-guide/advanced/model_wrappers.md
   - Reference API:
     - reference/index.md
     - Interface: reference/interface.md

src/eva/models/networks/wrappers/__init__.py

@@ -1,7 +1,8 @@
 """Model Wrappers API."""
 
+from eva.models.networks.wrappers.base import BaseModel
 from eva.models.networks.wrappers.from_function import ModelFromFunction
 from eva.models.networks.wrappers.huggingface import HuggingFaceModel
 from eva.models.networks.wrappers.onnx import ONNXModel
 
-__all__ = ["ModelFromFunction", "HuggingFaceModel", "ONNXModel"]
+__all__ = ["BaseModel", "ModelFromFunction", "HuggingFaceModel", "ONNXModel"]