Add Advanced user guide section with Model Wrappers doc

docs/index.md

@@ -39,7 +39,7 @@ Support for additional modalities and tasks will be added in future releases.
 
 ### 1. Evaluate your own FMs on public benchmark datasets
 
-With a trained FM as input, you can run ***eva*** on several publicly available datasets & tasks for which ***eva*** provides out-of the box support. One ***eva*** run will automatically download and preprocess the relevant data, compute embeddings with the trained FM, fit and evaluate a classification head and report the mean and standard deviation of the relevant performance metrics the selected task.
+With a trained FM as input, you can run ***eva*** on several publicly available datasets & tasks for which ***eva*** provides out-of the box support. One ***eva*** run will automatically download and preprocess the relevant data, compute embeddings with the trained FM, fit and evaluate a downstream head and report the mean and standard deviation of the relevant performance metrics the selected task.
 
 Supported datasets & tasks include:
 

docs/user-guide/index.md

@@ -2,5 +2,7 @@
 
 - [Getting started](getting_started.md)
 - [How to use eva](how_to_use.md)
-- [Tutorials](tutorials.md)
-- [Replicate evaluations](replicate_evaluations.md)
+- [Tutorials](tutorials/online_vs_offline.md)
+    - [Online v.s. Offline Evaluations](tutorials/online_vs_offline.md)
+    - [Model Wrappers](tutorials/model_wrappers.md)
+- [Replicate evaluations](replicate_evaluations.md)

docs/user-guide/tutorials/model_wrappers.md

@@ -0,0 +1,83 @@
+# Model Wrappers
+
+In this Tutorial, we demonstrate how to use `eva`'s model wrapper API (`eva.models.networks.wrappers`) to load different model formats from a series of sources such as PyTorch Hub, HuggingFace Model Hub and ONNX. 
+
+
+## Loading PyTorch models
+The `eva` framework is built on top of PyTorch Lightning and thus naturally supports loading PyTorch models.
+You just need to specify the class path of your model in the backbone section of the `.yaml` config file.
+
+```
+backbone:
+  class_path: path.to.your.ModelClass
+  init_args:
+    arg_1: ...
+    arg_2: ...
+```
+
+Note that your `ModelClass` should subclass `torch.nn.Module` and implement the `forward()` method to return embedding tensors of shape `[embedding_dim]`.
+
+### PyTorch Hub
+To load models from PyTorch Hub or other torch model providers, the easiest way is to use the `ModelFromFunction` wrapper class:
+
+```
+backbone:
+  class_path: eva.models.networks.wrappers.ModelFromFunction
+  init_args:
+    path: torch.hub.load
+    arguments:
+      repo_or_dir: facebookresearch/dino:main
+      model: dino_vits16
+      pretrained: false
+    checkpoint_path: path/to/your/checkpoint.torch
+```
+
+
+Note that if a `checkpoint_path` is provided, `ModelFromFunction` will automatically initialize the specified model using the provided weights from that checkpoint file.
+
+
+### timm
+Similar to the above example, we can easily load models using the common vision library `timm`:
+```
+backbone:
+  class_path: eva.models.networks.wrappers.ModelFromFunction
+  init_args:
+    path: timm.create_model
+    arguments:
+      model_name: resnet18
+      pretrained: true
+```
+
+
+## Loading models from HuggingFace Hub
+For loading models from HuggingFace Hub, `eva` provides a custom wrapper class `HuggingFaceModel` which can be used as follows:
+
+```
+backbone:
+  class_path: eva.models.networks.wrappers.HuggingFaceModel
+  init_args:
+    model_name_or_path: owkin/phikon
+    tensor_transforms: 
+      class_path: eva.vision.data.transforms.model_output.ExtractCLSFeatures
+```
+
+In the above example, the forward pass implemented by the `owkin/phikon` model returns an output tensor containing the hidden states of all input tokens. In order to extract the state corresponding to the CLS token only, we can specify a transformation via the `tensor_transforms` argument which will be applied to the model output.
+
+## Loading ONNX models
+`.onnx` model checkpoints can be loaded using the `ONNXModel` wrapper class as follows:
+
+```
+class_path: eva.models.networks.wrappers.ONNXModel
+init_args:
+  path: path/to/model.onnx
+  device: cuda
+```
+
+## Implementing custom model wrappers
+
+You can also implement your own model wrapper classes, in case your model format is not supported by the wrapper classes that `eva` already provides. To do so, you need to subclass `eva.models.networks.wrappers.BaseModel` and implement the following abstract methods: 
+
+- `load_model`: Returns an instantiated model object & loads pre-trained model weights from a checkpoint if available. 
+- `model_forward`: Implements the forward pass of the model and returns the output as a `torch.Tensor` of shape `[embedding_dim]`
+
+You can take the implementations of `ModelFromFunction`, `HuggingFaceModel` and `ONNXModel` wrappers as a reference.

docs/user-guide/tutorials.md → docs/user-guide/tutorials/online_vs_offline.md

@@ -1,5 +1,8 @@
-# Tutorials
+# Online v.s. offline evaluations
 
+This Tutorial shows in detail how to run *online* & *offline* evaluations using eva.
+
+Remember that the *online* mode computes the foundation model embeddings on the fly while training the downstream decoder, while in *offline* mode we generate first generate and save the embeddings for all images, and then load them in a 2nd step from disk to train the downstream head network. *Offline* evaluations run faster, because the foundation forward pass to generate the embeddings is done only once per image. On the other hand, the *online* mode doesn't require storing the embeddings on disk and might be more handy for small datasets or experimental runs.
 
 If not done so already, download the configs for this tutorial from the [***eva*** GitHub repo](https://github.com/kaiko-ai/eva/tree/main):
 

mkdocs.yml

@@ -32,7 +32,9 @@ nav:
       - user-guide/index.md
       - Getting started: user-guide/getting_started.md
       - How to use eva: user-guide/how_to_use.md
-      - Tutorials: user-guide/tutorials.md
+      - Tutorials: 
+        - Online v.s. Offline Evaluations: user-guide/tutorials/online_vs_offline.md
+        - Model Wrappers: user-guide/tutorials/model_wrappers.md
       - Replicate evaluations: user-guide/replicate_evaluations.md
   - Reference API:
     - reference/index.md