[WIP] Add basic transforms for multimodal #1141

RdoubleA · 2024-07-03T16:23:38Z

No description provided.

pytorch-bot · 2024-07-03T16:23:41Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1141

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 1 Cancelled Job, 4 Unrelated Failures

As of commit 5f7e8aa with merge base f158577 ():

NEW FAILURES - The following jobs have failed:

Build Linux Wheels / __default / upload / manywheel-py3_8-cpu (gh)
Unable to find any artifacts for the associated workflow
Build Linux Wheels / __default / upload / manywheel-py3_8-cuda11_8 (gh)
Unable to find any artifacts for the associated workflow
Build Linux Wheels / __default / upload / manywheel-py3_8-cuda12_1 (gh)
Unable to find any artifacts for the associated workflow
Build Linux Wheels / __default / upload / manywheel-py3_8-cuda12_4 (gh)
Unable to find any artifacts for the associated workflow

CANCELLED JOB - The following job was cancelled. Please retry:

Multi-GPU Recipe Tests / recipe_test_multi_gpu (3.9, stable) (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

Build Docs / doc-preview (gh) (detected as infra flaky with no log or failing log classifier)
Multi-GPU Recipe Tests / recipe_test_multi_gpu (3.10, stable) (gh) (detected as infra flaky with no log or failing log classifier)
Multi-GPU Recipe Tests / recipe_test_multi_gpu (3.11, stable) (gh) (detected as infra flaky with no log or failing log classifier)
Multi-GPU Recipe Tests / recipe_test_multi_gpu (3.8, stable) (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

felipemello1

it looks great! I left a few comments more around naming, since tiles/patches/image is so confusing in general. I think that this class would benefit from a visual small example too.

regarding the interval parts, IMO we should try to make the operations as explicit as possible. Instead of tok1, tok2; i; vision_mask[0], we should try to give them names, like idx_start, idx_end, etc.

felipemello1 · 2024-07-03T16:32:29Z

torchtune/modules/transforms/_transforms.py

+    participate in cross-attention with an image token will show True in the mask
+    and follow these rules:
+    1) Text tokens immediately following the image token up until the next image token
+    2) Consecutive image tokens attend to all subsequent text tokens


can we make it more visual? Something like this: https://github.com/huggingface/transformers/blob/60bb571e993b7d73257fb64044726b569fef9403/src/transformers/models/llava_next/modeling_llava_next.py#L446

Or a link to the paper + page where they have an image for it

felipemello1 · 2024-07-03T16:33:06Z

torchtune/modules/transforms/_transforms.py

+
+class CrossAttentionMask(Transform):
+    """
+    Computes the cross-attention mask for text + image inputs. Text tokens that


if this CrossAttention is specific for text + images, should we indicate it in the name? Something like MultimodalCrossAttentionMask or VisionTextCrossAttentionMask?

Yeah that's a good point. Will rename

felipemello1 · 2024-07-03T17:08:49Z

torchtune/modules/transforms/_transforms.py

+    text sequence.
+
+    Args:
+        num_patches (int): Number of patches per image, excluding class token.


I wonder if we should add a link to modules/VisionTransformer for a in-depth explanation of what num_patches mean. For better clarity, would it make sense to rename it num_patches_per_tile, since later we multiply it by n_tiles?

If we say "number of patches per image", it may be confusing, because an image can have a variable number of patches.

later on you say:

single_image_seq_len = n_tiles * self.num_patches + 1 image_seq_len = single_image_seq_len * n_img

So tile != image. Image is a set of tiles.

ah sorry, probably my shallow understanding of patches/images/tiles. What I intended was num_patches per tile. If it makes sense I'd like to keep the name consistent with your vision transformer (either patch_grid_size or patch_size maybe?), whichever parameter you use to compute num patches.

I also assumed at this point tiles is padded to the max in all the images. Is this incorrect? Where does the padding happen?

patches_per_tile is a fixed size, and its calculated as (tile_size // patch_size)**2.

What I did in VisionTransformer was to ask the user to pass tile_size and patch_size, and I calculated it for them. The VisionTransformer has a helper function that saves this value: https://github.com/felipemello1/torchtune/blob/f683812626ad4559464840112ddce516487bea5c/torchtune/modules/vision_transformer.py#L249

Maybe get it from the model, or ask for tile_size and patch_size, to avoid user confusion?

felipemello1 · 2024-07-03T17:13:00Z

torchtune/modules/transforms/_transforms.py

+        pass
+
+
+class Compose(Transform):


Since the torchvision compose has a different behavior, I wonder if it makes sense to change Compose to something else, so users dont get confused with tv.Compose. Maybe "ComposeTransforms"?

how about Pipeline?

Just for my own understanding, the main difference with torchvision compose is that we support multiple inputs and multiple outputs here? Can we not just use torchvision compose with a single dict?

I tried naming something as Pipeline, as Kartikay said it would confuse people, because it is also used by other libraries :P. I guess sklearn?

felipemello1 · 2024-07-03T17:14:02Z

torchtune/modules/transforms/_transforms.py

+        """
+        Returns a list of tuples of the form (start, end) where start is the index
+        of the current image token and end is the index of the next image token, exclusive.
+        If the image token attends until the end of the sequence, end will be -1.


nit: should we add Args:,Returns, Examples?

felipemello1 · 2024-07-03T17:17:55Z

torchtune/modules/transforms/_transforms.py

+        vision_masks = [
+            [tok1, tok2]
+            for tok1, tok2 in zip(
+                vision_token_locations[:-1], vision_token_locations[1:]


nit:

Maybe add a comment:
"offset by one and get consecutive indices"

I guess renaming tok1, tok2 to something like tok_idx_prev, tok_idx_next may be more intuitive

felipemello1 · 2024-07-03T17:30:05Z

torchtune/modules/transforms/_transforms.py

+
+    def __call__(self, *, tokens, images, **kwargs):
+        # We are still at sample level pre-collating
+        n_img, n_tiles, _, _, _ = images.shape


nit: Maybe add a comment with the other dimensions, so we know what they are, but keep the "_", so we know they are not used?

You said "# We are still at sample level pre-collating"
So is n_img == bsz? If so, for consistency with VisionTransformer, should we rename it?

yeah please add type and shape info for arguments to __call__

felipemello1 · 2024-07-03T17:32:21Z

torchtune/modules/transforms/_transforms.py

+        # We are still at sample level pre-collating
+        n_img, n_tiles, _, _, _ = images.shape
+        text_seq_len = len(tokens)
+        single_image_seq_len = n_tiles * self.num_patches + 1


nit: maybe add comment explaining that +1 is for CLS, if thats the case

ebsmothers · 2024-07-03T20:45:35Z

torchtune/modules/transforms/_transforms.py

+                image_num
+                * single_image_seq_len : (image_num + 1)
+                * single_image_seq_len,


nit: split line differently if linter allows it, as written this is confusing

ebsmothers · 2024-07-03T21:08:25Z

torchtune/modules/transforms/_transforms.py

+                * single_image_seq_len,
+            ] = True
+
+        kwargs.update({"encoder_mask": mask, "tokens": tokens, "images": images})


Why do we need to also update with tokens and images? Isn't this a no-op for those args?

Since they are explicit keyword args, they get unfolded from kwargs and you have to add them back in

RdoubleA added 2 commits July 2, 2024 12:19

add basic transforms

d90e5a9

add xattn mask transform

5f7e8aa

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 3, 2024

felipemello1 reviewed Jul 3, 2024

View reviewed changes

ebsmothers reviewed Jul 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add basic transforms for multimodal #1141

[WIP] Add basic transforms for multimodal #1141

RdoubleA commented Jul 3, 2024

pytorch-bot bot commented Jul 3, 2024 •

edited

Loading

felipemello1 left a comment •

edited

Loading

felipemello1 Jul 3, 2024

felipemello1 Jul 3, 2024 •

edited

Loading

RdoubleA Jul 3, 2024

felipemello1 Jul 3, 2024

RdoubleA Jul 3, 2024

felipemello1 Jul 3, 2024 •

edited

Loading

felipemello1 Jul 3, 2024

RdoubleA Jul 3, 2024

ebsmothers Jul 3, 2024

felipemello1 Jul 3, 2024

felipemello1 Jul 3, 2024

felipemello1 Jul 3, 2024

felipemello1 Jul 3, 2024

ebsmothers Jul 3, 2024

felipemello1 Jul 3, 2024

ebsmothers Jul 3, 2024

ebsmothers Jul 3, 2024

RdoubleA Jul 5, 2024

[WIP] Add basic transforms for multimodal #1141

Are you sure you want to change the base?

[WIP] Add basic transforms for multimodal #1141

Conversation

RdoubleA commented Jul 3, 2024

pytorch-bot bot commented Jul 3, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1141

❌ 4 New Failures, 1 Cancelled Job, 4 Unrelated Failures

felipemello1 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felipemello1 Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felipemello1 Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pytorch-bot bot commented Jul 3, 2024 •

edited

Loading

felipemello1 left a comment •

edited

Loading

felipemello1 Jul 3, 2024 •

edited

Loading

felipemello1 Jul 3, 2024 •

edited

Loading