error in self attention visualisation

[twangnh]

I beleive there is a error for idexing the self attetion:

for idx_o, ax in zip(idxs, axs):
    idx = (idx_o[0] // fact, idx_o[1] // fact)
    ax.imshow(sattn[..., idx[0], idx[1]], cmap='cividis', interpolation='nearest')
    ax.axis('off')
    ax.set_title(f'self-attention{idx_o}')

should be
sattn[idx[0], idx[1], ...]as the last two dim sums to 1, so the first two are locations and are the dim to indexing, with
this the attention map is

[eagleanurag]

Yes ! we have some other little issues too. Please Pull request regarding your concerns and suggestions

[twangnh]

Could you pls share about what other little issues are? thank you. by the way, are you a contributor of the repo?

[fmassa]

Hi,

This is a fair point and is discussed in the bottom of the notebook as well

In the end, it's a matter of deciding if you want the attention of all pixels at location x, or the contribution of location x to all the attention maps.

I would say both information are useful to be considered, and as such I added both in the widget visualization.

[songhwanjun]

I also found an error in computing the scale factor. When computing the factor, the code currently divides the original image "height" with the input image "width".

scale = im.height / img.shape[-1]

should be changed into

scale = im.height / img.shape[-2]

Hence, the correct visualization is

[fmassa]

@songhwanjun good catch. I had fixed this for the class AttentionVisualizer, but looks like I missed it in the example cell just above. I'll send a PR fixing it

[fmassa]

@songhwanjun fixed in #210

[TimGor1997]

I beleive there is a error for idexing the self attetion:

for idx_o, ax in zip(idxs, axs):
    idx = (idx_o[0] // fact, idx_o[1] // fact)
    ax.imshow(sattn[..., idx[0], idx[1]], cmap='cividis', interpolation='nearest')
    ax.axis('off')
    ax.set_title(f'self-attention{idx_o}')

should be
sattn[idx[0], idx[1], ...]as the last two dim sums to 1, so the first two are locations and are the dim to indexing, with
this the attention map is

could you tell me how to "visualize encoder-decoder multi-head attention weights"?
I've copy the code in the pycharm, but nothing show at all……

[TimGor1997]

Yes ! we have some other little issues too. Please Pull request regarding your concerns and suggestions

could you tell me how to "visualize encoder-decoder multi-head attention weights"?
I've copy the code in the pycharm, but nothing show at all……

[TimGor1997]

I also found an error in computing the scale factor. When computing the factor, the code currently divides the original image "height" with the input image "width".

scale = im.height / img.shape[-1]

should be changed into

scale = im.height / img.shape[-2]

Hence, the correct visualization is

could you tell me how to "visualize encoder-decoder multi-head attention weights"?
I've copy the code in the pycharm, but nothing show at all……