How to visualize the attention?

[Johnzu-2019]

Do you have a way to visualize the attention mechanism?I want to know what exactly does the attention mechanism focus on?

[v-iashin]

Just propagate attention

BMT/model/multihead_attention.py

Line 19 in d45ad8f

softmax = F.softmax(sm_input, dim=-1)

to the output of the modules and apply an aggregation function of your liking.

[Johnzu-2019]

Just propagate attention

BMT/model/multihead_attention.py

Line 19 in d45ad8f

softmax = F.softmax(sm_input, dim=-1)

to the output of the modules and apply an aggregation function of your liking.

Thank you. I'll try that.

[Johnzu-2019]

softmax

Sorry, I didn't quite catch your meaning. Could you explain it in more detail?In which file should this line of code be added?

[v-iashin]

I meant that you will need to add this variable to the output of each module.

[Johnzu-2019]

I meant that you will need to add this variable to the output of each module.

Thank you. I will continue to study this issue