Repurpose the affinity matrix for direct frame-vs-frame similarity

[max810]

Hi!

I was wondering if it's possible to directly measure similarity between any 2 frames (as a real number).
The idea is simple - since in the process of memory matching XMem takes top-k most similar frames to the query frame, it should be pretty straightforward to measure this "similarity" explicitly.

However, I'm stuck on implementation. The affinity matrix of size [N*H*W x H*W] implies that every single pixel in the output image/feature map is a linear combination of every single pixel in all the N memory frames.

So I have 2 questions:

1. How do I interpret the affinity matrix? E.g. I have only 2 images with a single channel in each, the affinity matrix A would be of size HW x HW, so what do the individual entries mean? If A[i, j] is a large positive number, does this mean that the pixel i in image 1 is very similar to the pixel j in image 2?
2. How do I reduce the affinity matrix between a pair of images/feature maps to a single number that would describe the similarity between them (the similarity in the same sense that is used for memory matching operations in XMem)

[hkchengrex]

1. That depends on how the affinity matrix is constructed. See https://github.com/hkchengrex/XMem/blob/main/model/memory_util.py. If frame 1 is the memory frame, and frame 2 is the query frame, that would be correct.
2. Our network is not trained this way, but I think you can sum over all the pixels in the H*W dimension (e.g., THW*HW -> T*1).

You might also want to try using the matrix before/after softmax.

[max810]

But there is no fundamental difference between memory frame and query frame, right? Except that in the memory matching process we use shrinkage of memory and selection from the query frame.

Both are just key features extracted from the frames. So if I just switch frame 1 and frame 2 (and use appropriate shrinkage and selection matrices), is that output similarity supposed to be symmetrical?

[hkchengrex]

Yes.