How does Masking work?

[poyuwu]

I'm wondering how Masking Layer works.
I try to write simple model to test Masking on Activation Layer

from keras.models import Model
import numpy as np
from keras.layers import Masking, Activation, Input
a = np.array([[3.,1.,2.,2.,0.,0.]])

inputs = Input(shape=(6,))
mask = Masking(mask_value=0.0)(inputs)
softmax = Activation('softmax')(mask)
model = Model(input=inputs,output=softmax)
model.predict(a)

and the result of prediction is

array([[ 0.50744212,  0.06867483,  0.18667753,  0.18667753,  0.02526405,
         0.02526405]])

Is this the correct behavior?
My keras version is 1.0.5

[ipoletaev]

I'm also interested with this question. It seems to me you expected to get something like the following:
[[ 0.53444666, 0.07232948, 0.19661194, 0.19661194]] ?
But from the other side, - according to the explanations in core.py : class Masking(Layer) - masking doesn't work with 1D input data. So, if you try this, for example:

from keras.models import Model
import numpy as np
from keras.layers import Masking,Input,TimeDistributed,Dense
a = np.array([[[3,1,2,2,0,0],[0,0,0,0,0,0],[2,1,1,2,0,0]]])

input = Input(shape=(3,6))
mask = Masking(mask_value=0)(input)
out = TimeDistributed(Dense(1,activation='linear'))(mask)
model = Model(input=input,output=out)
q = model.predict(a)
print (q[0])

...you will get [[-0.20101213],[ 0. ],[-0.51546627]] as expected.
But I think that, most likely, there's something wrong in my understanding.

[poyuwu]

@ipoletaev
yes, sure. However keras can also return [[ 0.53444666, 0.07232948, 0.19661194, 0.19661194, 0.0, 0.0 ]] by padding it to keep shape.
Here is another example about Masknig on bi-LSTM layer but sum two layer

from keras.models import Model
import numpy as np
from keras.layers import Masking, Activation, Input, LSTM, merge
a = np.array([[[.3,.1,.2,.2,.1,.1],[.2,.3,.3,.3,.3,.1],[0,0,0,0,0,0]]])

inputs = Input(shape=(3,6))
mask = Masking(mask_value=0.0)(inputs)
fw = LSTM(1,return_sequences=True)(mask)
bw = LSTM(1,return_sequences=True,go_backwards=True)(mask)
merged = merge([fw,bw],mode='sum')
model = Model(input=inputs,output=fw)
model2 = Model(input=inputs,output=bw)
model3 = Model(input=inputs,output=merged)

the fw's output is
array([[[-0.07041532], [-0.12203699], [-0.12203699]]])
the bw's output is
array([[[ 0. ], [-0.03112165], [ 0.02271803]]])
the merge's output is
array([[[-0.07041532], [-0.15315863], [-0.09931896]]])
but I think it should be (Here it also can padding 0 to keep shape.)
array([[[-0.10153697], [-0.09931896]]])
which -0.10153697 = (-0.07041532) + (-0.03112165) and -0.09931896 = -0.12203699 + 0.02271803
Is is anything wrong on Keras?

[ipoletaev]

@poyuwu

However keras can also return [[ 0.53444666, 0.07232948, 0.19661194, 0.19661194, 0.0, 0.0 ]] by padding it to keep shape.

Hmm... I don't know how to make such out only through the Keras.

About your example: I think it's similar to the aforementioned example, so you should get array([[[-0.07041532], [-0.15315863], [0.02271803]]]). And it's really strange that bw works right but fw doesn't, because of its third output is not equal to zero, but it must...

[lomizandtyd]

Hi guys, I got this question too.. Especially for LSTM (BRNN).

Masking Layer gives a masked vector, only work for the inputs, not for inner states.
So in @poyuwu 's example, the fw's output still has value in step 3.

This might be correct because inputs are surely masked.
While, I want to find a way to skip the calculation step when coming masked value, like some special tags.

However, I think using Masking layer in bidirectional RNN for sequences with different lengths may be totally wrong.

[ipoletaev]

@lomizandtyd

So in @poyuwu 's example, the fw's output still has value in step 3.

Yes, it's logically, but in any case we want to get zero at the third place, isn't it?

While, I want to find a way to skip the calculation step when coming masked value.

I think it doesn't matter because of, as I understood, you should specify output sample_weights in fit() in order to skip necessary timesteps with all zeros in feature vector (I have already asked about this #3023 ). But if this is so, then it is not clear why do we need masking if we can specify it in fit(): what are the steps in the examples is using for training, and what - no. I mean it is not important to process these "empty" vectors by network, it is important to train network without backpropagation with errors calculated on such vectors.

Maybe there is some way to use a batch_size=1 and do not bother with padding?

[lomizandtyd]

@ipoletaev Wow, thanks a lot for this!

Yes, we want to get zero at the masked position.
The problem is we also want to keep the inner states across the masked step.

Maybe we can deliver another sample_weights in the predict() function?.
If do so, BRNN is still wrong...

[ipoletaev]

@lomizandtyd

...keep the inner states across the masked step.

I think it's not necessary, because the network shouldn't remember what responses it need to get at empty vectors...

Maybe we can deliver another sample_weights in the predict() function?.

I don't understand for what task you want to use it? After all you always know in advance what data you process, and you respectively know - which output of the network corresponds to the empty vectors, so you can just skip such positions in output, I guess.

If do so, BRNN is still wrong...

As far as I understood Keras has been "fighting" with RNN masking task about year :)

[poyuwu]

from keras.models import Model
import numpy as np
from keras.layers import Masking,Input,TimeDistributed,Dense
a = np.array([[[3,1,2,2,0,0],[0,0,0,0,0,0],[2,1,1,2,0,0]]])
input = Input(shape=(3,6))
mask = Masking(mask_value=0)(input)
out = TimeDistributed(Dense(1,activation='linear'))(mask)
model = Model(input=input,output=out)
q = model.predict(a)
print (q[0])

@ipoletaev I think it's just Dense Layer that have zero inputs, so that its output is 0. If you change activation function to softmax, then you will get wrong answer.
Besides, batch_size set None on time steps will raise other error in some case (especially on merge layer).

In lasagne, it seems to use Masking matrix to deal with padding. (I do not test its accuracy)

[ipoletaev]

@poyuwu : yes, I had checked it - and you are right. It means,as I understood, that and simple Dense doesn't keep masked values in the way we want...

I write again what does not converge with the expectations:

• Forward RNN doesn't keep mask values, backward does it. It's strange.
• Is this task solving with batch_size = 1?
• How to specify correctly what timesteps the network should to skip.
• And it's not clear in which moment BiLSTM does reset_state - only in the end of timesteps in current sample, or when the network meets with empty vector?

[poyuwu]

@ipoletaev I don't think

• Forward RNN doesn't keep mask values, backward does it. It's strange.

this statement is true.
That's because padding argument is 'post', not 'pre'. Hence, the reason is the same as Dense layer I said.

• How to specify correctly what timesteps the network should to skip.

As I said, in lasagne, we provide a mask numpy.array (the same shape as input) to deal with it. If go_backwards=True, it needs to keep padding argument the same.

Besides, Embedding layer mask_zeros seems to be the same.

[ipoletaev]

@poyuwu so you want to say that now, there's no way to solve this issue with Keras?
I mean is it necessary to use masking if we use sample weights?

[xuewei4d]

Same here. It seems masking mechanism in Keras is not fully supported.

[fferroni]

I don't think Masking masks input values (neither during forward or back-propagation). It just skips a time-step where all features are equal to the mask value (i.e. when you pad a sequence). You can confirm this by:

from keras.models import Model
import numpy as np
from keras.layers import Masking, Activation, Input, TimeDistributed, Dense

if __name__ == "__main__":
	a = np.array([[[3,1,2,2,0.1,0.1],[0,0,0,0,0,0],[2,1,1,2,0.1,0.1]]])
	print 'Input array:'
	print a
	print ''
	input = Input(shape=(3,6))
	mask = Masking(mask_value=0.1)(input)
	out = TimeDistributed(Dense(1, activation='linear'))(mask)
	model = Model(input=input, output=out)

	model.set_weights([np.array([[ 1. ],
							     [ 1. ],
							     [ 1. ],
							     [ 1. ],
							     [ 1. ],
							     [ 1.]], dtype=np.float32), 
	                   np.array([ 0.], dtype=np.float32)])

	print 'Weights'
	print model.get_weights()
	q = model.predict(a)
	print q

The answer is:

Input array:
[[[ 3.   1.   2.   2.   0.1  0.1]
  [ 0.   0.   0.   0.   0.   0. ]
  [ 2.   1.   1.   2.   0.1  0.1]]]

Weights
[array([[ 1.],
       [ 1.],
       [ 1.],
       [ 1.],
       [ 1.],
       [ 1.]], dtype=float32), array([ 0.], dtype=float32)]
[[[ 8.20000076]
  [ 0.        ]
  [ 6.19999981]]]

If it masked the inputs of value 0.1, you would expect result to be

[[[ 8.       ]
  [ 0.        ]
  [ 6.        ]]]