Possible RNN design for comment.

[aterzis-google]

Alternative RNN design due to david-berthelot@

[david-berthelot]

Following on the feedback on the original RNN pull request #97 Andreas and I did a pair-programming session to prototype a better design with the following goals:

• flexible (accommodate any kind of internal primitive)
• does not explicitly refers to a batch size
• does not make decisions for inference (e.g. there are too many cases to provide a solution: like feeding previous prediction, beam search, ...)

Not being an expert myself in recurrent network, I would really love feedback @ebrevdo @AlexeyKurakin - feel free to loop in more people whose opinions could help.

[ebrevdo]

Some rough thoughts: 1. There will always be some RNNs that will want to do something across batch and time. These will be some combination of scan + input/output layers that can handle the time dimension (e.g., Dense layer). So expect that users will still subclass/override RNN. LSTM is a good example. 2. get_initial_state is still super useful. why? there are RNNs whose state is a nested combination of tensors including tensors of different *types*. for example, some special RNNs have as the state an integer scalar corresponding to the current time step. so each iteration of the RNN increments that state by one. but the other states are typical floating point vectors.

…

On Thu, Dec 17, 2020 at 1:19 PM David Berthelot ***@***.***> wrote: Following on the feedback on the original RNN pull request #97 <#97> Andreas and I did a pair-programming session to prototype a better design with the following goals: - flexible (accommodate any kind of internal primitive) - does not explicitly refers to a batch size - does not make decisions for inference (e.g. there are too many cases to provide a solution: like feeding previous prediction, beam search, ...) Not being an expert myself in recurrent network, I would really love feedback @ebrevdo <https://github.com/ebrevdo> @AlexeyKurakin <https://github.com/AlexeyKurakin> - feel free to loop in more people whose opinions could help. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANWFG4XEXERYH3TDFGX7FLSVJYUVANCNFSM4VACK3FQ> .

[aterzis-google]

See #211 instead

[aterzis-google]

Instead, see: #211