Adam multitensor

[MoisesHer]

Description

FEATURE: multi-tensor Update - Adam Optimizer

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Modification in Adam optimizer for adding the possibility of updating several weight tensors within same operator. This expose more parallelism several. Up to 50 tensors can be updated within the operator. Observed ~7% throughput increase on V100. The multi-tensor version will be used when MXNET_OPTIMIZER_AGGREGATION_SIZE > 1
• Added multi-tensor Adam optimizer test

Comments

[codecov]

Codecov Report

Merging #1111 into master will decrease coverage by 0.04%.
The diff coverage is 81.57%.

@@            Coverage Diff             @@
##           master    #1111      +/-   ##
==========================================
- Coverage   88.25%   88.21%   -0.05%     
==========================================
  Files          67       67              
  Lines        6275     6302      +27     
==========================================
+ Hits         5538     5559      +21     
- Misses        737      743       +6

Impacted Files	Coverage Δ
src/gluonnlp/optimizer/bert_adam.py	87.32% <81.57%> (-5.86%)	⬇️

[mli]

Job PR-1111/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1111/1/index.html