[Feature] allow custom dropout, number of layers/units for BERT

[eric-haibin-lin]

Description

reopen #851

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

cc @dmlc/gluon-nlp-team

[codecov]

Codecov Report

Merging #950 into master will increase coverage by 9.86%.
The diff coverage is 86.34%.

@@            Coverage Diff             @@
##           master     #950      +/-   ##
==========================================
+ Coverage   78.41%   88.27%   +9.86%     
==========================================
  Files          67       67              
  Lines        6277     6287      +10     
==========================================
+ Hits         4922     5550     +628     
+ Misses       1355      737     -618

Impacted Files	Coverage Δ
src/gluonnlp/utils/parameter.py	87.09% <100%> (+3.16%)	⬆️
src/gluonnlp/optimizer/__init__.py	100% <100%> (ø)	⬆️
src/gluonnlp/metric/__init__.py	100% <100%> (ø)	⬆️
src/gluonnlp/metric/masked_accuracy.py	100% <100%> (+39.53%)	⬆️
src/gluonnlp/utils/version.py	100% <100%> (ø)	⬆️
src/gluonnlp/utils/files.py	42.62% <18.18%> (-6.4%)	⬇️
src/gluonnlp/model/train/language_model.py	88.51% <25%> (+48.62%)	⬆️
src/gluonnlp/optimizer/bert_adam.py	87.32% <81.57%> (-5.86%)	⬇️
src/gluonnlp/metric/length_normalized_loss.py	89.28% <89.28%> (ø)
src/gluonnlp/data/utils.py	86.39% <95.34%> (+12.34%)	⬆️
... and 22 more

[mli]

Job PR-950/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-950/1/index.html

[mli]

Job PR-950/2 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-950/2/index.html

[mli]

Job PR-950/3 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-950/3/index.html

[mli]

Job PR-950/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-950/4/index.html

[leezu]

@eric-haibin-lin

[2020-01-15T17:07:50.259Z] tests/unittest/test_models.py::test_bert_models[False] FAILED            [ 60%]

[2020-01-15T17:08:37.013Z] tests/unittest/test_models.py::test_bert_models[True] FAILED             [ 61%]

[eric-haibin-lin]

I saw that.. And i am not able to reproduce it locally.

[eric-haibin-lin]

@leezu were you able to reproduce the err?

[mli]

Job PR-950/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-950/5/index.html

[mli]

Job PR-950/6 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-950/6/index.html

[mli]

Job PR-950/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-950/7/index.html

[mli]

Job PR-950/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-950/8/index.html

[leezu]

Not changed in this PR, but why is embed_size not part of this?

[eric-haibin-lin]

I copied some from the transformer get model function. Actually, i'm not sure whether we want to have a whitelist of mutable args. It make the fucntion harder to maintain, since we have hparam_allow_override flag.

[leezu]

Do you want to remove the whitelist then? In this PR or a separate one?

[eric-haibin-lin]

I'd prefer a separate one