[Model] DistilBERT #922

eric-haibin-lin · 2019-09-07T17:42:23Z

Description

The distilBERT model from https://arxiv.org/abs/1910.01108

Checklist

Essentials

PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage
Code is well-documented

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

codecov · 2019-09-07T17:42:24Z

Codecov Report

❗ No coverage uploaded for pull request head (convert@087dbcd). Click here to learn what that means.
The diff coverage is n/a.

codecov · 2019-09-07T17:42:24Z

Codecov Report

Merging #922 into master will decrease coverage by 0.52%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master    #922      +/-   ##
=========================================
- Coverage   88.32%   87.8%   -0.53%     
=========================================
  Files          67      71       +4     
  Lines        6330    6723     +393     
=========================================
+ Hits         5591    5903     +312     
- Misses        739     820      +81

Impacted Files	Coverage Δ
src/gluonnlp/data/question_answering.py	`100% <100%> (ø)`	⬆️
src/gluonnlp/model/seq2seq_encoder_decoder.py	`50% <0%> (-30%)`	⬇️
src/gluonnlp/model/utils.py	`70.76% <0%> (-6.93%)`	⬇️
src/gluonnlp/model/transformer.py	`86.85% <0%> (-4.81%)`	⬇️
src/gluonnlp/model/attention_cell.py	`87.7% <0%> (-4.47%)`	⬇️
src/gluonnlp/model/block.py	`51.06% <0%> (-2.13%)`	⬇️
src/gluonnlp/calibration/__init__.py	`100% <0%> (ø)`
src/gluonnlp/data/datasetloader.py	`85.34% <0%> (ø)`
src/gluonnlp/calibration/collector.py	`26.66% <0%> (ø)`
src/gluonnlp/data/dataloader.py	`87.93% <0%> (ø)`
... and 4 more

mli · 2019-09-07T18:16:15Z

Job PR-922/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/1/index.html

mli · 2020-01-25T02:20:34Z

Job PR-922/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/5/index.html

mli · 2020-01-25T03:16:55Z

Job PR-922/6 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/6/index.html

mli · 2020-01-26T05:06:25Z

Job PR-922/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/7/index.html

mli · 2020-01-26T05:30:08Z

Job PR-922/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/8/index.html

mli · 2020-01-26T05:34:22Z

Job PR-922/9 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/9/index.html

szha · 2020-01-27T05:30:15Z

scripts/bert/conversion_tools/convert_pytorch_transformers.py

@@ -0,0 +1,211 @@
+# coding: utf-8


How do you intend to maintain and verify this script?

They're currently not maintained, and not documented. Ideally we should test them in CI (adding fairseq, tf, pytorch transformer). But I don't have the bandwidth to do that now...

mli · 2020-01-31T09:01:19Z

Job PR-922/10 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/10/index.html

mli · 2020-01-31T22:29:10Z

Job PR-922/11 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/11/index.html

leezu · 2020-01-31T23:51:40Z

scripts/conversion_tools/convert_pytorch_transformers.py

+
+Usage: 
+
+pip3 install pytorch-transformers


That package seems not maintained anymore. Why not convert from transformers package? Can be addressed in a separate PR.

At the time of writing this script, there was only the pytorch-transformer package. I think it's still useful to some users, who started to use bert with pytorch-transformers.

mli · 2020-02-01T22:07:12Z

Job PR-922/13 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/13/index.html

mli · 2020-02-01T22:07:14Z

Job PR-922/12 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/12/index.html

mli · 2020-02-02T07:10:53Z

Job PR-922/14 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/14/index.html

scripts/index.rst

mli · 2020-02-03T06:19:23Z

Job PR-922/15 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-922/15/index.html

cnlewis3 · 2020-05-04T15:40:26Z

@mli Super happy DistilBERT was added as a model option!
Is exporting symbols file from the distilbert model an option? It looks like the only choices for that are the base and large bert models. Wasn't sure if I should open an issue to ask.

leezu · 2020-05-04T18:49:11Z

It's a HybridBlock, so you can call .export() to get the symbol

commit file

087dbcd

eric-haibin-lin requested a review from szha as a code owner September 7, 2019 17:42

szha requested a review from a team September 8, 2019 03:52

add DistilBERT class

f7502ce

eric-haibin-lin requested a review from a team as a code owner January 25, 2020 01:23

Ubuntu and others added 2 commits January 25, 2020 01:27

add test

65340bd

Merge branch 'master' into convert

57c4edc

eric-haibin-lin added the release focus Progress focus for release label Jan 25, 2020

eric-haibin-lin changed the title ~~[WIP] port distilBERT~~ [Model] DistilBERT Jan 25, 2020

Fix typos

92714e2

Update bert.py

75bc366

eric-haibin-lin and others added 3 commits January 25, 2020 20:33

Add use_pooler flag

bb92dbd

remove mask positions

cf6bd17

fix lint

0913242

szha reviewed Jan 27, 2020

View reviewed changes

Ubuntu added 2 commits January 31, 2020 06:59

resolve conflicts

a89bf7e

add conversion tool page

9375b53

update doc

72dd114

leezu reviewed Jan 31, 2020

View reviewed changes

eric-haibin-lin closed this Feb 1, 2020

eric-haibin-lin added 3 commits February 1, 2020 13:18

Update convert_fairseq_model.py

744d4b3

Update convert_paddle_to_gluon.py

7bbf95e

Update convert_pytorch_model.py

37af1f4

eric-haibin-lin reopened this Feb 1, 2020

Update convert_tf_model.py

b634849

Merge branch 'master' into convert

66e6f36

leezu approved these changes Feb 3, 2020

View reviewed changes

leezu reviewed Feb 3, 2020

View reviewed changes

scripts/index.rst Outdated Show resolved Hide resolved

Update index.rst

b90fee7

leezu merged commit acee226 into dmlc:master Feb 3, 2020


		Usage:

		pip3 install pytorch-transformers

[Model] DistilBERT #922

[Model] DistilBERT #922

Uh oh!

Conversation

eric-haibin-lin commented Sep 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Essentials

Changes

Comments

Uh oh!

codecov bot commented Sep 7, 2019

Codecov Report

Uh oh!

codecov bot commented Sep 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mli commented Sep 7, 2019

Uh oh!

mli commented Jan 25, 2020

Uh oh!

mli commented Jan 25, 2020

Uh oh!

mli commented Jan 26, 2020

Uh oh!

mli commented Jan 26, 2020

Uh oh!

mli commented Jan 26, 2020

Uh oh!

szha Jan 27, 2020

Choose a reason for hiding this comment

Uh oh!

eric-haibin-lin Jan 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mli commented Jan 31, 2020

Uh oh!

mli commented Jan 31, 2020

Uh oh!

leezu Jan 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-haibin-lin Feb 1, 2020

Choose a reason for hiding this comment

Uh oh!

mli commented Feb 1, 2020

Uh oh!

mli commented Feb 1, 2020

Uh oh!

mli commented Feb 2, 2020

Uh oh!

Uh oh!

mli commented Feb 3, 2020

Uh oh!

cnlewis3 commented May 4, 2020

Uh oh!

leezu commented May 4, 2020

Uh oh!

Uh oh!

eric-haibin-lin commented Sep 7, 2019 •

edited

Loading

codecov bot commented Sep 7, 2019 •

edited

Loading

eric-haibin-lin Jan 27, 2020 •

edited

Loading

leezu Jan 31, 2020 •

edited

Loading