Skip to content

Commit 814e9c4

Browse files
w5688414w5688414
authored
Add DeBERTa model (#8227)
* Add DeBERTa model * trigger unitest * Add hf mirror * fix unitest * Update deberta v2 model * Update debertav2 * Update deberta config * Update deberta docs * Update docs * support stabledropout * refine tokenizer and comments * fix tokenizer * try to fix connection error * Update deberta unitest * Add deberta doc * removev useless comments * Add deberta contents.rst
1 parent 9b89f45 commit 814e9c4

25 files changed

+5575
-0
lines changed

‎docs/model_zoo/index.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,8 @@ PaddleNLP的Transformer预训练模型包含从 `huggingface.co`_ 直接转换
8484
ChineseBert <transformers/ChineseBert/contents>
8585
ConvBert <transformers/ConvBert/contents>
8686
CTRL <transformers/CTRL/contents>
87+
Deberta <transformers/Deberta/contents>
88+
DebertaV2 <transformers/DebertaV2/contents>
8789
DistilBert <transformers/DistilBert/contents>
8890
ELECTRA <transformers/ELECTRA/contents>
8991
ERNIE <transformers/ERNIE/contents>
@@ -145,6 +147,10 @@ Transformer预训练模型适用任务汇总
145147
+--------------------+-------------------------+----------------------+--------------------+-----------------+-----------------+
146148
|CTRL_ | ✅ | ❌ | ❌ | ❌ | ❌ |
147149
+--------------------+-------------------------+----------------------+--------------------+-----------------+-----------------+
150+
|Deberta_ | ✅ | ✅ | ✅ | ❌ | ✅ |
151+
+--------------------+-------------------------+----------------------+--------------------+-----------------+-----------------+
152+
|DebertaV2_ | ✅ | ✅ | ✅ | ❌ | ✅ |
153+
+--------------------+-------------------------+----------------------+--------------------+-----------------+-----------------+
148154
|DistilBert_ | ✅ | ✅ | ✅ | ❌ | ❌ |
149155
+--------------------+-------------------------+----------------------+--------------------+-----------------+-----------------+
150156
|ELECTRA_ | ✅ | ✅ | ✅ | ❌ | ✅ |
@@ -220,6 +226,8 @@ Transformer预训练模型适用任务汇总
220226
.. _ChineseBert: https://arxiv.org/abs/2106.16038
221227
.. _ConvBert: https://arxiv.org/abs/2008.02496
222228
.. _CTRL: https://arxiv.org/abs/1909.05858
229+
.. _DeBERTa: https://arxiv.org/abs/2006.03654
230+
.. _DebertaV2: https://arxiv.org/abs/2111.09543
223231
.. _DistilBert: https://arxiv.org/abs/1910.01108
224232
.. _ELECTRA: https://arxiv.org/abs/2003.10555
225233
.. _ERNIE: https://arxiv.org/abs/1904.09223

‎docs/model_zoo/transformers/Deberta/contents.rst‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
2+
3+
------------------------------------
4+
Deberta模型汇总
5+
------------------------------------
6+
7+
8+
9+
下表汇总介绍了目前PaddleNLP支持的Deberta模型对应预训练权重。
10+
11+
+----------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------+
12+
| Pretrained Weight | Language | Details of the model |
13+
+==================================================================================+==============+==================================================================================+
14+
|``microsoft/deberta-base`` | English | 12-layer, 768-hidden, |
15+
| | | 12-heads, 100M parameters. |
16+
| | | It outperforms BERT and RoBERTa on majority of NLU tasks with 80GB training data.|
17+
+----------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------+

‎docs/model_zoo/transformers/DebertaV2/contents.rst‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
2+
3+
------------------------------------
4+
DebertaV2模型汇总
5+
------------------------------------
6+
7+
8+
9+
下表汇总介绍了目前PaddleNLP支持的DebertaV2模型对应预训练权重。
10+
11+
+----------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------+
12+
| Pretrained Weight | Language | Details of the model |
13+
+==================================================================================+==============+==================================================================================+
14+
|``deepset/deberta-v3-large-squad2`` | English | 24-layer, 1024-hidden, |
15+
| | | 16-heads, 304M parameters. |
16+
| | | The deberta-v3-large model fine-tuned using the SQuAD2.0 dataset. |
17+
+----------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------+
18+
|``microsoft/deberta-v2-xlarge`` | English | 24-layer, 1536-hidden, |
19+
| | | 24-heads, 900M parameters. |
20+
| | | The deberta-v2 model. |
21+
+----------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------+
22+
|``microsoft/deberta-v3-base`` | English | 12-layer, 768-hidden, |
23+
| | | 12-heads, 86M parameters. |
24+
| | | The deberta-v2 model. |
25+
+----------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------+
26+
|``microsoft/deberta-v3-large`` | English | 24-layer, 1024-hidden, |
27+
| | | 16-heads, 304M parameters. |
28+
| | | The deberta-v2 model. |
29+
+----------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------+
30+

‎docs/source/paddlenlp.transformers.deberta.modeling.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
modeling
2+
===========================================
3+
4+
.. automodule:: paddlenlp.transformers.deberta.modeling
5+
:members:
6+
:no-undoc-members:
7+
:show-inheritance:

‎docs/source/paddlenlp.transformers.deberta.rst‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
deberta
2+
========================================
3+
4+
.. automodule:: paddlenlp.transformers.deberta
5+
:members:
6+
:no-undoc-members:
7+
:show-inheritance:
8+
9+
10+
.. toctree::
11+
:maxdepth: 4
12+
13+
paddlenlp.transformers.deberta.modeling
14+
paddlenlp.transformers.deberta.tokenizer

‎docs/source/paddlenlp.transformers.deberta.tokenizer.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
tokenizer
2+
=================================================
3+
4+
.. automodule:: paddlenlp.transformers.deberta.tokenizer
5+
:members:
6+
:no-undoc-members:
7+
:show-inheritance:

‎docs/source/paddlenlp.transformers.deberta_v2.modeling.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
modeling
2+
===========================================
3+
4+
.. automodule:: paddlenlp.transformers.deberta_v2.modeling
5+
:members:
6+
:no-undoc-members:
7+
:show-inheritance:

‎docs/source/paddlenlp.transformers.deberta_v2.rst‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
deberta_v2
2+
========================================
3+
4+
.. automodule:: paddlenlp.transformers.deberta_v2
5+
:members:
6+
:no-undoc-members:
7+
:show-inheritance:
8+
9+
10+
.. toctree::
11+
:maxdepth: 4
12+
13+
paddlenlp.transformers.deberta_v2.modeling
14+
paddlenlp.transformers.deberta_v2.tokenizer

‎docs/source/paddlenlp.transformers.deberta_v2.tokenizer.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
tokenizer
2+
=================================================
3+
4+
.. automodule:: paddlenlp.transformers.deberta_v2.tokenizer
5+
:members:
6+
:no-undoc-members:
7+
:show-inheritance:

‎docs/source/paddlenlp.transformers.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ paddlenlp.transformers
2424
paddlenlp.transformers.convbert
2525
paddlenlp.transformers.ctrl
2626
paddlenlp.transformers.dallebart
27+
paddlenlp.transformers.deberta
28+
paddlenlp.transformers.deberta_v2
2729
paddlenlp.transformers.distilbert
2830
paddlenlp.transformers.electra
2931
paddlenlp.transformers.ernie

0 commit comments

Comments
 (0)