Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP into zoo_ernie_m

Author: Yam0214

CODE_OF_CONDUCT.md renamed to .github/CODE_OF_CONDUCT.md

@@ -1,3 +1,4 @@
+**简体中文**🀄 | [English🌎](./CODE_OF_CONDUCT_en.md)
 
 # 贡献者公约
 

CODE_OF_CONDUCT_en.md renamed to .github/CODE_OF_CONDUCT_en.md

@@ -1,3 +1,4 @@
+[简体中文🀄](./CODE_OF_CONDUCT.md) |  **English**🌎
 
 # Contributor Covenant Code of Conduct
 

.github/CONTRIBUTING_en.md

@@ -0,0 +1,154 @@
+[简体中文🀄](../CONTRIBUTING.md) |  **English**🌎
+
+# Contributing to PaddleNLP
+
+We highly welcome and value your contributions to `PaddleNLP`. The first step to start your contribution is to sign the [PaddlePaddle Contributor License Agreement](https://cla-assistant.io/PaddlePaddle/PaddleNLP).
+
+This document explains our workflow and work style:
+
+## Finding out what to work on Workflow
+
+## Development Workflow
+
+PaddleNLP uses the [Git branching model](http://nvie.com/posts/a-successful-git-branching-model/).  The following steps guide usual contributions.
+
+#### 1. Fork
+
+   Our development community has been growing fastly; it doesn't make sense for everyone to write into the official repo.  So, please file Pull Requests from your fork.  To make a fork,  just head over to the GitHub page and click the ["Fork" button](https://help.github.com/articles/fork-a-repo/).
+
+#### 2. Clone
+
+   To make a copy of your fork to your local computers, please run
+
+   ```bash
+   git clone https://github.com/<your-github-account>/PaddleNLP
+   cd PaddleNLP
+   ```
+
+#### 3. Create the local feature branch
+
+   For daily works like adding a new feature or fixing a bug, please open your feature branch before coding:
+
+   ```bash
+   git checkout -b my-cool-feature
+   ```
+
+#### 4. Set up the development environment
+
+   Before you start coding, you need to setup the development environment. We highly recommend doing all your development in a virtual environment such as
+   [venv](https://docs.python.org/3/library/venv.html) or [conda](https://docs.conda.io/en/latest/). After you setup and activated your virtual environment,
+   run the following command:
+
+   ```bash
+   make install
+   ```
+
+   This will setup all the dependencies of `PaddleNLP` as well as the [`pre-commit`](http://pre-commit.com/) tool.
+
+   If you are working on the `examples` or `applications` module and require importing from `PaddleNLP`, make sure you install `PaddleNLP` in editable mode.
+   If `PaddleNLP` is already installed in the virtual environment, remove it with `pip uninstall paddlenlp` before reinstalling it in editable mode with
+   `pip install -e .`
+
+#### 5. Develop
+
+   As you develop your new exciting feature, keep in mind that it should be covered by unit tests. All of our unit tests can be found under the `tests` directory.
+   You can either modify existing unit test to cover the new feature, or create a new test from scratch.
+   As you finish up the your code, you should make sure the test suite passes. You can run the tests impacted by your changes like this:
+
+   ```bash
+   pytest tests/<test_to_run>.py
+   ```
+
+#### 6. Commit
+
+   We utilizes [`pre-commit`](http://pre-commit.com/) (with [black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/) and
+   [flake8](https://flake8.pycqa.org/en/latest/) under the hood) to check the style of code and documentation in every commit. When you run run `git commit`, you will see
+   something like the following:
+
+   ```
+    ➜  (my-virtual-env) git commit -m "commiting my cool feature"
+    black....................................................................Passed
+    isort....................................................................Passed
+    flake8...................................................................Passed
+    check for merge conflicts................................................Passed
+    check for broken symlinks............................(no files to check)Skipped
+    detect private key.......................................................Passed
+    fix end of files.....................................(no files to check)Skipped
+    trim trailing whitespace.............................(no files to check)Skipped
+    CRLF end-lines checker...............................(no files to check)Skipped
+    CRLF end-lines remover...............................(no files to check)Skipped
+    No-tabs checker......................................(no files to check)Skipped
+    Tabs remover.........................................(no files to check)Skipped
+    copyright_checker........................................................Passed
+   ```
+
+   But most of the time things don't go so smoothly. When your code or documentation doesn't meet the standard, the `pre-commit` check will fail.
+   ```
+    ➜  (my-virtual-env) git commit -m "commiting my cool feature"
+    black....................................................................Passed
+    isort....................................................................Failed
+    - hook id: isort
+    - files were modified by this hook
+
+    Fixing examples/information_extraction/waybill_ie/run_ernie_crf.py
+
+    flake8...................................................................Passed
+    check for merge conflicts................................................Passed
+    check for broken symlinks............................(no files to check)Skipped
+    detect private key.......................................................Passed
+    fix end of files.....................................(no files to check)Skipped
+    trim trailing whitespace.............................(no files to check)Skipped
+    CRLF end-lines checker...............................(no files to check)Skipped
+    CRLF end-lines remover...............................(no files to check)Skipped
+    No-tabs checker......................................(no files to check)Skipped
+    Tabs remover.........................................(no files to check)Skipped
+    copyright_checker........................................................Passed
+   ```
+
+   But **don't panic**!
+   Our tooling will fix most of the style errors automatically. Some errors will need to be addressed manually. Fortunately, the error messages are straight forward and
+   the errors are usually simple to fix. After addressing the errors, you can run `git add <files>` and `git commit` again, which will trigger `pre-commit` again.
+   Once the `pre-commit` checks pass, you are ready to push the code.
+
+   [Google][http://google.com/] or [StackOverflow](https://stackoverflow.com/) are great tools to help you understand the code style errors.
+   Don't worry if you still can't figure it out. You can commit with `git commit -m "style error" --no-verify` and we are happy to help you once you create a Pull Request.
+
+#### 7. Keep pulling
+
+   An experienced Git user pulls from the official repo often -- daily or even hourly, so they notice conflicts with others work early, and it's easier to resolve smaller conflicts.
+
+   ```bash
+   git remote add upstream https://github.com/PaddlePaddle/PaddleNLP
+   git pull upstream develop
+   ```
+
+#### 8. Push and file a pull request
+
+   You can "push" your local work into your forked repo:
+
+   ```bash
+   git push origin my-cool-stuff
+   ```
+
+   The push allows you to create a pull request, requesting owners of this [official repo](https://github.com/PaddlePaddle/PaddleNLP) to pull your change into the official one.
+
+   To create a pull request, please follow [these steps](https://help.github.com/articles/creating-a-pull-request/).
+
+#### 9. Delete local and remote branches
+
+   To keep your local workspace and your fork clean, you might want to remove merged branches:
+
+   ```bash
+   git push origin my-cool-stuff
+   git checkout develop
+   git pull upstream develop
+   git branch -d my-cool-stuff
+   ```
+
+## Code Review
+
+-  Please feel free to ping your reviewers by @-mentioning the in the Pull Request.  Please do this after your pull request passes the CI.
+
+- Please answer reviewers' every comment.  If you are to follow the comment, please write "Done"; Otherwise, please start a discussion under the comment.
+
+- If you don't want your reviewers to get overwhelmed by email notifications, you might reply their comments by [in a batch](https://help.github.com/articles/reviewing-proposed-changes-in-a-pull-request/).

codecov.yml renamed to .github/codecov.yml

@@ -0,0 +1,48 @@
+name: FastTokenizer
+
+on:
+  push:
+    paths:
+      - 'fast_tokenizer/*'
+  pull_request:
+    paths:
+      - 'fast_tokenizer/*'
+
+jobs:
+  fast_tokenizer_cpp:
+    name: fast_tokenizer_cpp
+    runs-on: ubuntu-22.04
+    permissions:
+      pull-requests: write
+      contents: read
+      id-token: write
+    steps:
+      - uses: actions/checkout@v3
+      - name: compile
+        working-directory: ./fast_tokenizer
+        run: make fast_tokenizer_cpp_compile
+      - name: test
+        working-directory: ./fast_tokenizer
+        run: make fast_tokenizer_cpp_test
+  fast_tokenizer_python38:
+      name: fast_tokenizer_python38
+      runs-on: ubuntu-22.04
+      permissions:
+        pull-requests: write
+        contents: read
+        id-token: write
+      steps:
+        - uses: actions/checkout@v3
+        - uses: actions/setup-python@v1
+          with:
+            python-version: 3.8
+        - name: install
+          working-directory: ./fast_tokenizer
+          run: make fast_tokenizer_python_install
+        - name: compile
+          working-directory: ./fast_tokenizer
+          run: make fast_tokenizer_python_compile
+        - name: test
+          working-directory: ./fast_tokenizer
+          run: make fast_tokenizer_python_test
+  

.github/workflows/fast_tokenizer.yml

@@ -17,9 +17,10 @@ jobs:
           else
             echo "local develop branch exist, skipping"
           fi
-      - uses: actions/setup-python@v1
+      - uses: actions/setup-python@v4
         with:
           python-version: 3.8
+          cache: 'pip' # caching pip dependencies
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip

.github/workflows/lint.yml

@@ -12,9 +12,10 @@ jobs:
       id-token: write
     steps:
       - uses: actions/checkout@v3
-      - uses: actions/setup-python@v1
+      - uses: actions/setup-python@v4
         with:
           python-version: 3.8
+          cache: 'pip' # caching pip dependencies
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip

.github/workflows/tests.yml

@@ -7,7 +7,7 @@ repos:
         files: \.(py|pyi)$
         additional_dependencies: [toml]
 -   repo: https://github.com/PyCQA/isort
-    rev: 5.10.1
+    rev: 5.11.5
     hooks:
     -   id: isort
 -   repo: https://github.com/PyCQA/flake8

.pre-commit-config.yaml

@@ -21,8 +21,8 @@ sphinx:
 python:
   version: 3.8
   install:
-    - requirements: docs/requirements.txt
     - requirements: requirements.txt
+    - requirements: docs/requirements.txt
     - method: setuptools
       path: .
-  system_packages: true
+  system_packages: true