Skip to content
This repository was archived by the owner on Jan 15, 2024. It is now read-only.

Switch CI to Python 3.5 and declare Python 3.5 support #1009

Merged
merged 7 commits into from
Nov 15, 2019
Merged

Switch CI to Python 3.5 and declare Python 3.5 support #1009

merged 7 commits into from
Nov 15, 2019

Conversation

leezu
Copy link
Contributor

@leezu leezu commented Nov 15, 2019

Description

Fixes #994

cc @dmlc/gluon-nlp-team

@leezu leezu requested a review from a team as a code owner November 15, 2019 05:46
@codecov
Copy link

codecov bot commented Nov 15, 2019 •
edited
Loading

Codecov Report

Merging #1009 into master will increase coverage by 0.07%.
The diff coverage is 87.5%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1009      +/-   ##
==========================================
+ Coverage   89.85%   89.93%   +0.07%     
==========================================
  Files          67       67              
  Lines        6351     6340      -11     
==========================================
- Hits         5707     5702       -5     
+ Misses        644      638       -6
Impacted Files Coverage Δ
src/gluonnlp/data/utils.py 74.04% <ø> (ø) ⬆️
src/gluonnlp/utils/version.py 100% <ø> (ø) ⬆️
src/gluonnlp/utils/files.py 49.01% <0%> (ø) ⬆️
src/gluonnlp/data/stream.py 84.97% <0%> (ø) ⬆️
src/gluonnlp/model/highway.py 100% <100%> (ø) ⬆️
src/gluonnlp/model/bert.py 92.24% <100%> (ø) ⬆️
src/gluonnlp/model/convolutional_encoder.py 97.67% <100%> (ø) ⬆️
src/gluonnlp/model/language_model.py 98.49% <100%> (ø) ⬆️
src/gluonnlp/vocab/vocab.py 95.4% <100%> (ø) ⬆️
src/gluonnlp/data/batchify/embedding.py 97.69% <100%> (ø) ⬆️
... and 7 more

@leezu leezu force-pushed the cipy3.5 branch 3 times, most recently from 5447507 to e0c3c26 Compare November 15, 2019 05:53
@mli
Copy link
Member

mli commented Nov 15, 2019

Job PR-1009/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1009/4/index.html

@leezu leezu force-pushed the cipy3.5 branch 2 times, most recently from dd894e4 to d113c52 Compare November 15, 2019 08:49
@mli
Copy link
Member

mli commented Nov 15, 2019

Job PR-1009/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1009/8/index.html

@mli
Copy link
Member

mli commented Nov 15, 2019

Job PR-1009/10 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1009/10/index.html

@mli
Copy link
Member

mli commented Nov 15, 2019

Job PR-1009/11 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1009/11/index.html

@leezu leezu requested a review from sxjscience November 15, 2019 11:32
@mli
Copy link
Member

mli commented Nov 15, 2019

Job PR-1009/12 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1009/12/index.html

@mli
Copy link
Member

mli commented Nov 15, 2019

Job PR-1009/14 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1009/14/index.html

@leezu
Copy link
Contributor Author

leezu commented Nov 15, 2019

This currently removes multiprocessing.Pool usage with BERTDatasetTransform due to incompatibility with Python 3.5's pickle. Maybe we can refactor BERTDatasetTransform later.

@leezu
Delete Python3.5 incompatible multiprocessing.Pool usage of BERTDatas…
86f4621 
…etTransform

Traceback (most recent call last):
  File "./scripts/bert/finetune_classifier.py", line 373, in <module>
    bert_tokenizer, task, batch_size, dev_batch_size, args.max_len, vocabulary, args.pad)
  File "./scripts/bert/finetune_classifier.py", line 306, in preprocess_data
    data_train = mx.gluon.data.SimpleDataset(pool.map(trans, train_tsv))
  File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3-master@6/conda/gpu/py3-master/lib/python3.5/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3-master@6/conda/gpu/py3-master/lib/python3.5/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3-master@6/conda/gpu/py3-master/lib/python3.5/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3-master@6/conda/gpu/py3-master/lib/python3.5/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3-master@6/conda/gpu/py3-master/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'module'>: attribute lookup module on builtins failed
@mli
Copy link
Member

mli commented Nov 15, 2019

Job PR-1009/16 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1009/16/index.html

@sxjscience
Copy link
Member

In terms of python3.5 v.s. python3.6, I think we can move to python3.6 in the next-gen (1.0) release. What do you think?

@leezu leezu merged commit 5e11334 into dmlc:master Nov 15, 2019
@leezu leezu deleted the cipy3.5 branch November 15, 2019 23:40
@leezu leezu mentioned this pull request Nov 18, 2019
Merged
6 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
1 more reviewer

@sxjscience sxjscience sxjscience approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support Python 3.5
3 participants
@leezu @mli @sxjscience