pylint Python 3 porting report for `idigbio_ingestion`

[danstoner]

~/git/IDIGBIO/idb-backend]$ pylint --py3k idigbio_ingestion
************* Module idigbio_ingestion.db_check
idigbio_ingestion/db_check.py:304:56: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
idigbio_ingestion/db_check.py:308:52: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
idigbio_ingestion/db_check.py:395:13: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
idigbio_ingestion/db_check.py:396:13: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.db_rsids
idigbio_ingestion/db_rsids.py:1:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.verify_ceph_objects
idigbio_ingestion/verify_ceph_objects.py:49:31: W1645: Exception.message removed in Python 3 (exception-message-attribute)
************* Module idigbio_ingestion.update_publisher_recordset
idigbio_ingestion/update_publisher_recordset.py:2:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.lib.eml
idigbio_ingestion/lib/eml.py:2:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
idigbio_ingestion/lib/eml.py:141:15: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.lib.util
idigbio_ingestion/lib/util.py:1:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.lib.delimited
idigbio_ingestion/lib/delimited.py:1:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
idigbio_ingestion/lib/delimited.py:97:20: W1654: dict.items referenced when not iterating (dict-items-not-iterating)
idigbio_ingestion/lib/delimited.py:122:4: W1653: next method defined (next-method-defined)
************* Module idigbio_ingestion.lib.fileproxy
idigbio_ingestion/lib/fileproxy.py:60:4: W1653: next method defined (next-method-defined)
idigbio_ingestion/lib/fileproxy.py:61:12: W1622: Called a next() method on an object (next-method-called)
idigbio_ingestion/lib/fileproxy.py:108:4: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.lib.dwca
idigbio_ingestion/lib/dwca.py:1:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.lib.xmlDictTools
idigbio_ingestion/lib/xmlDictTools.py:5:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.mediaing.fetcher
idigbio_ingestion/mediaing/fetcher.py:49:12: W1656: dict.values referenced when not iterating (dict-values-not-iterating)
idigbio_ingestion/mediaing/fetcher.py:116:16: W1636: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.mediaing.derivatives
idigbio_ingestion/mediaing/derivatives.py:87:39: W1639: filter built-in referenced when not iterating (filter-builtin-not-iterating)
idigbio_ingestion/mediaing/derivatives.py:180:16: W1636: map built-in referenced when not iterating (map-builtin-not-iterating)

-----------------------------------
Your code has been rated at 9.90/10

[danstoner]

From that big list, the following are the ones that would need to befixed for Python 3 conversion project:

W1655 dict-keys-not-iterating
Message
'dict.keys referenced when not iterating'

Description
Used when dict.keys is referenced in a non-iterating context (returns an iterator in Python 3)

W1653 next-method-defined
Message
'next method defined'

Description
Used when a next method is defined that would be an iterator in Python 2 but is treated as a normal function in Python 3.

W1622 next-method-called
Message
'Called a next() method on an object'

Description
Used when an object’s next() method is called (Python 3 uses the next() built-in function)

W1639 filter-builtin-not-iterating
Message
'filter built-in referenced when not iterating'

Description
Used when the filter built-in is referenced in a non-iterating context (returns an iterator in Python 3)

W1636 map-builtin-not-iterating
Message
'map built-in referenced when not iterating'

Description
Used when the map built-in is referenced in a non-iterating context (returns an iterator in Python 3)

Issues in verify_ceph_objects.py can wait since that is not a program we are going to run on a regular basis, or possibly, ever again.

[danstoner]

Get rid of the no-absolute-import lines (basically a false positive since we don't need that in py3):

pylint --py3k --disable=no-absolute-import   idigbio_ingestion
************* Module idigbio_ingestion.verify_ceph_objects
idigbio_ingestion/verify_ceph_objects.py:49:31: W1645: Exception.message removed in Python 3 (exception-message-attribute)
************* Module idigbio_ingestion.db_check
idigbio_ingestion/db_check.py:308:52: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.lib.fileproxy
idigbio_ingestion/lib/fileproxy.py:60:4: W1653: next method defined (next-method-defined)
idigbio_ingestion/lib/fileproxy.py:61:12: W1622: Called a next() method on an object (next-method-called)
************* Module idigbio_ingestion.lib.delimited
idigbio_ingestion/lib/delimited.py:97:20: W1654: dict.items referenced when not iterating (dict-items-not-iterating)
idigbio_ingestion/lib/delimited.py:122:4: W1653: next method defined (next-method-defined)
************* Module idigbio_ingestion.lib.eml
idigbio_ingestion/lib/eml.py:141:15: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.mediaing.fetcher
idigbio_ingestion/mediaing/fetcher.py:48:12: W1656: dict.values referenced when not iterating (dict-values-not-iterating)
idigbio_ingestion/mediaing/fetcher.py:115:16: W1636: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.mediaing.derivatives
idigbio_ingestion/mediaing/derivatives.py:87:39: W1639: filter built-in referenced when not iterating (filter-builtin-not-iterating)
idigbio_ingestion/mediaing/derivatives.py:180:16: W1636: map built-in referenced when not iterating (map-builtin-not-iterating)

------------------------------------------------------------------

[danstoner]

The --py3k option only works in python2 pylint.

Updated 3/23/2022:

pylint --py3k --disable=no-absolute-import   idigbio_ingestion
No config file found, using default configuration
************* Module idigbio_ingestion.db_rsids
E: 36, 8: print statement used (print-statement)
E: 41,12: print statement used (print-statement)
E: 51, 8: print statement used (print-statement)
************* Module idigbio_ingestion.verify_ceph_objects
W: 49,31: Exception.message removed in Python 3 (exception-message-attribute)
************* Module idigbio_ingestion.mediaing.fetcher
W: 48,12: dict.values referenced when not iterating (dict-values-not-iterating)
W:115,16: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.mediaing.derivatives
W: 87,39: filter built-in referenced when not iterating (filter-builtin-not-iterating)
W:180,16: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.lib.xmlDictTools
E:147, 4: print statement used (print-statement)
E:270, 4: print statement used (print-statement)
************* Module idigbio_ingestion.lib.delimited
W: 97,20: dict.items referenced when not iterating (dict-items-not-iterating)
W:122, 4: next method defined (next-method-defined)
************* Module idigbio_ingestion.lib.eml
W:141,15: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.lib.dwca
E:101,28: print statement used (print-statement)
************* Module idigbio_ingestion.lib.fileproxy
W: 60, 4: next method defined (next-method-defined)
W: 61,12: Called a next() method on an object (next-method-called)
E:120,12: print statement used (print-statement)

-----------------------------------
Your code has been rated at 9.76/10

[danstoner]

The --py3k option generates false positives since it must run under Python 2.

Disable checks that are fixed and ready for Python3 at this point in the convert_python_3 branch.

This is the "final" list of issues needed to make idigbio_ingestion compatible with Python 3:

$ pylint --py3k --disable=no-absolute-import,print-statement,exception-message-attribute   idigbio_ingestion
No config file found, using default configuration
************* Module idigbio_ingestion.mediaing.fetcher
W: 48,12: dict.values referenced when not iterating (dict-values-not-iterating)
W:115,16: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.mediaing.derivatives
W: 87,39: filter built-in referenced when not iterating (filter-builtin-not-iterating)
W:180,16: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.lib.delimited
W: 97,20: dict.items referenced when not iterating (dict-items-not-iterating)
W:122, 4: next method defined (next-method-defined)
************* Module idigbio_ingestion.lib.eml
W:141,15: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.lib.fileproxy
W: 60, 4: next method defined (next-method-defined)
W: 61,12: Called a next() method on an object (next-method-called)

------------------------------------------------------------------
Your code has been rated at 9.95/10 (previous run: 9.95/10, +0.01)

[danstoner]

Tried to run pylint under python3 and search the output for just those issues found above. Strangely, these have extra error in parens, not sure how that relates and the codes like E0011 don't seem to match the expected lint error. I can no longer find the website that I recall having good information on these.

And python3 linter does not actually find the same issues The two items for mediaing.fetcher are in the pylint output but not filter-builtin-not-iterating, dict-items-not-iterating, and the others.

$ pylint --version
pylint 2.12.2
astroid 2.9.3
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0]

$ pylint --output=pylint3.txt idigbio_ingestion

These are the ones that match the --py3k list:

idigbio_ingestion/mediaing/fetcher.py:48:0: E0011: Unrecognized file option 'dict-values-not-iterating' (unrecognized-inline-option)

idigbio_ingestion/mediaing/fetcher.py:115:0: E0011: Unrecognized file option 'map-builtin-not-iterating' (unrecognized-inline-option)

[danstoner]

Errors only:

(venv3) (convert_python_3) dan@dan-Latitude-7370:~/git/IDIGBIO/idb-backend$ pylint --errors-only --output=E.pylint3.20230209.txt idigbio_ingestion
(venv3) (convert_python_3) dan@dan-Latitude-7370:~/git/IDIGBIO/idb-backend$ cat E.pylint3.20230209.txt 
************* Module idigbio_ingestion.db_rsids
idigbio_ingestion/db_rsids.py:43:12: E1101: Class 'traceback' has no 'print_exc' member (no-member)
idigbio_ingestion/db_rsids.py:43:12: E0602: Undefined variable 'traceback' (undefined-variable)
************* Module idigbio_ingestion.lib.fileproxy
idigbio_ingestion/lib/fileproxy.py:91:4: E0301: __iter__ returns non-iterator (non-iterator-returned)
************* Module idigbio_ingestion.lib.xmlDictTools
idigbio_ingestion/lib/xmlDictTools.py:132:4: E0401: Unable to import 'StringIO' (import-error)
************* Module idigbio_ingestion.mediaing.fetcher
idigbio_ingestion/mediaing/fetcher.py:48:0: E0011: Unrecognized file option 'dict-values-not-iterating' (unrecognized-inline-option)
idigbio_ingestion/mediaing/fetcher.py:115:0: E0011: Unrecognized file option 'map-builtin-not-iterating' (unrecognized-inline-option)
************* Module idigbio_ingestion.mediaing.cli
idigbio_ingestion/mediaing/cli.py:18:8: E1101: Module 'click' has no 'abort' member; maybe 'Abort'? (no-member)