Multi-level package namespaces break with Setuptools 38

[georgevreilly]

Multi-level packages imported via setup.py develop work in Setuptools 37, but fail in Setuptools 38.x due to missing __init__.py in the upper levels of the namespace.

For example, after installing protobuf, a simple import google.protobuf.message should work but doesn't. It can be hacked around by touch protobuf-3.5.0.post1-py2.7.egg/google/__init__.py in site-packages.

Setuptools 37.0

~/tmp/s38.py:

from setuptools import setup

setup(
    name="s38",
    version="0.0.1",
    install_requires=['protobuf', 'zope.deprecation', 'pastedeploy']
)

$ virtualenv --no-setuptools -p python2.7 setuptools37
Running virtualenv with interpreter /usr/local/bin/python2.7
New python executable in /Users/georgevreilly/setuptools37/bin/python2.7
Also creating executable in /Users/georgevreilly/setuptools37/bin/python
Installing pip, wheel...done.

$ source ./setuptools37/bin/activate

(setuptools37) $ pip install 'setuptools==37.0'
Collecting setuptools==37.0
  Using cached setuptools-37.0.0-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-37.0.0

(setuptools37) $ python -c 'import setuptools; print(setuptools.__version__)'
37.0.0

(setuptools37) $ python ~/tmp/s38.py develop
    running develop
    running egg_info
    writing requirements to s38.egg-info/requires.txt
    writing s38.egg-info/PKG-INFO
    writing top-level names to s38.egg-info/top_level.txt
    writing dependency_links to s38.egg-info/dependency_links.txt
    reading manifest file 's38.egg-info/SOURCES.txt'
    writing manifest file 's38.egg-info/SOURCES.txt'
    running build_ext
    Creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/s38.egg-link (link to .)
    Adding s38 0.0.1 to easy-install.pth file

    Installed /Users/georgevreilly
    Processing dependencies for s38==0.0.1
    Searching for pastedeploy
    Reading https://pypi.python.org/simple/pastedeploy/
    Downloading https://pypi.python.org/packages/0f/90/8e20cdae206c543ea10793cbf4136eb9a8b3f417e04e40a29d72d9922cbd/PasteDeploy-1.5.2.tar.gz#md5=352b7205c78c8de4987578d19431af3b
    Best match: PasteDeploy 1.5.2
    Processing PasteDeploy-1.5.2.tar.gz
    Writing /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-TZTFFm/PasteDeploy-1.5.2/setup.cfg
    Running PasteDeploy-1.5.2/setup.py -q bdist_egg --dist-dir /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-TZTFFm/PasteDeploy-1.5.2/egg-dist-tmp-a2YwV8
    creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg
    Extracting PasteDeploy-1.5.2-py2.7.egg to /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Adding PasteDeploy 1.5.2 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg
    Searching for zope.deprecation
    Reading https://pypi.python.org/simple/zope.deprecation/
    Downloading https://pypi.python.org/packages/a1/18/2dc5e6bfe64fdc3b79411b67464c55bb0b43b127051a20f7f492ab767758/zope.deprecation-4.3.0.tar.gz#md5=2166b2cb7e0e96a21104e6f8f9b696bb
    Best match: zope.deprecation 4.3.0
    Processing zope.deprecation-4.3.0.tar.gz
    Writing /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-hWzALQ/zope.deprecation-4.3.0/setup.cfg
    Running zope.deprecation-4.3.0/setup.py -q bdist_egg --dist-dir /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-hWzALQ/zope.deprecation-4.3.0/egg-dist-tmp-hANPOl
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/zope.deprecation-4.3.0-py2.7.egg
    Extracting zope.deprecation-4.3.0-py2.7.egg to /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Adding zope.deprecation 4.3.0 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/zope.deprecation-4.3.0-py2.7.egg
    Searching for protobuf
    Reading https://pypi.python.org/simple/protobuf/
    Downloading https://pypi.python.org/packages/e9/96/61a46b3cb4b25e4eb1b685266a98463f4a1887b7d4bf8142986c443215d2/protobuf-3.5.0.post1-py2.7.egg#md5=17755f6139c1d85aa75d1c4d77d04cf5
    Best match: protobuf 3.5.0.post1
    Processing protobuf-3.5.0.post1-py2.7.egg
    creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/protobuf-3.5.0.post1-py2.7.egg
    Extracting protobuf-3.5.0.post1-py2.7.egg to /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Adding protobuf 3.5.0.post1 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/protobuf-3.5.0.post1-py2.7.egg
    Searching for six>=1.9
    Reading https://pypi.python.org/simple/six/
    Downloading https://pypi.python.org/packages/16/d8/bc6316cf98419719bd59c91742194c111b6f2e85abac88e496adefaf7afe/six-1.11.0.tar.gz#md5=d12789f9baf7e9fb2524c0c64f1773f8
    Best match: six 1.11.0
    Processing six-1.11.0.tar.gz
    Writing /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-84SWpD/six-1.11.0/setup.cfg
    Running six-1.11.0/setup.py -q bdist_egg --dist-dir /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-84SWpD/six-1.11.0/egg-dist-tmp-Kh1NXs
    no previously-included directories found matching 'documentation/_build'
    zip_safe flag not set; analyzing archive contents...
    six: module references __path__
    creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/six-1.11.0-py2.7.egg
    Extracting six-1.11.0-py2.7.egg to /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Adding six 1.11.0 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/six-1.11.0-py2.7.egg
    Searching for setuptools==37.0.0
    Best match: setuptools 37.0.0
    Adding setuptools 37.0.0 to easy-install.pth file
    Installing easy_install script to /Users/georgevreilly/setuptools37/bin
    Installing easy_install-3.6 script to /Users/georgevreilly/setuptools37/bin

    Using /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Finished processing dependencies for s38==0.0.1

(setuptools37) $ lssitepackages
    PasteDeploy-1.5.2-py2.7.egg      pkg_resources                    six-1.11.0-py2.7.egg
    easy-install.pth                 protobuf-3.5.0.post1-py2.7.egg   wheel
    easy_install.py                  s38.egg-link                     wheel-0.30.0.dist-info
    easy_install.pyc                 setuptools                       zope.deprecation-4.3.0-py2.7.egg
    pip                              setuptools-37.0.0.dist-info
    pip-9.0.1.dist-info              setuptools.pth

(setuptools37) $ python -c 'from paste.deploy.loadwsgi import AttrDict' && echo yes
yes

(setuptools37) $ python -c 'import google.protobuf.message' && echo yes
yes

(setuptools37) $ python -c 'from zope.deprecation import deprecated' && echo yes
yes

Setuptools 38.2.1

$ virtualenv -p python2.7 setuptools38b
    Running virtualenv with interpreter /usr/local/bin/python2.7
    New python executable in /Users/georgevreilly/setuptools38b/bin/python2.7
    Also creating executable in /Users/georgevreilly/setuptools38b/bin/python
    Installing setuptools, pip, wheel...done.

$ source setuptools38b/bin/activate

(setuptools38b) $ python -c 'import setuptools; print(setuptools.__version__)'
38.2.1

(setuptools38b) $ python ~/tmp/s38.py develop
    running develop
    running egg_info
    creating s38.egg-info
    writing requirements to s38.egg-info/requires.txt
    writing s38.egg-info/PKG-INFO
    writing top-level names to s38.egg-info/top_level.txt
    writing dependency_links to s38.egg-info/dependency_links.txt
    writing manifest file 's38.egg-info/SOURCES.txt'
    reading manifest file 's38.egg-info/SOURCES.txt'
    writing manifest file 's38.egg-info/SOURCES.txt'
    running build_ext
    Creating /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/s38.egg-link (link to .)
    Adding s38 0.0.1 to easy-install.pth file

    Installed /Users/georgevreilly
    Processing dependencies for s38==0.0.1
    Searching for pastedeploy
    Reading https://pypi.python.org/simple/pastedeploy/
    Downloading https://pypi.python.org/packages/31/28/51201a54aeecbd02eff767d17050b302f6fd98fdfecb4e3f4c9301ba6ef8/PasteDeploy-1.5.2-py2.py3-none-any.whl#md5=1134e25af0d52190abd9f8d7c6867f9e
    Best match: PasteDeploy 1.5.2
    Processing PasteDeploy-1.5.2-py2.py3-none-any.whl
    Installing PasteDeploy-1.5.2-py2.py3-none-any.whl to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    writing requirements to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg/EGG-INFO/requires.txt
    Adding PasteDeploy 1.5.2 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg
    Searching for zope.deprecation
    Reading https://pypi.python.org/simple/zope.deprecation/
    Downloading https://pypi.python.org/packages/ee/33/625098914ec59b3006adf2cdf44a721e9671f4836af9eeb8cbe14e485954/zope.deprecation-4.3.0-py2.py3-none-any.whl#md5=fb3b24ef645dcfaff8ab3028da7209f2
    Best match: zope.deprecation 4.3.0
    Processing zope.deprecation-4.3.0-py2.py3-none-any.whl
    Installing zope.deprecation-4.3.0-py2.py3-none-any.whl to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    writing requirements to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/zope.deprecation-4.3.0-py2.7.egg/EGG-INFO/requires.txt
    Adding zope.deprecation 4.3.0 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/zope.deprecation-4.3.0-py2.7.egg
    Searching for protobuf
    Reading https://pypi.python.org/simple/protobuf/
    Downloading https://pypi.python.org/packages/74/34/52e4fcc023f36ba8c408765032e6b9052eff115b01a17e3d2af48fac3a97/protobuf-3.5.0.post1-py2.py3-none-any.whl#md5=74c4b42404b6f55348c280dc511b95c1
    Best match: protobuf 3.5.0.post1
    Processing protobuf-3.5.0.post1-py2.py3-none-any.whl
    Installing protobuf-3.5.0.post1-py2.py3-none-any.whl to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    writing requirements to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/protobuf-3.5.0.post1-py2.7.egg/EGG-INFO/requires.txt
    Adding protobuf 3.5.0.post1 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/protobuf-3.5.0.post1-py2.7.egg
    Searching for six>=1.9
    Reading https://pypi.python.org/simple/six/
    Downloading https://pypi.python.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl#md5=866ab722be6bdfed6830f3179af65468
    Best match: six 1.11.0
    Processing six-1.11.0-py2.py3-none-any.whl
    Installing six-1.11.0-py2.py3-none-any.whl to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    Adding six 1.11.0 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/six-1.11.0-py2.7.egg
    Searching for setuptools==38.2.1
    Best match: setuptools 38.2.1
    Adding setuptools 38.2.1 to easy-install.pth file
    Installing easy_install script to /Users/georgevreilly/setuptools38b/bin
    Installing easy_install-3.6 script to /Users/georgevreilly/setuptools38b/bin

    Using /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    Finished processing dependencies for s38==0.0.1

(setuptools38b) $ lssitepackages
    PasteDeploy-1.5.2-py2.7.egg      pkg_resources                    six-1.11.0-py2.7.egg
    easy-install.pth                 protobuf-3.5.0.post1-py2.7.egg   wheel
    easy_install.py                  s38.egg-link                     wheel-0.30.0.dist-info
    easy_install.pyc                 setuptools                       zope.deprecation-4.3.0-py2.7.egg
    pip                              setuptools-38.2.1.dist-info
    pip-9.0.1.dist-info              setuptools.pth

(setuptools38b) $ python -c 'from paste.deploy.loadwsgi import AttrDict' && echo yes
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named paste.deploy.loadwsgi

(setuptools38b) $ python -c 'import google.protobuf.message' && echo yes
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named google.protobuf.message

(setuptools38b) $ python -c 'from zope.deprecation import deprecated' && echo yes
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named zope.deprecation

Diffing the Trees

(setuptools38b) $ diff -u <(tree setuptools37/lib/python2.7/site-packages) <(tree setuptools38b/lib/python2.7/site-packages)
--- /dev/fd/63	2017-11-26 19:06:51.000000000 -0800
+++ /dev/fd/62	2017-11-26 19:06:51.000000000 -0800
@@ -1,28 +1,23 @@
-setuptools37/lib/python2.7/site-packages
+setuptools38b/lib/python2.7/site-packages
 ├── PasteDeploy-1.5.2-py2.7.egg
 │   ├── EGG-INFO
+│   │   ├── DESCRIPTION.rst
 │   │   ├── PKG-INFO
-│   │   ├── SOURCES.txt
-│   │   ├── dependency_links.txt
+│   │   ├── RECORD
+│   │   ├── WHEEL
 │   │   ├── entry_points.txt
 │   │   ├── namespace_packages.txt
-│   │   ├── not-zip-safe
+│   │   ├── pydist.json
 │   │   ├── requires.txt
 │   │   └── top_level.txt
+│   ├── PasteDeploy-1.5.2-py2.6-nspkg.pth
 │   └── paste
-│       ├── __init__.py
-│       ├── __init__.pyc
 │       └── deploy
 │           ├── __init__.py
-│           ├── __init__.pyc
 │           ├── compat.py
-│           ├── compat.pyc
 │           ├── config.py
-│           ├── config.pyc
 │           ├── converters.py
-│           ├── converters.pyc
 │           ├── loadwsgi.py
-│           ├── loadwsgi.pyc
 │           ├── paster_templates
 │           │   └── paste_deploy
 │           │       ├── +package+
@@ -31,9 +26,7 @@
 │           │       └── docs
 │           │           └── devel_config.ini_tmpl
 │           ├── paster_templates.py
-│           ├── paster_templates.pyc
-│           ├── util.py
-│           └── util.pyc
+│           └── util.py
 ├── easy-install.pth
 ├── easy_install.py
 ├── easy_install.pyc
@@ -607,210 +600,116 @@
 │   └── py31compat.pyc
 ├── protobuf-3.5.0.post1-py2.7.egg
 │   ├── EGG-INFO
+│   │   ├── DESCRIPTION.rst
 │   │   ├── PKG-INFO
-│   │   ├── SOURCES.txt
-│   │   ├── dependency_links.txt
+│   │   ├── RECORD
+│   │   ├── WHEEL
+│   │   ├── metadata.json
 │   │   ├── namespace_packages.txt
-│   │   ├── not-zip-safe
 │   │   ├── requires.txt
 │   │   └── top_level.txt
-│   └── google
-│       ├── __init__.py
-│       ├── __init__.pyc
-│       └── protobuf
-│           ├── __init__.py
-│           ├── __init__.pyc
-│           ├── any_pb2.py
-│           ├── any_pb2.pyc
-│           ├── any_test_pb2.py
-│           ├── any_test_pb2.pyc
-│           ├── api_pb2.py
-│           ├── api_pb2.pyc
-│           ├── compiler
-│           │   ├── __init__.py
-│           │   ├── __init__.pyc
-│           │   ├── plugin_pb2.py
-│           │   └── plugin_pb2.pyc
-│           ├── descriptor.py
-│           ├── descriptor.pyc
-│           ├── descriptor_database.py
-│           ├── descriptor_database.pyc
-│           ├── descriptor_pb2.py
-│           ├── descriptor_pb2.pyc
-│           ├── descriptor_pool.py
-│           ├── descriptor_pool.pyc
-│           ├── duration_pb2.py
-│           ├── duration_pb2.pyc
-│           ├── empty_pb2.py
-│           ├── empty_pb2.pyc
-│           ├── field_mask_pb2.py
-│           ├── field_mask_pb2.pyc
-│           ├── internal
-│           │   ├── __init__.py
-│           │   ├── __init__.pyc
-│           │   ├── _parameterized.py
-│           │   ├── _parameterized.pyc
-│           │   ├── any_test_pb2.py
-│           │   ├── any_test_pb2.pyc
-│           │   ├── api_implementation.py
-│           │   ├── api_implementation.pyc
-│           │   ├── containers.py
-│           │   ├── containers.pyc
-│           │   ├── decoder.py
-│           │   ├── decoder.pyc
-│           │   ├── descriptor_database_test.py
-│           │   ├── descriptor_database_test.pyc
-│           │   ├── descriptor_pool_test.py
-│           │   ├── descriptor_pool_test.pyc
-│           │   ├── descriptor_pool_test1_pb2.py
-│           │   ├── descriptor_pool_test1_pb2.pyc
-│           │   ├── descriptor_pool_test2_pb2.py
-│           │   ├── descriptor_pool_test2_pb2.pyc
-│           │   ├── descriptor_test.py
-│           │   ├── descriptor_test.pyc
-│           │   ├── encoder.py
-│           │   ├── encoder.pyc
-│           │   ├── enum_type_wrapper.py
-│           │   ├── enum_type_wrapper.pyc
-│           │   ├── factory_test1_pb2.py
-│           │   ├── factory_test1_pb2.pyc
-│           │   ├── factory_test2_pb2.py
-│           │   ├── factory_test2_pb2.pyc
-│           │   ├── file_options_test_pb2.py
-│           │   ├── file_options_test_pb2.pyc
-│           │   ├── generator_test.py
-│           │   ├── generator_test.pyc
-│           │   ├── import_test_package
-│           │   │   ├── __init__.py
-│           │   │   ├── __init__.pyc
-│           │   │   ├── inner_pb2.py
-│           │   │   ├── inner_pb2.pyc
-│           │   │   ├── outer_pb2.py
-│           │   │   └── outer_pb2.pyc
-│           │   ├── json_format_test.py
-│           │   ├── json_format_test.pyc
-│           │   ├── message_factory_test.py
-│           │   ├── message_factory_test.pyc
-│           │   ├── message_listener.py
-│           │   ├── message_listener.pyc
-│           │   ├── message_set_extensions_pb2.py
-│           │   ├── message_set_extensions_pb2.pyc
-│           │   ├── message_test.py
-│           │   ├── message_test.pyc
-│           │   ├── missing_enum_values_pb2.py
-│           │   ├── missing_enum_values_pb2.pyc
-│           │   ├── more_extensions_dynamic_pb2.py
-│           │   ├── more_extensions_dynamic_pb2.pyc
-│           │   ├── more_extensions_pb2.py
-│           │   ├── more_extensions_pb2.pyc
-│           │   ├── more_messages_pb2.py
-│           │   ├── more_messages_pb2.pyc
-│           │   ├── packed_field_test_pb2.py
-│           │   ├── packed_field_test_pb2.pyc
-│           │   ├── proto_builder_test.py
-│           │   ├── proto_builder_test.pyc
-│           │   ├── python_message.py
-│           │   ├── python_message.pyc
-│           │   ├── reflection_test.py
-│           │   ├── reflection_test.pyc
-│           │   ├── service_reflection_test.py
-│           │   ├── service_reflection_test.pyc
-│           │   ├── symbol_database_test.py
-│           │   ├── symbol_database_test.pyc
-│           │   ├── test_bad_identifiers_pb2.py
-│           │   ├── test_bad_identifiers_pb2.pyc
-│           │   ├── test_util.py
-│           │   ├── test_util.pyc
-│           │   ├── testing_refleaks.py
-│           │   ├── testing_refleaks.pyc
-│           │   ├── text_encoding_test.py
-│           │   ├── text_encoding_test.pyc
-│           │   ├── text_format_test.py
-│           │   ├── text_format_test.pyc
-│           │   ├── type_checkers.py
-│           │   ├── type_checkers.pyc
-│           │   ├── unknown_fields_test.py
-│           │   ├── unknown_fields_test.pyc
-│           │   ├── well_known_types.py
-│           │   ├── well_known_types.pyc
-│           │   ├── well_known_types_test.py
-│           │   ├── well_known_types_test.pyc
-│           │   ├── wire_format.py
-│           │   ├── wire_format.pyc
-│           │   ├── wire_format_test.py
-│           │   └── wire_format_test.pyc
-│           ├── json_format.py
-│           ├── json_format.pyc
-│           ├── map_proto2_unittest_pb2.py
-│           ├── map_proto2_unittest_pb2.pyc
-│           ├── map_unittest_pb2.py
-│           ├── map_unittest_pb2.pyc
-│           ├── message.py
-│           ├── message.pyc
-│           ├── message_factory.py
-│           ├── message_factory.pyc
-│           ├── proto_builder.py
-│           ├── proto_builder.pyc
-│           ├── pyext
-│           │   ├── __init__.py
-│           │   ├── __init__.pyc
-│           │   ├── cpp_message.py
-│           │   ├── cpp_message.pyc
-│           │   ├── python_pb2.py
-│           │   └── python_pb2.pyc
-│           ├── reflection.py
-│           ├── reflection.pyc
-│           ├── service.py
-│           ├── service.pyc
-│           ├── service_reflection.py
-│           ├── service_reflection.pyc
-│           ├── source_context_pb2.py
-│           ├── source_context_pb2.pyc
-│           ├── struct_pb2.py
-│           ├── struct_pb2.pyc
-│           ├── symbol_database.py
-│           ├── symbol_database.pyc
-│           ├── test_messages_proto2_pb2.py
-│           ├── test_messages_proto2_pb2.pyc
-│           ├── test_messages_proto3_pb2.py
-│           ├── test_messages_proto3_pb2.pyc
-│           ├── text_encoding.py
-│           ├── text_encoding.pyc
-│           ├── text_format.py
-│           ├── text_format.pyc
-│           ├── timestamp_pb2.py
-│           ├── timestamp_pb2.pyc
-│           ├── type_pb2.py
-│           ├── type_pb2.pyc
-│           ├── unittest_arena_pb2.py
-│           ├── unittest_arena_pb2.pyc
-│           ├── unittest_custom_options_pb2.py
-│           ├── unittest_custom_options_pb2.pyc
-│           ├── unittest_import_pb2.py
-│           ├── unittest_import_pb2.pyc
-│           ├── unittest_import_public_pb2.py
-│           ├── unittest_import_public_pb2.pyc
-│           ├── unittest_mset_pb2.py
-│           ├── unittest_mset_pb2.pyc
-│           ├── unittest_mset_wire_format_pb2.py
-│           ├── unittest_mset_wire_format_pb2.pyc
-│           ├── unittest_no_arena_import_pb2.py
-│           ├── unittest_no_arena_import_pb2.pyc
-│           ├── unittest_no_arena_pb2.py
-│           ├── unittest_no_arena_pb2.pyc
-│           ├── unittest_no_generic_services_pb2.py
-│           ├── unittest_no_generic_services_pb2.pyc
-│           ├── unittest_pb2.py
-│           ├── unittest_pb2.pyc
-│           ├── unittest_proto3_arena_pb2.py
-│           ├── unittest_proto3_arena_pb2.pyc
-│           ├── util
-│           │   ├── __init__.py
-│           │   ├── __init__.pyc
-│           │   ├── json_format_proto3_pb2.py
-│           │   └── json_format_proto3_pb2.pyc
-│           ├── wrappers_pb2.py
-│           └── wrappers_pb2.pyc
+│   ├── google
+│   │   └── protobuf
+│   │       ├── __init__.py
+│   │       ├── any_pb2.py
+│   │       ├── any_test_pb2.py
+│   │       ├── api_pb2.py
+│   │       ├── compiler
+│   │       │   ├── __init__.py
+│   │       │   └── plugin_pb2.py
+│   │       ├── descriptor.py
+│   │       ├── descriptor_database.py
+│   │       ├── descriptor_pb2.py
+│   │       ├── descriptor_pool.py
+│   │       ├── duration_pb2.py
+│   │       ├── empty_pb2.py
+│   │       ├── field_mask_pb2.py
+│   │       ├── internal
+│   │       │   ├── __init__.py
+│   │       │   ├── _parameterized.py
+│   │       │   ├── any_test_pb2.py
+│   │       │   ├── api_implementation.py
+│   │       │   ├── containers.py
+│   │       │   ├── decoder.py
+│   │       │   ├── descriptor_database_test.py
+│   │       │   ├── descriptor_pool_test.py
+│   │       │   ├── descriptor_pool_test1_pb2.py
+│   │       │   ├── descriptor_pool_test2_pb2.py
+│   │       │   ├── descriptor_test.py
+│   │       │   ├── encoder.py
+│   │       │   ├── enum_type_wrapper.py
+│   │       │   ├── factory_test1_pb2.py
+│   │       │   ├── factory_test2_pb2.py
+│   │       │   ├── file_options_test_pb2.py
+│   │       │   ├── generator_test.py
+│   │       │   ├── import_test_package
+│   │       │   │   ├── __init__.py
+│   │       │   │   ├── inner_pb2.py
+│   │       │   │   └── outer_pb2.py
+│   │       │   ├── json_format_test.py
+│   │       │   ├── message_factory_test.py
+│   │       │   ├── message_listener.py
+│   │       │   ├── message_set_extensions_pb2.py
+│   │       │   ├── message_test.py
+│   │       │   ├── missing_enum_values_pb2.py
+│   │       │   ├── more_extensions_dynamic_pb2.py
+│   │       │   ├── more_extensions_pb2.py
+│   │       │   ├── more_messages_pb2.py
+│   │       │   ├── packed_field_test_pb2.py
+│   │       │   ├── proto_builder_test.py
+│   │       │   ├── python_message.py
+│   │       │   ├── reflection_test.py
+│   │       │   ├── service_reflection_test.py
+│   │       │   ├── symbol_database_test.py
+│   │       │   ├── test_bad_identifiers_pb2.py
+│   │       │   ├── test_util.py
+│   │       │   ├── testing_refleaks.py
+│   │       │   ├── text_encoding_test.py
+│   │       │   ├── text_format_test.py
+│   │       │   ├── type_checkers.py
+│   │       │   ├── unknown_fields_test.py
+│   │       │   ├── well_known_types.py
+│   │       │   ├── well_known_types_test.py
+│   │       │   ├── wire_format.py
+│   │       │   └── wire_format_test.py
+│   │       ├── json_format.py
+│   │       ├── map_proto2_unittest_pb2.py
+│   │       ├── map_unittest_pb2.py
+│   │       ├── message.py
+│   │       ├── message_factory.py
+│   │       ├── proto_builder.py
+│   │       ├── pyext
+│   │       │   ├── __init__.py
+│   │       │   ├── cpp_message.py
+│   │       │   └── python_pb2.py
+│   │       ├── reflection.py
+│   │       ├── service.py
+│   │       ├── service_reflection.py
+│   │       ├── source_context_pb2.py
+│   │       ├── struct_pb2.py
+│   │       ├── symbol_database.py
+│   │       ├── test_messages_proto2_pb2.py
+│   │       ├── test_messages_proto3_pb2.py
+│   │       ├── text_encoding.py
+│   │       ├── text_format.py
+│   │       ├── timestamp_pb2.py
+│   │       ├── type_pb2.py
+│   │       ├── unittest_arena_pb2.py
+│   │       ├── unittest_custom_options_pb2.py
+│   │       ├── unittest_import_pb2.py
+│   │       ├── unittest_import_public_pb2.py
+│   │       ├── unittest_mset_pb2.py
+│   │       ├── unittest_mset_wire_format_pb2.py
+│   │       ├── unittest_no_arena_import_pb2.py
+│   │       ├── unittest_no_arena_pb2.py
+│   │       ├── unittest_no_generic_services_pb2.py
+│   │       ├── unittest_pb2.py
+│   │       ├── unittest_proto3_arena_pb2.py
+│   │       ├── util
+│   │       │   ├── __init__.py
+│   │       │   └── json_format_proto3_pb2.py
+│   │       └── wrappers_pb2.py
+│   └── protobuf-3.5.0.post1-py2.7-nspkg.pth
 ├── s38.egg-link
 ├── setuptools
 │   ├── __init__.py
@@ -887,6 +786,8 @@
 │   ├── extern
 │   │   ├── __init__.py
 │   │   └── __init__.pyc
+│   ├── glibc.py
+│   ├── glibc.pyc
 │   ├── glob.py
 │   ├── glob.pyc
 │   ├── gui-32.exe
@@ -904,6 +805,8 @@
 │   ├── namespaces.pyc
 │   ├── package_index.py
 │   ├── package_index.pyc
+│   ├── pep425tags.py
+│   ├── pep425tags.pyc
 │   ├── py27compat.py
 │   ├── py27compat.pyc
 │   ├── py31compat.py
@@ -924,9 +827,11 @@
 │   ├── unicode_utils.pyc
 │   ├── version.py
 │   ├── version.pyc
+│   ├── wheel.py
+│   ├── wheel.pyc
 │   ├── windows_support.py
 │   └── windows_support.pyc
-├── setuptools-37.0.0.dist-info
+├── setuptools-38.2.1.dist-info
 │   ├── DESCRIPTION.rst
 │   ├── INSTALLER
 │   ├── LICENSE.txt
@@ -941,13 +846,13 @@
 ├── setuptools.pth
 ├── six-1.11.0-py2.7.egg
 │   ├── EGG-INFO
+│   │   ├── DESCRIPTION.rst
 │   │   ├── PKG-INFO
-│   │   ├── SOURCES.txt
-│   │   ├── dependency_links.txt
-│   │   ├── not-zip-safe
+│   │   ├── RECORD
+│   │   ├── WHEEL
+│   │   ├── metadata.json
 │   │   └── top_level.txt
-│   ├── six.py
-│   └── six.pyc
+│   └── six.py
 ├── wheel
 │   ├── __init__.py
 │   ├── __init__.pyc
@@ -999,24 +904,20 @@
 │   └── top_level.txt
 └── zope.deprecation-4.3.0-py2.7.egg
     ├── EGG-INFO
+    │   ├── DESCRIPTION.rst
     │   ├── PKG-INFO
-    │   ├── SOURCES.txt
-    │   ├── dependency_links.txt
+    │   ├── RECORD
+    │   ├── WHEEL
+    │   ├── metadata.json
     │   ├── namespace_packages.txt
-    │   ├── not-zip-safe
     │   ├── requires.txt
     │   └── top_level.txt
-    └── zope
-        ├── __init__.py
-        ├── __init__.pyc
-        └── deprecation
-            ├── __init__.py
-            ├── __init__.pyc
-            ├── deprecation.py
-            ├── deprecation.pyc
-            ├── fixture.py
-            ├── fixture.pyc
-            ├── tests.py
-            └── tests.pyc
+    ├── zope
+    │   └── deprecation
+    │       ├── __init__.py
+    │       ├── deprecation.py
+    │       ├── fixture.py
+    │       └── tests.py
+    └── zope.deprecation-4.3.0-py3.6-nspkg.pth

-69 directories, 950 files
+69 directories, 851 files

[benoit-pierre]

Proposed fix here: #1215

[reinout]

It doesn't work yet for me. I'm trying to get buildout to work with the new wheel support in buildout/buildout#428, but the tests don't run because the namespace package zope.testrunner isn't importable

https://pypi.python.org/packages/df/a6/899d21e90e2fccec31481ed5c137c4f967c59ad6f827546d7f60b74a8455/zope.testrunner-4.7.0-py2.py3-none-any.whl

In the fix, on line da1c78f#diff-17b52a268b98d01dd98fa746689f9383R135, you look for namespace-packages.txt in the egg-info directory. But zope.testrunner's wheel has that file in its dist-info directory:

I don't know if the wheel is incorrect or if dist-info should be checked "in addition to" or "instead of" egg-info.

If I add an __init__.py in the zope/ directory of the unpacked wheel, zope.testrunner is importable.

[leorochael]

I suggest we reopen this issue until @reinout's comment is addressed. I've made a concrete suggestion for a fix here.

[leorochael]

I suggest we reopen this issue until @reinout's comment is addressed. I've made a concrete suggestion for a fix here.

@reinout, Can you check if the fix I proposed above solves the namespace package issue on zope.testrunner?

[benoit-pierre]

@reinout: I can't reproduce the issue, can you give more detailed steps? Here is what I used:

> python -m venv venv
> ./venv/bin/pip install -q -U setuptools
> ./venv/bin/easy_install -q zope.testrunner
> ./venv/bin/pip freeze --all
pip==9.0.1
setuptools==38.2.3
six==1.11.0
zope.exceptions==4.2.0
zope.interface==4.4.3
zope.testrunner==4.8.1
>  ./venv/bin/python -c 'import zope'
> ls venv/lib/python3.6/site-packages/zope.testrunner-4.8.1-py3.6.egg/zope/__init__.py
venv/lib/python3.6/site-packages/zope.testrunner-4.8.1-py3.6.egg/zope/__init__.py

[reinout]

The link of @leorochael is faulty, this is the right one: #1215 (comment)

@benoit-pierre I "tested" it through buildout's tests. I'll see if I can make a small example (though yours looks good).

[leorochael]

@reinout: I can't reproduce the issue, can you give more detailed steps?

@benoit-pierre, here is a reproduction:

$ virtualenv setuptools_1214 --python=python2
$ cd setuptools_1214
$ bin/pip install --upgrade pip setuptools
$ bin/pip freeze --all
pip==9.0.1
setuptools==38.2.3

Here we make use of an explicit (though not often used) use-case of setuptools, a plugin directory that is populated with eggs that are dynamically installed by the aplication:

$ mkdir plugins scripts
$ bin/easy_install -qm -s scripts -d plugins zope.testrunner
$ ls plugins/
six-1.11.0-py2.7.egg
zope.exceptions-4.2.0-py2.7.egg
zope.interface-4.4.3-py2.7-linux-x86_64.egg
zope.testrunner-4.8.1-py2.7.egg

And here we make use of the plugins directory:

$ bin/python
Python 2.7.6 (default, Nov 23 2017, 15:49:48) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, glob
>>> sys.path[:0] = glob.glob('plugins/*.egg')
>>> import zope.testrunner
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named zope.testrunner
>>> import zope.interface
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named zope.interface
>>> import zope.exceptions
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named zope.exceptions
>>> import six
>>> print six.__file__
plugins/six-1.11.0-py2.7.egg/six.pyc

All of the packages above should have been imported. Not only six.

[reinout]

That's a different problem from mine. In my case, zope.interface and zope.exceptions can be imported just fine (being build from a *.tar.gz).

zope.testrunner is a .whl, not a .egg. The problem seems to be that buildout is asking setuptools to extract it like an egg instead of converting the wheel to an egg.

[benoit-pierre]

I can't reproduce:

> python2 -m virtualenv venv
New python executable in venv/bin/python2
Also creating executable in venv/bin/python
Installing setuptools, pip, wheel...done.
> ./venv/bin/pip install -q -U setuptools
> ./venv/bin/pip freeze --all
pip==9.0.1
setuptools==38.2.3
wheel==0.30.0
> mkdir plugins
> PYTHONPATH=plugins ./venv/bin/easy_install -qm -s scripts -d plugins zope.testrunner
> ls plugins
six-1.11.0-py2.7.egg/             zope.interface-4.4.3-py2.7-linux-x86_64.egg/
zope.exceptions-4.2.0-py2.7.egg/  zope.testrunner-4.8.1-py2.7.egg/
> ./venv/bin/python
Python 2.7.14 (default, Sep 20 2017, 01:25:59)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import glob, sys
>>> sys.path[:0] = glob.glob('plugins/*.egg')
>>> import zope.testrunner
>>> zope.__file__
'[...]/plugins/zope.interface-4.4.3-py2.7-linux-x86_64.egg/zope/__init__.py'
>>>

[reinout]

I think we can close this again. Setuptools handles it just fine.
I was just calling the wrong function. buildout/buildout@4904f96 is what I should have done.

[leorochael]

It's still closed ;-)

And I can't reproduce it either. don't know what I did wrong the first time on my reproduction. Sorry for the noise.