ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels #4884

pitrou · 2019-07-15T17:40:28Z

No description provided.

pitrou · 2019-07-15T17:40:37Z

@ursabot crossbow package -g wheel

ursabot · 2019-07-15T17:44:42Z

AMD64 Conda Crossbow (#37639) builder has been succeeded.

Revision: 47bce41deee0a70d816572b4c820fcf07f546f77

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-122

Task	Status
wheel-manylinux1-cp27m
wheel-manylinux1-cp27mu
wheel-manylinux1-cp35m
wheel-manylinux1-cp36m
wheel-manylinux1-cp37m
wheel-manylinux2010-cp27m
wheel-manylinux2010-cp27mu
wheel-manylinux2010-cp35m
wheel-manylinux2010-cp36m
wheel-manylinux2010-cp37m
wheel-osx-cp27m
wheel-osx-cp35m
wheel-osx-cp36m
wheel-osx-cp37m
wheel-win-cp35m
wheel-win-cp36m
wheel-win-cp37m

kszucs · 2019-07-15T18:21:51Z

@ursabot crossbow package -g wheel

kszucs · 2019-07-15T18:22:45Z

Turned out that using non-slim images also works.

ursabot · 2019-07-15T18:26:03Z

AMD64 Conda Crossbow (#37658) builder has been succeeded.

Revision: 2a09a13076a9bd577488d7ff94d3013a3dcc96d1

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-123

Task	Status
wheel-manylinux1-cp27m
wheel-manylinux1-cp27mu
wheel-manylinux1-cp35m
wheel-manylinux1-cp36m
wheel-manylinux1-cp37m
wheel-manylinux2010-cp27m
wheel-manylinux2010-cp27mu
wheel-manylinux2010-cp35m
wheel-manylinux2010-cp36m
wheel-manylinux2010-cp37m
wheel-osx-cp27m
wheel-osx-cp35m
wheel-osx-cp36m
wheel-osx-cp37m
wheel-win-cp35m
wheel-win-cp36m
wheel-win-cp37m

kszucs · 2019-07-15T18:56:13Z

@pitrou "ValueError: pyarrow.lib.DictionaryMemo size changed, may indicate binary incompatibility. Expected 184 from C header, got 160 from PyObject"

pitrou · 2019-07-15T19:37:47Z

@kszucs Yes, it's cython...

kszucs · 2019-07-15T19:39:50Z

@pitrou Just one of the builds is failing with that, in the rest of the builds the linker cannot find libarrow, the library paths and the library names look correct though. It fails for me locally too.

pitrou · 2019-07-15T19:43:53Z

@kszucs I'll ask once again: is there an easy way to reproduce locally?

kszucs · 2019-07-15T19:54:58Z

@pitrou I wouldn't say it easy, but:

$ docker-compose run -e PYTHON_VERSION=3.7 python-manylinux1 bash
# /io/build_arrow.sh

Then you'll have the .whl under /io/dist (arrow/python/manylinux1 is mounted under /io)

After that (pwd should be your local arrow path)

$ docker run -v `pwd`:/arrow -it python:3.7 bash
# export WHEEL_TAG=manylinux1
# /arrow/dev/tasks/python-wheels/manylinux-test.sh

pitrou · 2019-07-15T19:58:59Z

Yeah... I'll take a look tomorrow.

kszucs · 2019-07-15T19:59:21Z

Any hints I should go on (ld doesn't seem to be too verbose)?

pitrou · 2019-07-15T20:00:36Z

@kszucs Perhaps try LD_LIBRARY_PATH on non-Windows. Also perhaps silence some warnings for the Cython issue...

kszucs · 2019-07-15T20:01:21Z

Ok, thanks!

kszucs · 2019-07-15T20:49:30Z

We could also bundle the shared libs without the version suffix...

pitrou · 2019-07-15T20:50:24Z

I think @xhochy wanted the version-suffixed ones? Though I really wonder why we should cater to absolutely every use case.

wesm · 2019-07-15T20:54:24Z

I think given all the problems we've had with wheels, this is asking a lot. I think we should do the easiest thing that works

kszucs · 2019-07-15T21:25:27Z

@ursabot crossbow package wheel-osx-cp27m wheel-osx-cp35m wheel-osx-cp36m wheel-osx-cp37m

ursabot · 2019-07-15T21:29:20Z

AMD64 Conda Crossbow (#37695) builder has been succeeded.

Revision: feb0d7cef69d661d351878cd93ed6cfe4044596a

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-124

Task	Status
wheel-osx-cp27m
wheel-osx-cp35m
wheel-osx-cp36m
wheel-osx-cp37m

kszucs · 2019-07-15T22:50:17Z

@ursabot crossbow package -g wheel

ursabot · 2019-07-15T22:54:12Z

AMD64 Conda Crossbow (#37713) builder has been succeeded.

Revision: 496b54ab236946e5c1955f55a81ba3cfad6e5d6d

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-125

Task	Status
wheel-manylinux1-cp27m
wheel-manylinux1-cp27mu
wheel-manylinux1-cp35m
wheel-manylinux1-cp36m
wheel-manylinux1-cp37m
wheel-manylinux2010-cp27m
wheel-manylinux2010-cp27mu
wheel-manylinux2010-cp35m
wheel-manylinux2010-cp36m
wheel-manylinux2010-cp37m
wheel-osx-cp27m
wheel-osx-cp35m
wheel-osx-cp36m
wheel-osx-cp37m
wheel-win-cp35m
wheel-win-cp36m
wheel-win-cp37m

kszucs · 2019-07-16T00:58:49Z

@pitrou seems like the cython issue is manylinux1 related

mbrobbel · 2019-07-16T07:47:50Z

@pitrou "ValueError: pyarrow.lib.DictionaryMemo size changed, may indicate binary incompatibility. Expected 184 from C header, got 160 from PyObject"

@pitrou seems like the cython issue is manylinux1 related

@kszucs We've also been getting this error for Fletcher manylinux2010 wheels since 0.14. I can provide a Dockerfile to reproduce the error for our project.

xhochy · 2019-07-16T08:25:27Z

I think @xhochy wanted the version-suffixed ones? Though I really wonder why we should cater to absolutely every use case.

It would have been nice but this seems to be a considerable investment and there is nobody doing it, so we should just ship the non-suffixed ones.

xhochy · 2019-07-16T08:25:52Z

@pitrou "ValueError: pyarrow.lib.DictionaryMemo size changed, may indicate binary incompatibility. Expected 184 from C header, got 160 from PyObject"

@pitrou seems like the cython issue is manylinux1 related

@kszucs We've also been getting this error for Fletcher manylinux2010 wheels since 0.14. I can provide a Dockerfile to reproduce the error for our project.

@mbrobbel How stable is your current master?

mbrobbel · 2019-07-16T08:59:24Z

@pitrou "ValueError: pyarrow.lib.DictionaryMemo size changed, may indicate binary incompatibility. Expected 184 from C header, got 160 from PyObject"

@pitrou seems like the cython issue is manylinux1 related

@kszucs We've also been getting this error for Fletcher manylinux2010 wheels since 0.14. I can provide a Dockerfile to reproduce the error for our project.

@mbrobbel How stable is your current master?

Our master is outdated. We're currently working towards a 0.1 release in which will merge develop into master. This release is currently blocked on getting our python runtime manylinux wheels to work. We're considering to release with arrow 0.13.

pitrou · 2019-07-16T09:06:43Z

I think the Cython issue stems from compiling with one Cython version and c-importing with another. I've opened cython/cython#3042 to ask for guidance.

pitrou · 2019-07-16T10:36:03Z

@ursabot crossbow package -g wheel

ursabot · 2019-07-16T10:39:56Z

AMD64 Conda Crossbow (#38015) builder has been succeeded.

Revision: a411d7a

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-128

Task	Status
wheel-manylinux1-cp27m
wheel-manylinux1-cp27mu
wheel-manylinux1-cp35m
wheel-manylinux1-cp36m
wheel-manylinux1-cp37m
wheel-manylinux2010-cp27m
wheel-manylinux2010-cp27mu
wheel-manylinux2010-cp35m
wheel-manylinux2010-cp36m
wheel-manylinux2010-cp37m
wheel-osx-cp27m
wheel-osx-cp35m
wheel-osx-cp36m
wheel-osx-cp37m
wheel-win-cp35m
wheel-win-cp36m
wheel-win-cp37m

codecov-io · 2019-07-16T11:09:02Z

Codecov Report

Merging #4884 into master will decrease coverage by 22.28%.
The diff coverage is 64.28%.

@@             Coverage Diff             @@
##           master    #4884       +/-   ##
===========================================
- Coverage    87.4%   65.12%   -22.29%     
===========================================
  Files         996      486      -510     
  Lines      140313    64503    -75810     
  Branches     1418        0     -1418     
===========================================
- Hits       122644    42008    -80636     
- Misses      17307    22495     +5188     
+ Partials      362        0      -362

Impacted Files	Coverage Δ
python/pyarrow/ipc.pxi	`81.32% <0%> (ø)`	⬆️
python/pyarrow/types.pxi	`68.74% <0%> (-0.22%)`	⬇️
python/pyarrow/tests/conftest.py	`70.64% <60%> (-0.52%)`	⬇️
python/pyarrow/__init__.py	`64.13% <67.56%> (+23.58%)`	⬆️
python/pyarrow/tests/test_cython.py	`94.73% <88.88%> (-2.24%)`	⬇️
cpp/src/arrow/util/memory.h	`0% <0%> (-100%)`	⬇️
cpp/src/gandiva/date_utils.h	`0% <0%> (-100%)`	⬇️
cpp/src/arrow/util/memory.cc	`0% <0%> (-100%)`	⬇️
cpp/src/arrow/filesystem/util-internal.cc	`0% <0%> (-100%)`	⬇️
cpp/src/arrow/util/sse-util.h	`0% <0%> (-100%)`	⬇️
... and 753 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f976629...a411d7a. Read the comment docs.

pitrou · 2019-07-16T11:36:51Z

So, it looks like things finally work here.

xhochy

+1. LGTM

Would be good to have comments at the places where I commented here so we know in some weeks why we did it this way.

xhochy · 2019-07-16T11:37:46Z

dev/tasks/python-wheels/osx-build.sh

+
    # Run pyarrow tests
-    py.test --pyargs pyarrow
+    pytest -rs --pyargs pyarrow


Why is this without a . in the name?

One is just an alias for the other.

I think we generally use the "pytest" spelling. At least that's the one I use locally (it's also consistent with "python -m pytest").

pytest is the preferred command since a couple of releases now, the documentation also refers that https://docs.pytest.org/en/latest/usage.html

py.test was once the preferred way but browsing the documentation, it seems this is no longer the case.

xhochy · 2019-07-16T11:39:46Z

python/setup.py

    lib_filename = os.path.basename(libs[0])
    shutil.move(pjoin(build_prefix, lib_filename),
                pjoin(build_lib, 'pyarrow', lib_filename))
+    for lib in libs[1:]:


Can you add a comment on what the purpose of this is?

It's just reinstating code that was removed in ARROW-5082. The comment is just above ("Longest suffix library should be copied, all others symlinked").

Author: Antoine Pitrou <[email protected]> Closes #4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels

Author: Antoine Pitrou <[email protected]> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels

* Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <[email protected]> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <[email protected]> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]>

* [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <[email protected]> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <[email protected]> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <[email protected]> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <[email protected]> Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]>

* [FEDE-5150] Upgrade arrow version from 4.0.0 to 7.0.0 (#26) * [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <[email protected]> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <[email protected]> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <[email protected]> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <[email protected]> Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]> * Release siren-7.0.0-1 Signed-off-by: George Apaaboah <[email protected]> * Fix rebase * Update services configuration (#32) * Update services configuration This is needed for the module declaration in federate * Create a uber jar with netty and arrow memory dependencies This avoids conflicts with java modules. Arrows provides different modules (memory-netty, memory-core) that export the same packages (org.apache.arrow.memory), which is forbidden with java modules. The federate code can then just dependend on the shaded jar from vector. * use reduced pom in jar Co-authored-by: Issac <[email protected]> * Add developer tip link to readme * Remove test scope for arrow-memory-netty * Change version from siren-9.0.0-1-SNAPSHOT to siren-9.0.0-2-SNAPSHOT since 2 contain the module changes * Update as per review * Update comment as per review * Use stable netty version siren-4.1.78-1 Signed-off-by: George Apaaboah <[email protected]> Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]> Co-authored-by: Issac <[email protected]>

* [FEDE-5623] Upgrade Siren Arrow from 7.0.0 to 9.0.0 (#35) * [FEDE-5150] Upgrade arrow version from 4.0.0 to 7.0.0 (#26) * [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <[email protected]> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <[email protected]> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <[email protected]> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <[email protected]> Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]> * Release siren-7.0.0-1 Signed-off-by: George Apaaboah <[email protected]> * Fix rebase * Update services configuration (#32) * Update services configuration This is needed for the module declaration in federate * Create a uber jar with netty and arrow memory dependencies This avoids conflicts with java modules. Arrows provides different modules (memory-netty, memory-core) that export the same packages (org.apache.arrow.memory), which is forbidden with java modules. The federate code can then just dependend on the shaded jar from vector. * use reduced pom in jar Co-authored-by: Issac <[email protected]> * Add developer tip link to readme * Remove test scope for arrow-memory-netty * Change version from siren-9.0.0-1-SNAPSHOT to siren-9.0.0-2-SNAPSHOT since 2 contain the module changes * Update as per review * Update comment as per review * Use stable netty version siren-4.1.78-1 Signed-off-by: George Apaaboah <[email protected]> Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]> Co-authored-by: Issac <[email protected]> * Fix cherry-pick * Temporarily comment out the jdk version * chore: revert commented out jdk version in pom * update: use stable version of netty after release of siren-4.1.82-1 --------- Signed-off-by: George Apaaboah <[email protected]> Co-authored-by: Laurent Goujon <[email protected]> Co-authored-by: Stéphane Campinas <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Martin Anseaume <[email protected]> Co-authored-by: Johnny Hujol <[email protected]> Co-authored-by: ggdupont <[email protected]> Co-authored-by: Issac <[email protected]>

xhochy closed this Jul 16, 2019

xhochy reopened this Jul 16, 2019

pitrou mentioned this pull request Jul 16, 2019

"size changed, may indicate binary incompatibility" cython/cython#3042

Closed

pitrou changed the title ~~ARROW-5856: [Python] Try to fix use of C++ / Cython API from wheels~~ ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Jul 16, 2019

xhochy approved these changes Jul 16, 2019

View reviewed changes

pitrou closed this in 690823c Jul 16, 2019

pitrou deleted the ARROW-5856-cython-so-version branch July 16, 2019 12:23

asfimport mentioned this pull request Aug 1, 2019

[Python] linking 3rd party cython modules against pyarrow fails since 0.14.0 #16800

Closed

ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels #4884

ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels #4884

Uh oh!

Conversation

pitrou commented Jul 15, 2019

Uh oh!

pitrou commented Jul 15, 2019

Uh oh!

ursabot commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019

Uh oh!

ursabot commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019

Uh oh!

pitrou commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019

Uh oh!

pitrou commented Jul 15, 2019

Uh oh!

wesm commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019

Uh oh!

ursabot commented Jul 15, 2019

Uh oh!

kszucs commented Jul 15, 2019

Uh oh!

ursabot commented Jul 15, 2019

Uh oh!

kszucs commented Jul 16, 2019

Uh oh!

mbrobbel commented Jul 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xhochy commented Jul 16, 2019

Uh oh!

xhochy commented Jul 16, 2019

Uh oh!

mbrobbel commented Jul 16, 2019

Uh oh!

pitrou commented Jul 16, 2019

Uh oh!

pitrou commented Jul 16, 2019

Uh oh!

ursabot commented Jul 16, 2019

Uh oh!

codecov-io commented Jul 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pitrou commented Jul 16, 2019

Uh oh!

xhochy left a comment

Choose a reason for hiding this comment

Uh oh!

xhochy Jul 16, 2019

Choose a reason for hiding this comment

Uh oh!

kszucs commented Jul 15, 2019 •

edited

Loading

kszucs commented Jul 15, 2019 •

edited

Loading

kszucs commented Jul 15, 2019 •

edited

Loading

mbrobbel commented Jul 16, 2019 •

edited

Loading

codecov-io commented Jul 16, 2019 •

edited

Loading