ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels #9096

kszucs · 2021-01-04T18:25:27Z

Also resolves:

ARROW-11213: [Packaging][Python] Dockerize wheel building on windows
ARROW-11215: [CI] Use named volumes by default for caching in docker-compose
ARROW-11231: [Python][Packaging] Enable mimalloc in wheels

Main features:

dockerize windows builds
use vcpkg as the dependency source where we can explicitly pin a working version without worrying about the drag of build environment over time

Potential follow-up:

use vcpkg on macos as well

cc @kou @xhochy

Manylinux testing

Should be straightforward (I'm going to shared pre-built images for quicker testing).

Windows testing

Only windows host is able to run windows containers. I'm virtualizing windows on macOS (should work on linux as well) using virtualbox:

git clone https://github.com/StefanScherer/windows-docker-machine
cd windows-docker-machine
# grant more resources in the vagrantfile under the virtualbox section (I use 8 cores and 16GiB of ram)
vagrant up --provider virtualbox 2019-box
docker context ls
docker context use 2019-box
docker image ls

Now docker should use the windows docker daemon.

pip install -e dev/archery[docker]
# the volumes defined in the docker compose file should be commented out and pass arrow's source explicitly
# this is required because compose is being executed on a unix host and the directory is mounted through a virtual machine
PYTHON=3.6 archery docker run --no-pull --using-docker-cli -v C:$(pwd):C:/arrow python-wheel-windows-vs2017 cmd.exe
# then within the container execute a build or do other interactive things
arrow\ci\scripts\python_wheel_windown_build.bat

github-actions · 2021-01-04T18:34:43Z

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

kou · 2021-01-04T22:06:12Z

Related works:

ARROW-10224: [Python] Add support for Python 3.9 except macOS wheel and Windows wheel #8386 (comment)
ARROW-9400: [Python] Do not depend on conda-forge static libraries in Windows wheel builds #8881

kszucs · 2021-01-05T20:04:54Z

Currently integrating with crossbow using pre-built images, the manylinux build already works except the artifact uploading.
Pull & build took ~13m: https://github.com/ursa-labs/crossbow/runs/1652174767

Passing build under ~19m including testing: https://github.com/ursa-labs/crossbow/runs/1652276423

kszucs · 2021-01-05T20:40:58Z

@github-actions crossbow submit wheel-manylinux*

github-actions · 2021-01-05T21:08:03Z

Revision: aeb902f4f50e70fb6be844ecaf1283820bee3c64

Submitted crossbow builds: ursa-labs/crossbow @ actions-825

Task	Status
wheel-manylinux2010-cp36m
wheel-manylinux2010-cp37m
wheel-manylinux2010-cp38
wheel-manylinux2010-cp39
wheel-manylinux2014-cp36m
wheel-manylinux2014-cp37m
wheel-manylinux2014-cp38
wheel-manylinux2014-cp39

kszucs · 2021-01-06T21:20:37Z

@xhochy we have a running build for windows on github actions now. I'll do a couple of things to try to improve the build times a bit (make the image smaller because pulling takes ~15m and enable unity builds).

Since we don't bundle msvcp140.dll these wheels are not working in the official windows python docker containers, so I'm using the same image as for building. I may try to bundle it (guess it should be about copying it into the archive) to ensure that it works in the base images.

kszucs · 2021-01-07T20:13:05Z

@github-actions crossbow submit wheel-manylinux2010-cp36m wheel-manylinux2014-cp36m wheel-windows-cp36m

github-actions · 2021-01-07T20:14:33Z

Revision: 233aabae02f16ef8edf55b5470cb7ab12c58c920

Submitted crossbow builds: ursa-labs/crossbow @ actions-838

Task	Status
wheel-manylinux2010-cp36m
wheel-manylinux2014-cp36m
wheel-windows-cp36m

kszucs · 2021-01-07T20:37:03Z

@kou @xhochy bundled the msvc runtime. Locally tested with the stock windows python images and seems to work fine. Hopefully crossbow will confirm it.

I'm going to write a more detailed description of the changes tomorrow. Any reviews are welcome!

kou · 2021-01-07T22:10:43Z

It seems that https://github.com/ursa-labs/crossbow/releases/tag/actions-838-github-wheel-windows-cp36m doesn't have artifacts...
Where did you download built wheels?

kszucs · 2021-01-07T23:30:14Z

@github-actions crossbow submit wheel-windows-cp36m

kszucs · 2021-01-07T23:31:33Z

The build also executes the tests in a different docker container, see the "Test Wheel" step in the build.

The following build should properly upload the windows wheel to crossbow releases now.

github-actions · 2021-01-07T23:31:53Z

Revision: 009d9d049ef48e3cd6e4e17126d6093b25331312

Submitted crossbow builds: ursa-labs/crossbow @ actions-842

Task	Status
wheel-windows-cp36m

kou

+1

ci/scripts/wheel_windows_build.bat

kszucs · 2021-01-08T12:46:43Z

@github-actions crossbow submit wheel-manylinux2010-cp36m wheel-manylinux2014-cp36m wheel-windows-cp36m

python/setup.py

dev/tasks/tasks.yml

github-actions · 2021-01-08T13:08:06Z

Revision: b47406064789942bacaea9d3b1d62d5778796d37

Submitted crossbow builds: ursa-labs/crossbow @ actions-844

Task	Status
wheel-manylinux2010-cp36m
wheel-manylinux2014-cp36m
wheel-windows-cp36m

kszucs · 2021-01-08T15:01:28Z

@github-actions crossbow submit wheel-*

github-actions · 2021-01-08T15:36:51Z

Revision: 8081526709fc1642d0de3c305e4bb7cfa8af6912

Submitted crossbow builds: ursa-labs/crossbow @ actions-845

Task	Status
wheel-manylinux2010-cp36m
wheel-manylinux2010-cp37m
wheel-manylinux2010-cp38
wheel-manylinux2010-cp39
wheel-manylinux2014-cp36m
wheel-manylinux2014-cp37m
wheel-manylinux2014-cp38
wheel-manylinux2014-cp39
wheel-osx-high-sierra-cp36m
wheel-osx-high-sierra-cp37m
wheel-osx-high-sierra-cp38
wheel-osx-high-sierra-cp39
wheel-osx-mavericks-cp36m
wheel-osx-mavericks-cp37m
wheel-osx-mavericks-cp38
wheel-osx-mavericks-cp39
wheel-windows-cp36m
wheel-windows-cp37m
wheel-windows-cp38m
wheel-windows-cp39m

kszucs · 2021-01-08T18:15:28Z

The osx builds shouldn't be failing, but that's out of the scope of this PR.

xhochy · 2021-01-08T20:43:53Z

The OSX issue is that protobuf is installed through brew and that doesn't install it using cmake. You could conditionally add the following line

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Line 2459 in a576cd7

-DgRPC_PROTOBUF_PACKAGE_TYPE=CONFIG

only when protobuf is also build using bundled.

docker-compose.yml

…he cache plugin functional

kszucs · 2021-01-14T18:00:01Z

I submitted the crossbow tasks using the CLI, the in-progress results are promising:
https://github.com/ursacomputing/crossbow/branches/all?query=build-17

kszucs · 2021-01-14T19:47:30Z

@kou this should be ready to merge. One of the arm linux builds is failing with a timeout error in addition to the arm conda drone builds, but these shouldn't block this PR. The latest crossbow submission is here https://github.com/ursacomputing/crossbow/branches/all?query=build-18

kou · 2021-01-14T23:46:41Z

Thanks!

I'll merge this. We need to work on some follow-up tasks:

Fix .deb/.rpm builds on Travis CI
Fix Python version (3.8) in dev/tasks/conda-recipes/.ci_support/osx_pytho3.7.____cpython.yaml

@kszucs

@kszucs could you review this please? My main purpose in adding this is to improve the experience for Arrow C++ devs using Windows, but I noticed it also relates to your [TODO in #9096](https://github.com/apache/arrow/pull/9096/files#diff-990134cce6657dbbcf95457cf1a56810a7efa1f6cd58ecc27557c7d6ff45b533R67-R68). vcpkg does not have any `requirements.txt`-style package enumeration mechanism, but it supports this JSON manifest as a mechanism of defining dependencies. In the `vcpkg install` command, you can specify the path to the directory containing this manifest file with `--x-manifest-root` which later will change to `--manifest-root`. See details at https://vcpkg.readthedocs.io/en/stable/specifications/manifests/. There are some differences between the packages listed in this manifest versus the packages you listed in the `vcpkg install` commands in #9096 - This installs `gtest` and `benchmark` - This installs `boost` instead of separate `boost-filesystem`, `boost-regex`, etc. - This does not explicitly include the `core` feature of `aws-sdk-cpp` because explicitly including it causes an error, and it gets installed anyway Closes #9287 from ianmcook/ARROW-11340 Lead-authored-by: Ian Cook <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Krisztián Szűcs <[email protected]>

github-actions bot added the Component: Python label Jan 4, 2021

kszucs force-pushed the wheel-vcpkg branch from 0e76a61 to f30e042 Compare January 5, 2021 19:28

kszucs mentioned this pull request Jan 5, 2021

ARROW-11132: [CI] Use pip to install crossbow's dependencies for the comment bot #9103

Closed

xhochy mentioned this pull request Jan 6, 2021

ARROW-9400: [Python] Do not depend on conda-forge static libraries in Windows wheel builds #8881

Closed

kou approved these changes Jan 8, 2021

View reviewed changes

ci/scripts/wheel_windows_build.bat Outdated Show resolved Hide resolved

kszucs force-pushed the wheel-vcpkg branch from 009d9d0 to b474060 Compare January 8, 2021 12:46

kszucs commented Jan 8, 2021

View reviewed changes

python/setup.py Outdated Show resolved Hide resolved

kszucs commented Jan 8, 2021

View reviewed changes

dev/tasks/tasks.yml Outdated Show resolved Hide resolved

kou reviewed Jan 8, 2021

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

kou mentioned this pull request Jan 9, 2021

ARROW-10905: [Python] Add support for Python 3.9 Windows wheels #8916

Closed

kszucs changed the title ~~[Python][Packaging] Refactor manylinux and windows wheel building [WIP]~~ [Python][Packaging] Refactor manylinux and windows wheel building Jan 11, 2021

kszucs and others added 16 commits January 14, 2021 16:04

Export DOCKER_VOLUME_PREFIX in the github actions workflows to keep t…

62181df

…he cache plugin functional

Fix typo in build script

a46653f

Switch grpc protobuf package type condition

f85a45d

Explicitly set protobuf package type in the osx wheel builds

494b91c

gemfury upload

7d9907e

Trigger CI builds

ee43d22

Remove outdated cron jobs

0ccb69b

Unskip s3 tests

b6a0358

Enable mimalloc on linux and macos

2f0900b

Enable unity on manylinux

bf0c47c

Update vcpkg pin to include 1.1.1i

82c88ef

Add Snappy patch for ARROW-7939

2244d83

Patch snappy on windows as well

7b5033d

Fix curl ssl issue

e7cf120

Implicitly handle grpc protobuf package type

9609ef6

Add vcpkg patch to rat exclusion list

b85ac17

kszucs force-pushed the wheel-vcpkg branch from b2f0a85 to b85ac17 Compare January 14, 2021 15:04

Fix artifact patterns

07c7aa8

kou added 2 commits January 15, 2021 07:19

Remove an unused file

19b91a1

Remove unused volume

8e4cbf2

kou closed this in ddd9c6d Jan 14, 2021

ianmcook mentioned this pull request Jan 22, 2021

ARROW-11340: [C++] Add vcpkg.json manifest to cpp project root #9287

Closed

asfimport mentioned this pull request Aug 5, 2021

[Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels #27117

Closed

ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels #9096

ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels #9096

Uh oh!

Conversation

kszucs commented Jan 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Manylinux testing

Windows testing

Uh oh!

github-actions bot commented Jan 4, 2021

Uh oh!

kou commented Jan 4, 2021

Uh oh!

kszucs commented Jan 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Jan 5, 2021

Uh oh!

github-actions bot commented Jan 5, 2021

Uh oh!

kszucs commented Jan 6, 2021

Uh oh!

kszucs commented Jan 7, 2021

Uh oh!

github-actions bot commented Jan 7, 2021

Uh oh!

kszucs commented Jan 7, 2021

Uh oh!

kou commented Jan 7, 2021

Uh oh!

kszucs commented Jan 7, 2021

Uh oh!

kszucs commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 7, 2021

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kszucs commented Jan 8, 2021

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 8, 2021

Uh oh!

kszucs commented Jan 8, 2021

Uh oh!

github-actions bot commented Jan 8, 2021

Uh oh!

kszucs commented Jan 8, 2021

Uh oh!

xhochy commented Jan 8, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kszucs commented Jan 14, 2021

Uh oh!

kszucs commented Jan 14, 2021

Uh oh!

kou commented Jan 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kszucs commented Jan 4, 2021 •

edited

Loading

kszucs commented Jan 5, 2021 •

edited

Loading

kszucs commented Jan 7, 2021 •

edited

Loading