Skip to content

Commit 99c840c

Browse files
franchuteriveramfeurer
franchuterivera
and
mfeurer
authored
Wheels support (#56)
* First try of auto-wheels * syntax fixes * syntax fixes -- try2 * dependencies * fix path * permissions * build_tools/build_source.sh * Debug info * try3 * moredebuginfo * moredebug2 * moredebug3 * moredebug4 * typo * try space * Files locally * yumaware * numpyfix * testpath * project * Cleanup for PR * Feedback from comments * Missing pytest update * Wheels centos (#1) * Disentangle wheels/dist * Update build_tools/build_wheels.sh Co-authored-by: Matthias Feurer <[email protected]> * Fixed build source Co-authored-by: Matthias Feurer <[email protected]>
1 parent 316c4d9 commit 99c840c

File tree

9 files changed

+263
-13
lines changed

9 files changed

+263
-13
lines changed

‎.github/workflows/pytest.yml‎

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -19,18 +19,8 @@ jobs:
1919
python-version: ${{ matrix.python-version }}
2020
- name: Install test dependencies
2121
run: |
22-
sudo apt-get install -y build-essential
23-
python -m pip install --upgrade pip
24-
pip install "numpy<=1.19"
25-
sudo apt-get -qq update
26-
sudo apt-get install -y libboost-all-dev
27-
sudo apt-get remove swig
28-
sudo apt-get -y install swig3.0
29-
sudo ln -s /usr/bin/swig3.0 /usr/bin/swig
30-
sudo gem install coveralls-lcov
31-
sudo apt-get install -y lcov
32-
sudo apt-get install doxygen
33-
pip3 install --user -U pip-tools
22+
chmod u+x ./build_tools/env.sh
23+
./build_tools/env.sh
3424
lcov --directory . --zerocounters
3525
- name: Setup cmake
3626
uses: jwlawson/[email protected]

‎.github/workflows/wheels.yml‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# Workflow to build and test wheels
2+
name: Wheel builder
3+
4+
on:
5+
push:
6+
branches:
7+
- master
8+
# Release branches
9+
- "[0-9]+.[0-9]+.X"
10+
11+
jobs:
12+
# Build the wheels for Linux
13+
build_wheels:
14+
name: Build wheel for cp${{ matrix.python }}-${{ matrix.platform_id }}-${{ matrix.manylinux_image }}
15+
runs-on: ${{ matrix.os }}
16+
17+
strategy:
18+
# Ensure that a wheel builder finishes even if another fails
19+
fail-fast: false
20+
matrix:
21+
os: [ubuntu-latest]
22+
python: [36, 37, 38, 39]
23+
bitness: [32, 64]
24+
manylinux_image: [manylinux2014]
25+
include:
26+
- os: ubuntu-latest
27+
bitness: 64
28+
platform_id: manylinux_x86_64
29+
- os: ubuntu-latest
30+
bitness: 32
31+
platform_id: manylinux_i686
32+
33+
steps:
34+
- name: Checkout pyrfr
35+
uses: actions/checkout@v1
36+
37+
- name: Setup Python
38+
uses: actions/setup-python@v2
39+
40+
- name: Build and test wheels
41+
env:
42+
CIBW_BUILD: cp${{ matrix.python }}-${{ matrix.platform_id }}
43+
CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.manylinux_image }}
44+
CIBW_MANYLINUX_I686_IMAGE: ${{ matrix.manylinux_image }}
45+
CIBW_TEST_REQUIRES: pytest threadpoolctl numpy<=1.19
46+
CIBW_BEFORE_ALL: "{project}/build_tools/build_wheels.sh"
47+
CIBW_TEST_COMMAND: bash {project}/build_tools/test_wheels.sh
48+
49+
run: |
50+
python -m pip install cibuildwheel
51+
python -m cibuildwheel --output-dir wheelhouse
52+
53+
- name: Store artifacts
54+
uses: actions/upload-artifact@v2
55+
with:
56+
path: wheelhouse/*.whl
57+
58+
# Build the source distribution under Linux
59+
build_sdist:
60+
name: Source distribution
61+
runs-on: ubuntu-latest
62+
63+
steps:
64+
- name: Checkout pyrfr
65+
uses: actions/checkout@v1
66+
67+
- name: Setup Python
68+
uses: actions/setup-python@v2
69+
70+
- name: Build source distribution
71+
run: bash build_tools/build_source.sh
72+
73+
- name: Test source distribution
74+
run: bash build_tools/test_source.sh
75+
76+
- name: Store artifacts
77+
uses: actions/upload-artifact@v2
78+
with:
79+
path: dist/*.tar.gz
80+
81+
# Upload the wheels and the source distribution
82+
release_assets:
83+
name: Upload Release
84+
runs-on: ubuntu-latest
85+
needs: [build_wheels, build_sdist]
86+
# The artifacts cannot be uploaded on PRs
87+
if: github.event_name != 'pull_request'
88+
89+
steps:
90+
- name: Checkout pyrfr
91+
uses: actions/checkout@v1
92+
93+
- name: Download artifacts
94+
uses: actions/download-artifact@v2
95+
with:
96+
path: dist
97+
98+
- name: Setup Python
99+
uses: actions/setup-python@v2
100+
101+
- name: Install dependencies
102+
run: |
103+
python -m pip install --upgrade pip
104+
pip install setuptools wheel twine
105+
106+
#- name: Publish python package
107+
# id: publish_package
108+
# env:
109+
# TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
110+
# TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
111+
# run: |
112+
# twine upload dist/*
113+
114+
- name: Create Release
115+
id: create_release
116+
uses: actions/create-release@v1
117+
env:
118+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
119+
with:
120+
tag_name: ${{ github.ref }}
121+
release_name: Release ${{ github.ref }}
122+
draft: false
123+
prerelease: false
124+
- name: Upload Release Asset
125+
id: upload-release-asset
126+
env:
127+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
128+
run: |
129+
python -m pip install --upgrade pip
130+
set -x
131+
assets=()
132+
for asset in ./dist/*/*; do
133+
assets+=("-a" "$asset")
134+
done
135+
tag_name="${GITHUB_REF##*/}"
136+
hub release create "${assets[@]}" -m "$tag_name" "$tag_name"

‎build_tools/build_source.sh‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
3+
set -e
4+
set -x
5+
6+
# Build the package
7+
# Install the environment
8+
chmod u+x ./build_tools/env.sh
9+
./build_tools/env.sh
10+
11+
# Build the package
12+
mkdir build
13+
cd build
14+
cmake .. && make pyrfr_docstrings
15+
cd python_package
16+
17+
# Also build the distribution
18+
python -m pip install twine
19+
python setup.py sdist -d ../../dist
20+
21+
# Check whether the source distribution will render correctly
22+
twine check ../../dist/*.tar.gz

‎build_tools/build_wheels.sh‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# The wheels are built on a centos docker image
2+
# To comply with many-linux support, we employ
3+
# cibuildwheel which build the wheels in a
4+
# Docker image quay.io with minimal support. Following files are required
5+
# for the pyrfr to be compiled on the desired target
6+
pip3 install cmake numpy==1.11.0 scipy==0.17.0
7+
echo 'echo "pyuic5 $@"' > /usr/local/bin/pyuic5
8+
chmod +x /usr/local/bin/pyuic5
9+
yum install -y curl gsl-devel pcre-devel
10+
curl -LO http://prdownloads.sourceforge.net/swig/swig-3.0.12.tar.gz
11+
tar xzvf swig-3.0.12.tar.gz
12+
cd swig-3.0.12
13+
./configure
14+
make
15+
make install
16+
cd ..
17+
rm -rf swig-3.0.12*
18+
swig -version
19+
20+
# Install the package building dependencies -- one line at
21+
# a time for easy debug -- yum errors out with not much info
22+
# if one package installation failed
23+
yum -y install boost
24+
yum -y install boost-thread
25+
yum -y install boost-devel
26+
yum -y install doxygen
27+
yum -y install openssl-devel
28+
yum -y install cmake
29+
yum -y install tree
30+
yum -y install rsync
31+
cmake --version
32+
33+
# After installing the dependencies build the python package
34+
mkdir build
35+
cd build
36+
cmake .. && make pyrfr_docstrings
37+
# Copy the files for testing
38+
cp -r ../test_data_sets python_package
39+
# Copy from /project/build to /project
40+
rsync -a --delete --exclude '*build*' python_package/ ../
41+
42+
# Wheel building process will create a package from
43+
# the contents of /project. For debug purposes, show
44+
# the contents of this directory
45+
tree /project

‎build_tools/env.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/bin/bash
2+
3+
set -e
4+
set -x
5+
6+
# Install the environment requirements
7+
sudo apt-get install -y build-essential
8+
python -m pip install --upgrade pip
9+
pip install "numpy<=1.19"
10+
sudo apt-get -qq update
11+
sudo apt-get install -y libboost-all-dev
12+
sudo apt-get remove swig
13+
sudo apt-get -y install swig3.0
14+
sudo ln -s /usr/bin/swig3.0 /usr/bin/swig
15+
sudo gem install coveralls-lcov
16+
sudo apt-get install -y lcov
17+
sudo apt-get install doxygen
18+
pip3 install --user -U pip-tools

‎build_tools/test_source.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/bin/bash
2+
3+
set -e
4+
set -x
5+
6+
# Test the distribution on an isolated venv
7+
# Directory structure is such that
8+
# PWD=random_forest_run/random_forest_run/<repo>
9+
cd ../../
10+
11+
python -m venv test_env
12+
source test_env/bin/activate
13+
14+
python -m pip install pytest "numpy<=1.19"
15+
python -m pip install random_forest_run/random_forest_run/dist/*.tar.gz
16+
17+
sed -i -- "s/[^']\+test_data_sets/random_forest_run\/random_forest_run\/test_data_sets/" random_forest_run/random_forest_run/tests/*
18+
pytest random_forest_run/random_forest_run/tests/*.py

‎build_tools/test_wheels.sh‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/bin/bash
2+
3+
set -e
4+
set -x
5+
6+
# The wheel builder will currently have the REPO cloned into
7+
# /project. Also unittest from pyrfr are compiled and have local
8+
# paths embedded. Modify such path so they are now pointing to /project
9+
sed -i -- "s/[^']\+test_data_sets/\/project\/test_data_sets/" /project/tests/*
10+
pytest -v /project/tests/*py
11+
12+
# Test that there are no links to system libraries
13+
python -m threadpoolctl -i pyrfr

‎pyproject.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[build-system]
2+
# Minimum requirements for the build system to execute.
3+
requires = [
4+
"setuptools",
5+
"wheel",
6+
]

‎pyrfr/MANIFEST.in‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
11
recursive-include include *
22
recursive-include pyrfr *.i
3-
include LICENSE_cereal-1.2.2
3+
include LICENSE_cereal-1.2.2
4+
include LICENSE.txt
5+
include pyproject.toml

0 commit comments

Comments
 (0)