Skip to content

ARROW-12619: [Python] pyarrow sdist should not require git #10342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kszucs wants to merge 3 commits into from
Closed

ARROW-12619: [Python] pyarrow sdist should not require git #10342

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4964100
Use the scm fallback mechanism to set pyarrow's version in the python…
kszucs May 17, 2021
dab0cdd
Remove git before testing sdist
kszucs May 17, 2021
c423b96
Fix bash condition
kszucs May 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion ci/scripts/python_sdist_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,16 @@ export PYARROW_WITH_DATASET=${ARROW_DATASET:-OFF}
# unset ARROW_HOME
# apt purge -y pkg-config

# ARROW-12619
if command -v git &> /dev/null; then
echo "Git exists, remove it from PATH before executing this script."
exit 1
fi

if [ -n "${PYARROW_VERSION:-}" ]; then
sdist="${arrow_dir}/python/dist/pyarrow-${PYARROW_VERSION}.tar.gz"
else
sdist=$(ls "${arrow_dir}/python/dist/pyarrow-*.tar.gz" | sort -r | head -n1)
sdist=$(ls ${arrow_dir}/python/dist/pyarrow-*.tar.gz | sort -r | head -n1)
Copy link
Member

@lidavidm lidavidm May 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very minor nit: you could keep the quoting around just ${arrow_dir}

Copy link
Member Author

@kszucs kszucs May 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly it didn't pick up the tarball with the quotes, which I wouldn't have expected either.

Copy link
Member Author

@kszucs kszucs May 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

Creating arrow_ubuntu-python-sdist-test_run ... done
root@202aa2bdd8d0:/# ls "/arrow/python/dist/pyarrow-*.tar.gz"
ls: cannot access '/arrow/python/dist/pyarrow-*.tar.gz': No such file or directory
root@202aa2bdd8d0:/# ls /arrow/python/dist/pyarrow-*.tar.gz
/arrow/python/dist/pyarrow-5.0.0.dev866+g49641009e.d20210517.tar.gz
root@202aa2bdd8d0:/#

Copy link
Member

@lidavidm lidavidm May 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is expected, but ls "${arrow_dir}"/python/dist/pyarrow-*.tar.gz doesn't work? e.g.

$ export arrow_dir=/tmp/foo
$ touch /tmp/foo/pyarrow-5.0.0.tar.gz
$ ls "${arrow_dir}"/pyarrow-*.tar.gz
/tmp/foo/pyarrow-5.0.0.tar.gz

but either way this is a very minor nit, just if you wanted to be extra-defensive.

Copy link
Member Author

@kszucs kszucs May 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$arrow_dir is going to be /arrow :)

fi
${PYTHON:-python} -m pip install ${sdist}

Expand Down
7 changes: 4 additions & 3 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -713,6 +713,7 @@ services:
volumes: *ubuntu-volumes
command: >
/bin/bash -c "
apt remove -y git &&
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"

Expand Down Expand Up @@ -814,9 +815,9 @@ services:
source: .
target: "C:/arrow"
command: arrow\\ci\\scripts\\python_wheel_windows_test.bat

java-bundled-jars:
# Docker image
# Docker image
image: ${REPO}:${ARCH}-java-bundled-jars-vcpkg-${VCPKG}
build:
args:
Expand All @@ -831,7 +832,7 @@ services:
volumes:
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
command:
command:
[/arrow/ci/scripts/java_bundled_jars_manylinux_build.sh /arrow /build /arrow/dist]

############################## Integration #################################
Expand Down
8 changes: 2 additions & 6 deletions python/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -525,12 +525,8 @@ def _move_shared_libs_unix(build_prefix, build_lib, lib_name):
default_version = '5.0.0-SNAPSHOT'
if (not os.path.exists('../.git') and
not os.environ.get('SETUPTOOLS_SCM_PRETEND_VERSION')):
if os.path.exists('PKG-INFO'):
# We're probably in a Python sdist, setuptools_scm will handle fine
pass
else:
os.environ['SETUPTOOLS_SCM_PRETEND_VERSION'] = \
default_version.replace('-SNAPSHOT', 'a0')
os.environ['SETUPTOOLS_SCM_PRETEND_VERSION'] = \
default_version.replace('-SNAPSHOT', 'a0')


# See https://github.com/pypa/setuptools_scm#configuration-parameters
Expand Down