Skip to content

ARROW-11132: [CI] Use pip to install crossbow's dependencies for the comment bot #9103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kszucs wants to merge 2 commits into from
Closed

ARROW-11132: [CI] Use pip to install crossbow's dependencies for the comment bot #9103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
71abf11
Use pip to install crossbow's dependencies
kszucs Jan 5, 2021
92b8611
Trigger on comment
kszucs Jan 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
200 changes: 98 additions & 102 deletions .github/workflows/comment_bot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,17 +34,12 @@ jobs:
uses: actions/checkout@v2
with:
path: arrow
# because libgit2 is a dependency of crossbow so prefer conda
Copy link
Member Author

@kszucs kszucs Jan 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pygit2 ships wheels now with libgit2 bundled

- name: Setup Conda
uses: s-weigand/setup-conda@v1
- name: Set up Python
uses: actions/setup-python@v2
with:
update-conda: true
python-version: 3.7
conda-channels: conda-forge
- name: Instal Archery and Crossbow dependencies
run: |
conda install -y --file arrow/ci/conda_env_crossbow.txt pygithub
pip install -e arrow/dev/archery[bot]
python-version: 3.8
- name: Install Archery and Crossbow dependencies
run: pip install -e arrow/dev/archery[bot]
- name: Handle Github comment event
env:
ARROW_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand All @@ -54,96 +49,97 @@ jobs:
--event-name ${{ github.event_name }} \
--event-payload ${{ github.event_path }}

autotune:
name: "Fix all the things"
if: startsWith(github.event.comment.body, '@github-actions autotune')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: r-lib/actions/pr-fetch@master
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: See what is different
run: |
set -ex
git remote add upstream https://github.com/apache/arrow
git fetch upstream
changed() {
git diff --name-only HEAD..upstream/master | grep -e "$1" >/dev/null 2>&1
}
if changed '^r/.*\.R$'; then
echo "R_DOCS=true" >> $GITHUB_ENV
fi
if changed 'cmake' || changed 'CMake'; then
echo "CMAKE_FORMAT=true" >> $GITHUB_ENV
fi
if changed '^cpp/src'; then
echo "CLANG_FORMAT_CPP=true" >> $GITHUB_ENV
fi
if changed '^r/src'; then
echo "CLANG_FORMAT_R=true" >> $GITHUB_ENV
fi
- name: Run cmake_format
if: env.CMAKE_FORMAT == 'true' || endsWith(github.event.comment.body, 'everything')
run: |
set -ex
export PATH=/home/runner/.local/bin:$PATH
python3 -m pip install --upgrade pip setuptools wheel
python3 -m pip install -r dev/archery/requirements-lint.txt
python3 run-cmake-format.py
- name: Run clang-format on cpp
if: env.CLANG_FORMAT_CPP == 'true' || endsWith(github.event.comment.body, 'everything')
run: |
. .env # To get the clang version we use
cpp/build-support/run_clang_format.py \
--clang_format_binary=clang-format-${CLANG_TOOLS} \
--exclude_glob=cpp/build-support/lint_exclusions.txt \
--source_dir=cpp/src --quiet --fix
- name: Run clang-format on r
if: env.CLANG_FORMAT_R == 'true' || endsWith(github.event.comment.body, 'everything')
run: |
. .env # To get the clang version we use
cpp/build-support/run_clang_format.py \
--clang_format_binary=clang-format-${CLANG_TOOLS} \
--exclude_glob=cpp/build-support/lint_exclusions.txt \
--source_dir=r/src --quiet --fix
- uses: r-lib/actions/setup-r@v1
if: env.R_DOCS == 'true' || endsWith(github.event.comment.body, 'everything')
- name: Update R docs
if: env.R_DOCS == 'true' || endsWith(github.event.comment.body, 'everything')
shell: Rscript {0}
run: |
source("ci/etc/rprofile")
install.packages(c("remotes", "roxygen2"))
remotes::install_deps("r")
roxygen2::roxygenize("r")
- name: Commit results
run: |
git config user.name "$(git log -1 --pretty=format:%an)"
git config user.email "$(git log -1 --pretty=format:%ae)"
git commit -a -m 'Autoformat/render all the things [automated commit]' || echo "No changes to commit"
- uses: r-lib/actions/pr-push@master
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
# TODO(kszucs): re-enable the following workflows once r-lib/* actions are allowed to run
# autotune:
# name: "Fix all the things"
# if: startsWith(github.event.comment.body, '@github-actions autotune')
# runs-on: ubuntu-latest
# steps:
# - uses: actions/checkout@v2
# - uses: r-lib/actions/pr-fetch@master
# with:
# repo-token: ${{ secrets.GITHUB_TOKEN }}
# - name: See what is different
# run: |
# set -ex
# git remote add upstream https://github.com/apache/arrow
# git fetch upstream
# changed() {
# git diff --name-only HEAD..upstream/master | grep -e "$1" >/dev/null 2>&1
# }
# if changed '^r/.*\.R$'; then
# echo "R_DOCS=true" >> $GITHUB_ENV
# fi
# if changed 'cmake' || changed 'CMake'; then
# echo "CMAKE_FORMAT=true" >> $GITHUB_ENV
# fi
# if changed '^cpp/src'; then
# echo "CLANG_FORMAT_CPP=true" >> $GITHUB_ENV
# fi
# if changed '^r/src'; then
# echo "CLANG_FORMAT_R=true" >> $GITHUB_ENV
# fi
# - name: Run cmake_format
# if: env.CMAKE_FORMAT == 'true' || endsWith(github.event.comment.body, 'everything')
# run: |
# set -ex
# export PATH=/home/runner/.local/bin:$PATH
# python3 -m pip install --upgrade pip setuptools wheel
# python3 -m pip install -r dev/archery/requirements-lint.txt
# python3 run-cmake-format.py
# - name: Run clang-format on cpp
# if: env.CLANG_FORMAT_CPP == 'true' || endsWith(github.event.comment.body, 'everything')
# run: |
# . .env # To get the clang version we use
# cpp/build-support/run_clang_format.py \
# --clang_format_binary=clang-format-${CLANG_TOOLS} \
# --exclude_glob=cpp/build-support/lint_exclusions.txt \
# --source_dir=cpp/src --quiet --fix
# - name: Run clang-format on r
# if: env.CLANG_FORMAT_R == 'true' || endsWith(github.event.comment.body, 'everything')
# run: |
# . .env # To get the clang version we use
# cpp/build-support/run_clang_format.py \
# --clang_format_binary=clang-format-${CLANG_TOOLS} \
# --exclude_glob=cpp/build-support/lint_exclusions.txt \
# --source_dir=r/src --quiet --fix
# - uses: r-lib/actions/setup-r@v1
# if: env.R_DOCS == 'true' || endsWith(github.event.comment.body, 'everything')
# - name: Update R docs
# if: env.R_DOCS == 'true' || endsWith(github.event.comment.body, 'everything')
# shell: Rscript {0}
# run: |
# source("ci/etc/rprofile")
# install.packages(c("remotes", "roxygen2"))
# remotes::install_deps("r")
# roxygen2::roxygenize("r")
# - name: Commit results
# run: |
# git config user.name "$(git log -1 --pretty=format:%an)"
# git config user.email "$(git log -1 --pretty=format:%ae)"
# git commit -a -m 'Autoformat/render all the things [automated commit]' || echo "No changes to commit"
# - uses: r-lib/actions/pr-push@master
# with:
# repo-token: ${{ secrets.GITHUB_TOKEN }}

rebase:
name: "Rebase"
if: startsWith(github.event.comment.body, '@github-actions rebase')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: r-lib/actions/pr-fetch@master
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Rebase on ${{ github.repository }} master
run: |
set -ex
git config user.name "$(git log -1 --pretty=format:%an)"
git config user.email "$(git log -1 --pretty=format:%ae)"
git remote add upstream https://github.com/${{ github.repository }}
git fetch --unshallow upstream master
git rebase upstream/master
- uses: r-lib/actions/pr-push@master
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
args: "--force"
# rebase:
# name: "Rebase"
# if: startsWith(github.event.comment.body, '@github-actions rebase')
# runs-on: ubuntu-latest
# steps:
# - uses: actions/checkout@v2
# - uses: r-lib/actions/pr-fetch@master
# with:
# repo-token: ${{ secrets.GITHUB_TOKEN }}
# - name: Rebase on ${{ github.repository }} master
# run: |
# set -ex
# git config user.name "$(git log -1 --pretty=format:%an)"
# git config user.email "$(git log -1 --pretty=format:%ae)"
# git remote add upstream https://github.com/${{ github.repository }}
# git fetch --unshallow upstream master
# git rebase upstream/master
# - uses: r-lib/actions/pr-push@master
# with:
# repo-token: ${{ secrets.GITHUB_TOKEN }}
# args: "--force"
4 changes: 3 additions & 1 deletion dev/archery/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@

extras = {
'benchmark': ['pandas'],
'bot': ['ruamel.yaml', 'pygithub'],
# bot extra includes crossbow's dependencies
Copy link
Contributor

@terencehonles terencehonles Jan 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might it make sense to separate crossbow's dependencies and specify the install as arrow/dev/archery[bot,crossbow]?

That way you could install the dependencies with one of the other extras. This is partially because I'm not sure if I should use arrow/dev/archery[bot] for what I'm copying in #8915 from 7f11abe#diff-90e4e8bfd9b23a7b4bf4cfa1cb4bd0d80a87e21ac7fbd3a39c30f19709d4c705R50-R61

Copy link
Member Author

@kszucs kszucs Jan 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good idea, the same has just crossed my mind during working on #9096. Feel free to add it in #8915.

Copy link
Contributor

@terencehonles terencehonles Jan 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is repeated a lot, and I'm probably going to just open a different PR for cleaning up the repeated crossbow dependencies that way #8915 can hopefully be merged in sooner rather than later. It looks like it's built correctly (it was working before Travis disappeared).

'bot': ['ruamel.yaml', 'pygithub', 'github3.py', 'jinja2', 'jira',
'pygit2', 'setuptools_scm', 'toolz'],
'docker': ['ruamel.yaml', 'python-dotenv'],
'release': ['jinja2', 'jira', 'semver', 'gitpython']
}
Expand Down