GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600

raulcd · 2025-09-19T09:33:13Z

Rationale for this change

There are several things that make this change wanted. We want to move some CI jobs from ursacomputing/crossbow to apache/arrow. Moving the Linux Packaging jobs will allow us to automate some release tasks and potentially (if we are able to make reproducible builds for linux packaging work) add automated signing to them avoiding having to require a PMC signature for the Linux packaging artifacts.

What changes are included in this PR?

Move check_labels and report_ci jobs to independent reusable workflows.
Update cpp_extra to use those.
Create new linux_packaging.yml workflow replicating work that was done on crossbow. Integrate that workflow with check_labels and report_ci
Update release binary submit and binary download to run workflow when tag is pushed and download the artifacts from the release instead of from the crossbow repository.

Are these changes tested?

Some via CI on fork and some manual testing.

Are there any user-facing changes?

No

GitHub Issue: [CI][Crossbow][Packaging] Use apache/arrow for linux-packaging jobs #47582

github-actions · 2025-09-19T09:33:41Z

⚠️ GitHub issue #47582 has been automatically assigned in GitHub to PR creator.

raulcd

Hi @kou I've started working on this and I think I have a working base that could work. I wanted to discuss it with you before moving more packages to validate we are all ok with the approach.
Let me know your thoughts. I have a couple of questions to discuss too, let me know what you think on those too.

.github/workflows/package_linux.yml

dev/release/04-binary-download.sh

.github/workflows/cpp_extra.yml

kou

Thanks!

.github/workflows/check_labels.yml

.github/workflows/cpp_extra.yml

.github/workflows/linux_packaging.yml

.github/workflows/package_linux.yml

.github/workflows/linux_packaging.yml

dev/release/04-binary-download.sh

raulcd

I've gone over all the comments and I think this is ready for another review.

dev/release/binary-task.rb

.github/workflows/package_linux.yml

dev/release/04-binary-download.sh

dev/release/binary-task.rb

dev/release/post-05-update-gh-release-notes.sh

dev/tasks/linux-packages/helper.rb

dev/release/utils-watch-gh-workflow.sh

kou · 2025-09-23T12:57:38Z

dev/release/04-binary-download.sh

+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
+
+RUN_ID=$(get_run_id)


Ah, you want to use a function defined in utils-watch-gh-workflow.sh.

dev/release/04-binary-download.sh

kou · 2025-10-07T01:39:32Z

This is ready.

I've fixed problems found by tag test on my fork.

raulcd

👍 I am happy to merge. I can't approve though as I created the PR

kou

+1

I merge this.

conbench-apache-arrow · 2025-10-07T13:06:04Z

After merging your PR, Conbench analyzed the 2 benchmarking runs that have been run so far on merge-commit e137a04.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.

apache/arrow#47600 is merged.

…ow (#3545) apache/arrow#47600 is merged. It changed required directory structure. We don't need to change directory structure now. Fixes #3510.

…arrow repository (apache#47600) ### Rationale for this change There are several things that make this change wanted. We want to move some CI jobs from `ursacomputing/crossbow` to `apache/arrow`. Moving the Linux Packaging jobs will allow us to automate some release tasks and potentially (if we are able to make reproducible builds for linux packaging work) add automated signing to them avoiding having to require a PMC signature for the Linux packaging artifacts. ### What changes are included in this PR? - Move `check_labels` and `report_ci` jobs to independent reusable workflows. - Update `cpp_extra` to use those. - Create new `linux_packaging.yml` workflow replicating work that was done on crossbow. Integrate that workflow with `check_labels` and `report_ci` - Update release binary submit and binary download to run workflow when tag is pushed and download the artifacts from the release instead of from the crossbow repository. ### Are these changes tested? Some via CI on fork and some manual testing. ### Are there any user-facing changes? No * GitHub Issue: apache#47582 Lead-authored-by: Raúl Cumplido <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

Apache Arrow changed SRPM output location from `source/SRPMS/` to `Source/Packages/`. - ref: apache/arrow#47600 This change breaks Yum repository compatibility for existed distributions. This commit adds path management after `yum:build` to maintain compatibility: - For AlmaLinux 10 and earlier and Amazon Linux 2023: Rename SRPMs path from `Source/Packages/` to `source/` for backward compatibility - For AlmaLinux 11 and later: Uses the new `Source/Packages/` path - Except for x86 architecture: Removes SRPMs --------- Co-authored-by: Sutou Kouhei <[email protected]>

github-actions bot added the awaiting committer review Awaiting committer review label Sep 19, 2025

raulcd commented Sep 19, 2025

View reviewed changes

.github/workflows/package_linux.yml Outdated Show resolved Hide resolved

dev/release/04-binary-download.sh Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 19, 2025

raulcd commented Sep 19, 2025

View reviewed changes

.github/workflows/cpp_extra.yml Outdated Show resolved Hide resolved

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Sep 19, 2025

raulcd force-pushed the GH-47582 branch from 9e3e8e4 to 66df8cb Compare September 19, 2025 11:30

kou reviewed Sep 19, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Sep 19, 2025

raulcd force-pushed the GH-47582 branch 3 times, most recently from 1121d15 to 78e0ff2 Compare September 22, 2025 10:35

raulcd commented Sep 23, 2025

View reviewed changes

dev/release/binary-task.rb Show resolved Hide resolved

dev/release/binary-task.rb Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Sep 23, 2025

kou reviewed Sep 23, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Sep 23, 2025

kou added 9 commits October 7, 2025 06:40

Fix download path

c7877b1

Remove nonexistent define_r_tasks

b08fb87

Adjust Ubuntu component

dd9eac3

Adjust SRPM path

4e380ff

Debug

276d977

Fix detection

eb958fd

Fix detection

72a91bb

Fix path

5c0281d

Fix syntax

fffe6cd

Fix name

b368bb6

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 7, 2025

raulcd commented Oct 7, 2025

View reviewed changes

kou approved these changes Oct 7, 2025

View reviewed changes

kou merged commit e137a04 into apache:main Oct 7, 2025
60 of 61 checks passed

kou removed the awaiting changes Awaiting changes label Oct 7, 2025

kou mentioned this pull request Oct 7, 2025

[CI][Crossbow][Packaging] Use apache/arrow for linux-packaging jobs #47582

Closed

github-actions bot added the awaiting merge Awaiting merge label Oct 7, 2025

raulcd deleted the GH-47582 branch October 7, 2025 09:22

kou added a commit to kou/arrow-adbc that referenced this pull request Oct 8, 2025

chore(dev/release): Follow Linux package upload changes in apache/arrow

cc91177

apache/arrow#47600 is merged.

kou mentioned this pull request Oct 8, 2025

chore(dev/release): Follow Linux package upload changes in apache/arrow apache/arrow-adbc#3545

Merged

kou added a commit to kou/arrow-adbc that referenced this pull request Oct 8, 2025

chore(dev/release): Follow Linux package upload changes in apache/arrow

7d631e3

apache/arrow#47600 is merged.

raulcd mentioned this pull request Oct 8, 2025

[CI] Zulip report for Extra CI jobs incorrecly shows report-extra-cpp / report-ci as failed #47751

Closed

otegami mentioned this pull request Oct 15, 2025

package yum: keep SRPM paths for backward compatibility groonga/groonga#2558

Merged

raulcd mentioned this pull request Oct 17, 2025

[CI] Nightly extra reports are not being sent when a job fails #47844

Closed

GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600

GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600

Uh oh!

Conversation

raulcd commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Sep 19, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kou Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kou commented Oct 7, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

conbench-apache-arrow bot commented Oct 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

raulcd commented Sep 19, 2025 •

edited

Loading