ARROW-18345: [R] Create a CRAN-specific packaging checklist that lives in the R package directory #14678
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -487,7 +487,39 @@ Be sure to go through on the following checklist: | |
| :class-title: sd-fs-5 | ||
| :class-container: sd-shadow-md | ||
|
|
||
| To publish the R package on CRAN, there are a few steps we need to do first | ||
| in order to ensure that binaries for Windows and macOS are available to CRAN. | ||
| Jeroen Ooms <[email protected]> maintains several projects that build C++ | ||
| dependencies for R packages for macOS and Windows. We test copies of these | ||
| same build scripts in our CI, and at release time, we need to send any | ||
| changes we have and update the versions/hashes upstream. | ||
|
|
||
| When the release candidate is made, make draft pull requests to each | ||
| repository using the rc, updating the version and SHA, as well as any cmake | ||
| build changes from the corresponding files in apache/arrow. Jeroen may | ||
| merge these PRs before the release vote passes, build the binary artifacts, | ||
| and publish them in the right places so that we can do pre-submission checks | ||
| (see below). After the release candidate vote passes, update these PRs | ||
| to point to the official (non-rc) URL and mark them as ready for review. | ||
| Jeroen will merge, build the binary artifacts, and publish them in the | ||
| right places. See the | ||
| `packaging checklist <https://github.com/apache/arrow/blob/master/r/PACKAGING.md>`_. | ||
| for a precise list of pull requests that must be made prior to submission | ||
| to CRAN. | ||
|
|
||
| Once these binary prerequisites have been satisfied, we can submit to CRAN. | ||
| Given the vagaries of the process, it is best if the R developers on the | ||
| project verify the CRAN-worthiness of the package before submitting. | ||
| Our CI systems give us some coverage for the things that CRAN checks, but | ||
| there are a couple of final tests we should do to confirm that the release | ||
| binaries will work and that everything runs on the same infrastructure that | ||
| CRAN has, which is difficult/impossible to emulate fully on Travis or with | ||
| Docker. For a precise list of checks, see the | ||
| `packaging checklist <https://github.com/apache/arrow/blob/master/r/PACKAGING.md>`_. | ||
|
|
||
| Once all checks are clean, we submit to CRAN, which has a web form for | ||
| uploading packages. The release process requires email confirmation | ||
| from the R package maintainer, currently Neal Richardson. | ||
|
|
||
| .. dropdown:: Update vcpkg port | ||
| :animate: fade-in-slide-down | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,3 +30,4 @@ STYLE.md | |
| ^cheatsheet$ | ||
| ^revdep$ | ||
| ^vignettes$ | ||
| ^PACKAGING\.md$ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,136 @@ | ||||||
|
|
||||||
| <!--- | ||||||
| Licensed to the Apache Software Foundation (ASF) under one | ||||||
| or more contributor license agreements. See the NOTICE file | ||||||
| distributed with this work for additional information | ||||||
| regarding copyright ownership. The ASF licenses this file | ||||||
| to you under the Apache License, Version 2.0 (the | ||||||
| "License"); you may not use this file except in compliance | ||||||
| with the License. You may obtain a copy of the License at | ||||||
|
|
||||||
| http://www.apache.org/licenses/LICENSE-2.0 | ||||||
|
|
||||||
| Unless required by applicable law or agreed to in writing, | ||||||
| software distributed under the License is distributed on an | ||||||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||||||
| KIND, either express or implied. See the License for the | ||||||
| specific language governing permissions and limitations | ||||||
| under the License. | ||||||
| --> | ||||||
|
|
||||||
| # Packaging checklist for CRAN release | ||||||
|
|
||||||
| For a high-level overview of the release process see the | ||||||
| [Apache Arrow Release Management Guide](https://arrow.apache.org/docs/developers/release.html#post-release-tasks). | ||||||
|
|
||||||
| Before the release candidate is cut: | ||||||
|
|
||||||
| - [ ] [Create a GitHub issue](https://github.com/apache/arrow/issues/new/) | ||||||
| entitled `[R] CRAN packaging checklist for version X.X.X` | ||||||
| and copy this checklist to the issue. | ||||||
| - [ ] Evaluate the status of any failing | ||||||
| [nightly tests and nightly packaging builds](http://crossbow.voltrondata.com). These checks | ||||||
| replicate most of the checks that CRAN runs, so we need them all to be passing | ||||||
| or to understand that the failures may (though won't necessarily) result in a rejection from CRAN. | ||||||
| - [ ] Check [current CRAN check results](https://cran.rstudio.org/web/checks/check_results_arrow.html) | ||||||
thisisnic marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
paleolimbot marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
| - [ ] Ensure the contents of the README are accurate and up to date | ||||||
| - [ ] Run `urlchecker::url_check()` on the R directory at the release candidate | ||||||
| commit. Ignore any errors with badges as they will be removed in the CRAN release branch. | ||||||
| - [ ] [Polish NEWS](https://style.tidyverse.org/news.html#news-release) but do **not** update version numbers (this is done automatically later). | ||||||
| - [ ] For major releases, prepare tweet thread highlighting new features | ||||||
|
|
||||||
| Wait for the release candidate to be cut: | ||||||
|
|
||||||
| - [ ] Release candidate! | ||||||
| - [ ] Create a CRAN-release branch from the release candidate commit | ||||||
|
|
||||||
| Make pull requests into the [autobrew](https://github.com/autobrew) and | ||||||
| [rtools-packages](https://github.com/r-windows/rtools-packages) repositories | ||||||
| used by the configure script on MacOS and Windows. These pull requests will | ||||||
| use the release candidate as the source. | ||||||
|
|
||||||
| - [ ] Pull request to modify | ||||||
| [the apache-arrow autobrew formula]( https://github.com/autobrew/homebrew-core/blob/high-sierra/Formula/apache-arrow.rb) | ||||||
| to update the release version, SHA256 checksum of the release source file (which can be found in the same directory as the release source file), and any changes to dependencies and build steps that have changed in the | ||||||
| [copy of the formula we have of that formula in the Arrow repo](https://github.com/apache/arrow/blob/master/dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb) | ||||||
| - [ ] Pull request to modify | ||||||
| [the apache-arrow-static autobrew formula]( https://github.com/autobrew/homebrew-core/blob/master/Formula/apache-arrow-static.rb) | ||||||
|
Member
Author
There was a problem hiding this comment. Fixed! (By searching https://github.com/autobrew/homebrew-core/pulls?q=is%3Apr+is%3Aclosed+arrow )
Member
There was a problem hiding this comment.
Suggested change
|
||||||
| to update the version, SHA, and any changes to dependencies and build steps that have changed in the | ||||||
| [copy of the formula we have of that formula in the Arrow repo](https://github.com/apache/arrow/blob/master/dev/tasks/homebrew-formulae/autobrew/apache-arrow-static.rb) | ||||||
| - [ ] Pull request to modify the | ||||||
|
Member
There was a problem hiding this comment. Might be good to note that there often are no changes in this script, and that's ok. |
||||||
| [autobrew script](https://github.com/autobrew/scripts/blob/master/apache-arrow) | ||||||
| to include any additions made to | ||||||
| [r/tools/autobrew](https://github.com/apache/arrow/blob/master/r/tools/autobrew). | ||||||
| - [ ] Pull request to modify the | ||||||
| [RTools PKGBUILD script](https://github.com/r-windows/rtools-packages/blob/master/mingw-w64-arrow/PKGBUILD) | ||||||
| to reflect changes in | ||||||
| [ci/PKGBUILD](https://github.com/apache/arrow/blob/master/ci/scripts/PKGBUILD), | ||||||
| uncommenting the line that says "uncomment to test the rc". | ||||||
|
|
||||||
| Prepare and check the .tar.gz that will be released to CRAN. | ||||||
|
|
||||||
| - [ ] `git fetch upstream && git checkout release-X.X.X-rcXX && git clean -f -d` | ||||||
| - [ ] Run `make build`. This copies Arrow C++ into tools/cpp, prunes some | ||||||
| unnecessary components, and runs `R CMD build` to generate the source tarball. | ||||||
| Because this will install the package, you will need to ensure that the version | ||||||
| of Arrow C++ available to the configure script is the same as the version | ||||||
| that is vendored into the R package (e.g., you may need to unset `ARROW_HOME`). | ||||||
| - [ ] `devtools::check_built("arrow_X.X.X.tar.gz")` locally | ||||||
|
Member
There was a problem hiding this comment. If you're doing |
||||||
| - [ ] Run reverse dependency checks. Currently this is a | ||||||
| [manual process](https://gist.github.com/paleolimbot/630fdab1e204d70fea97633d8fa15ccb); | ||||||
| however, in the future it may be a crossbow nightly job. | ||||||
|
|
||||||
| Wait for the official release... | ||||||
|
|
||||||
| - [ ] Release vote passed! | ||||||
| - [ ] If the release candidate commit updated, rebase the CRAN release branch | ||||||
| on that commit. | ||||||
| - [ ] Pick any commits that were made to master since the release commit that | ||||||
| were needed to fix CRAN-related submission issues identified in the above | ||||||
| steps. | ||||||
| - [ ] Remove badges from README.md | ||||||
| - [ ] Run `urlchecker::url_check()` on the R directory | ||||||
| - [ ] Create a PR entitled `WIP: [R] Verify CRAN release-10.0.1-rc0`. Add | ||||||
| a comment `@github-actions crossbow submit --group r` to run all R crossbow | ||||||
| jobs against the CRAN-specific release branch. | ||||||
|
Comment on lines
+93
to
+95
Member
There was a problem hiding this comment. This is great, much more clear, thanks!
Member
There was a problem hiding this comment. Actually, wait, what branches are we comparing on the PR? Does it even matter? I made an arbitrary one which branched off the release candidate and made a whitespace change. Should we recommend that here?
Member
Author
There was a problem hiding this comment. For crossbow I think we just need any PR to exist as long as "from" is correct. I think a whitespace change is fine for now but we should find a workflow that is less of a hack in the future. I don't have a good answer right now though! |
||||||
| - [ ] Regenerate arrow_X.X.X.tar.gz (i.e., `make build`) | ||||||
|
|
||||||
| Create new autobrew and r-windows PRs such that they use the *release* | ||||||
| instead of the *release candidate*; ensure linux binary packages are available: | ||||||
|
|
||||||
| - [ ] PR into autobrew/homebrew-core (apache-arrow autobrew formula) | ||||||
| - [ ] PR into autobrew/homebrew-core (apache-arrow-static autobrew formula) | ||||||
| - [ ] PR into autobrew/scripts | ||||||
| - [ ] PR into r-windows/rtools-packages | ||||||
| - [ ] Ensure linux binaries are available in the artifactory: | ||||||
| https://apache.jfrog.io/ui/repos/tree/General/arrow/r | ||||||
|
|
||||||
| Check binary Arrow C++ distributions specific to the R package: | ||||||
|
|
||||||
| - [ ] Upload the .tar.gz to [win-builder](https://win-builder.r-project.org/upload.aspx) (r-devel only) | ||||||
| and confirm (with Neal, who will automatically receive an email about the results) that the check is clean. | ||||||
| This step cannot be completed before Jeroen has put the binaries in the MinGW repository, i.e. [here](https://ftp.opencpu.org/rtools/ucrt64/), [here](https://ftp.opencpu.org/rtools/mingw64/), and [here](https://ftp.opencpu.org/rtools/mingw32/). | ||||||
| - [ ] Upload the .tar.gz to [MacBuilder](https://mac.r-project.org/macbuilder/submit.html) | ||||||
| and confirm that the check is clean | ||||||
| - [ ] Check `install.packages("arrow_X.X.X.tar.gz")` on Ubuntu and ensure that the | ||||||
| hosted binaries are used | ||||||
| - [ ] `devtools::check_built("arrow_X.X.X.tar.gz")` locally one more time (for luck) | ||||||
|
|
||||||
| Submit! | ||||||
|
|
||||||
| - [ ] Upload arrow_X.X.X.tar.gz to the | ||||||
| [CRAN submit page](https://xmpalantir.wu.ac.at/cransubmit/) | ||||||
| - [ ] Confirm the submission email | ||||||
|
|
||||||
| Wait for CRAN... | ||||||
|
|
||||||
| - [ ] Accepted! | ||||||
| - [ ] Tag the tip of the CRAN-specific release branch | ||||||
|
Member
There was a problem hiding this comment. Tag how? This isn't something we've done before
Member
Author
There was a problem hiding this comment. I suppose we don't have to (as long as there is a way to get back to the source at the time of the CRAN release in case we need to patch something?)
Member
There was a problem hiding this comment. Hmm, I've created a branch on my fork called r-10.0.1, but we should probably do something more robust.
Member
Author
There was a problem hiding this comment. I think for now that is OK. For 11.0.0, perhaps we can file an issue or ask on the mailing list what the appropriate branch name would be in the main Arrow repo? There is not currently any equivalent branch on the main repo for the packaging of a component but I imagine this has come up before (patches applied prior to packaging a component). |
||||||
| - [ ] Add a new line to the matrix in the [backwards compatability job](https://github.com/apache/arrow/blob/master/dev/tasks/r/github.linux.arrow.version.back.compat.yml) | ||||||
thisisnic marked this conversation as resolved.
Show resolved
Hide resolved
paleolimbot marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| - [ ] (patch releases only) Update the package version in `ci/scripts/PKGBUILD`, `dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb`, `r/DESCRIPTION`, and `r/NEWS.md` | ||||||
| - [ ] Update the packaging checklist template to reflect any new realities of the | ||||||
| packaging process. | ||||||
| - [ ] Wait for CRAN-hosted binaries on the | ||||||
| [CRAN package page](https://cran.r-project.org/package=arrow) to reflect the | ||||||
| new version | ||||||
| - [ ] Tweet! | ||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where would I find this official URL?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think they are embedded in the hombrew files (there are links to the homebrew scripts in the checklist). I think there will be a number of specific additions after we get through this release and I wonder if we could use the release issue itself as a place to collect additions that would be useful?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Each of these scripts has the official and rc URLs in them, and it's a matter of updating the versions and (un)commenting to enable one or the other.