Skip to content

Conversation

@kou
Copy link
Member

@kou kou commented Nov 16, 2022

PR to verify Release Candidate

@kou
Copy link
Member Author

kou commented Nov 16, 2022

@github-actions crossbow submit --group verify-rc-source --param release=10.0.1 --param rc=0

@github-actions
Copy link

Revision: a6eabc2

Submitted crossbow builds: ursacomputing/crossbow @ actions-36c520c6c6

Task Status
verify-rc-source-cpp-linux-almalinux-8-amd64 Github Actions
verify-rc-source-cpp-linux-conda-latest-amd64 Github Actions
verify-rc-source-cpp-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-cpp-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-cpp-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-cpp-macos-amd64 Github Actions
verify-rc-source-cpp-macos-arm64 Github Actions
verify-rc-source-cpp-macos-conda-amd64 Github Actions
verify-rc-source-csharp-linux-almalinux-8-amd64 Github Actions
verify-rc-source-csharp-linux-conda-latest-amd64 Github Actions
verify-rc-source-csharp-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-csharp-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-csharp-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-csharp-macos-amd64 Github Actions
verify-rc-source-csharp-macos-arm64 Github Actions
verify-rc-source-go-linux-almalinux-8-amd64 Github Actions
verify-rc-source-go-linux-conda-latest-amd64 Github Actions
verify-rc-source-go-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-go-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-go-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-go-macos-amd64 Github Actions
verify-rc-source-go-macos-arm64 Github Actions
verify-rc-source-integration-linux-almalinux-8-amd64 Github Actions
verify-rc-source-integration-linux-conda-latest-amd64 Github Actions
verify-rc-source-integration-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-integration-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-integration-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-integration-macos-amd64 Github Actions
verify-rc-source-integration-macos-arm64 Github Actions
verify-rc-source-integration-macos-conda-amd64 Github Actions
verify-rc-source-java-linux-almalinux-8-amd64 Github Actions
verify-rc-source-java-linux-conda-latest-amd64 Github Actions
verify-rc-source-java-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-java-macos-amd64 Github Actions
verify-rc-source-js-linux-almalinux-8-amd64 Github Actions
verify-rc-source-js-linux-conda-latest-amd64 Github Actions
verify-rc-source-js-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-js-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-js-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-js-macos-amd64 Github Actions
verify-rc-source-js-macos-arm64 Github Actions
verify-rc-source-python-linux-almalinux-8-amd64 Github Actions
verify-rc-source-python-linux-conda-latest-amd64 Github Actions
verify-rc-source-python-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-python-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-python-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-python-macos-amd64 Github Actions
verify-rc-source-python-macos-arm64 Github Actions
verify-rc-source-python-macos-conda-amd64 Github Actions
verify-rc-source-ruby-linux-almalinux-8-amd64 Github Actions
verify-rc-source-ruby-linux-conda-latest-amd64 Github Actions
verify-rc-source-ruby-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-ruby-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-ruby-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-ruby-macos-amd64 Github Actions
verify-rc-source-ruby-macos-arm64 Github Actions
verify-rc-source-windows Github Actions

@kou
Copy link
Member Author

kou commented Nov 16, 2022

Revision: a6eabc2

Submitted crossbow builds: ursacomputing/crossbow @ release-10.0.1-rc0-0

Task Status
almalinux-8-amd64 Github Actions
almalinux-8-arm64 Travis CI
almalinux-9-amd64 Github Actions
almalinux-9-arm64 Travis CI
amazon-linux-2-amd64 Github Actions
amazon-linux-2-arm64 Travis CI
centos-7-amd64 Github Actions
centos-8-stream-amd64 Github Actions
centos-8-stream-arm64 Travis CI
centos-9-stream-amd64 Github Actions
centos-9-stream-arm64 Travis CI
conan-maximum Github Actions
conan-minimum Github Actions
debian-bookworm-amd64 Github Actions
debian-bookworm-arm64 Travis CI
debian-bullseye-amd64 Github Actions
debian-bullseye-arm64 Travis CI
java-jars Github Actions
nuget Github Actions
python-sdist Github Actions
r-binary-packages Github Actions
test-ubuntu-default-docs Azure
ubuntu-bionic-amd64 Github Actions
ubuntu-bionic-arm64 Travis CI
ubuntu-focal-amd64 Github Actions
ubuntu-focal-arm64 Travis CI
ubuntu-jammy-amd64 Github Actions
ubuntu-jammy-arm64 Travis CI
wheel-macos-big-sur-cp310-arm64 Github Actions
wheel-macos-big-sur-cp311-arm64 Github Actions
wheel-macos-big-sur-cp38-arm64 Github Actions
wheel-macos-big-sur-cp39-arm64 Github Actions
wheel-macos-mojave-cp310-amd64 Github Actions
wheel-macos-mojave-cp311-amd64 Github Actions
wheel-macos-mojave-cp37-amd64 Github Actions
wheel-macos-mojave-cp38-amd64 Github Actions
wheel-macos-mojave-cp39-amd64 Github Actions
wheel-manylinux2014-cp310-amd64 Github Actions
wheel-manylinux2014-cp310-arm64 Travis CI
wheel-manylinux2014-cp311-amd64 Github Actions
wheel-manylinux2014-cp311-arm64 Travis CI
wheel-manylinux2014-cp37-amd64 Github Actions
wheel-manylinux2014-cp37-arm64 Travis CI
wheel-manylinux2014-cp38-amd64 Github Actions
wheel-manylinux2014-cp38-arm64 Travis CI
wheel-manylinux2014-cp39-amd64 Github Actions
wheel-manylinux2014-cp39-arm64 Travis CI
wheel-windows-cp310-amd64 Github Actions
wheel-windows-cp311-amd64 Github Actions
wheel-windows-cp37-amd64 Github Actions
wheel-windows-cp38-amd64 Github Actions
wheel-windows-cp39-amd64 Github Actions

@kou
Copy link
Member Author

kou commented Nov 16, 2022

@nealrichardson @paleolimbot @thisisnic r-binary-packages failed. Could you confirm this?

Attaching package: ‘arrow’

The following object is masked from ‘package:utils’:

    timestamp

Error in parquet___arrow___ArrowReaderProperties__Make(isTRUE(use_threads)) : 
  Cannot call parquet___arrow___ArrowReaderProperties__Make(). See https://arrow.apache.org/docs/r/articles/install.html for help installing Arrow C++ libraries. 
Calls: read_parquet ... <Anonymous> -> parquet___arrow___ArrowReaderProperties__Make
Execution halted

Note that the same error is happen on master too: https://github.com/ursacomputing/crossbow/actions/runs/3468776645/jobs/5795178913

@thisisnic
Copy link
Member

Weird, that error is triggered when the C++ library hasn't been built with enough functionality to support the R package, so I'd expect it to be something odd happening with the CI rather than a code change. I've re-triggered the job to make sure it's not some transient issue.

@thisisnic
Copy link
Member

thisisnic commented Nov 16, 2022

Hmm, failed again almost immediately: https://github.com/ursacomputing/crossbow/actions/runs/3475288883/jobs/5817793906

I'm not sure what's going on there. @assignUser - anything jump out at you there?

@assignUser
Copy link
Member

@thisisnic seems to be an issue with the rocker image overriding the repo settings and forcefull installing the packages via apt and thus installing r-cran-arrow instead of the local package: https://github.com/ursacomputing/crossbow/actions/runs/3475288883/jobs/5817793906#step:13:321

We might need/want to replace that image to stop that. Should not matter for this release though 👍

@kou
Copy link
Member Author

kou commented Nov 16, 2022

We might need/want to replace that image to stop that. Should not matter for this release though +1

If the test is failed, R binaries aren't uploaded: https://github.com/ursacomputing/crossbow/actions/runs/3475288883/jobs/5809989647

Is it OK for R? If it's OK, I'll continue with the failure.

@eitsupi
Copy link
Contributor

eitsupi commented Nov 16, 2022

seems to be an issue with the rocker image overriding the repo settings and forcefull installing the packages via apt and thus installing r-cran-arrow instead of the local package: ursacomputing/crossbow/actions/runs/3475288883/jobs/5817793906#step:13:321

This seems to be due to rocker/r-buntu:22.04 configuring bspm.
bspm replaces the behavior of the install.packages function and sets apt to install the R packages.
I recommend that you simply stop using this image.

@eddelbuettel Did you push the image to DockerHub without updating the Dockerfile on GitHub?
It seems that bspm is indeed installed on rocker/r-ubuntu@sha256:68f72a15f0e972556fb3dfd80b6394c34315722cb9b3e7a3a917a869152f7c40 even though the Dockerfile shows that bspm is not installed.

@eddelbuettel
Copy link
Contributor

@eitsupi If so then only by accident. But I did 'batch' update a few (layered) containers following the R 4.2.2 release and may inadvertendly (by doing this manually) crossed 'tags' and Dockerfiles. Building and pushing a new image '22.04' (also as 'jammy') now. And yes, you are correct: bspm should not be in this base layer but one above.

@kou
Copy link
Member Author

kou commented Nov 16, 2022

@eitsupi Thanks for the information!

Do you know Docker images that can be replaced with https://github.com/apache/arrow/blob/master/dev/tasks/r/github.packages.yml#L247-L250 ?

@eitsupi
Copy link
Contributor

eitsupi commented Nov 16, 2022

Do you know Docker images that can be replaced with master/dev/tasks/r/github.packages.yml#L247-L250 ?

I think rocker/r-ubuntu:22.04 can be replaced by rstudio/r-base:jammy.
As @eddelbuettel says, this change in behavior seems to have been an accident, so I think it can be left as is.
(However, I think the build system needs to be reviewed, as it is not good for this type of accident to happen.)

As for rocker/r-ver, it also should be possible to replace it with the image provided by rstudio (Posit).
https://github.com/rstudio/r-docker

However, the update mechanism of rocker/r-ver that I maintain has been fully automated on GitHub for over a year now, so I don't think an accident like this will happen.
https://github.com/rocker-org/rocker-versioned2

@eddelbuettel
Copy link
Contributor

eddelbuettel commented Nov 16, 2022

@kou "It's complicated." @eitsupi and I are both behind Rocker, and these choices present two different and not exactly interchangeable options with different settings. So in two characters: "no". In more words: "it depends, they both have their (different) use cases". I do all my CI with r-ubuntu / r-bspm / r2u as I want current toolchains and current CRAN easily. YMMV.

@eddelbuettel
Copy link
Contributor

in behavior seems to have been an accident,

And if so, has been fixed. Did you try the container I uploaded ~20 mins ago?

@assignUser
Copy link
Member

assignUser commented Nov 16, 2022

And if so, has been fixed. Did you try the container I uploaded ~20 mins ago?

I'll re-run the job

@eitsupi
Copy link
Contributor

eitsupi commented Nov 16, 2022

@eddelbuettel

And if so, has been fixed. Did you try the container I uploaded ~20 mins ago?

I pulled from DockerHub, but the digest does not seem to have changed.

image

It seems that bspm is indeed installed on rocker/r-ubuntu@sha256:68f72a15f0e972556fb3dfd80b6394c34315722cb9b3e7a3a917a869152f7c40 even though the Dockerfile shows that bspm is not installed.

@assignUser
Copy link
Member

@eddelbuettel
Copy link
Contributor

My bad, in the rush I did

 2157  docker build --no-cache -t rocker/r-ubuntu:jammy .
 2158  #docker build --no-cache -t rocker/r-ubuntu:jammy .
 2159  docker push rocker/r-ubuntu:jammy 
 2160  docker push rocker/r-ubuntu:22.04 

which replaced one but not both of these. I skipped the build step for 22,04

Will ship and update 22.04 tag in a minute or two -- now building.

@eddelbuettel
Copy link
Contributor

Rebuilt and repushed. I used --no-cache on both hence different sha1 (which confused me for a second so I did a rebuild or two). They should both work are identical, based on the same Dockerfile.

edd@rob:~/git/rocker-ubuntu-lts/jammy(master)$ docker images |head -3
REPOSITORY                           TAG             IMAGE ID       CREATED          SIZE
rocker/r-ubuntu                      jammy           7214e2c7ed54   7 minutes ago    743MB
rocker/r-ubuntu                      22.04           091427e0bcf7   11 minutes ago   743MB
edd@rob:~/git/rocker-ubuntu-lts/jammy(master)$ 

@kou
Copy link
Member Author

kou commented Nov 17, 2022

@eitsupi Sorry! I misunderstood.

@eddelbuettel Thanks! It worked: https://github.com/ursacomputing/crossbow/actions/runs/3468776645/jobs/5828688028

@kou
Copy link
Member Author

kou commented Nov 17, 2022

@github-actions crossbow submit --group verify-rc-binaries --group verify-rc-wheels --param release=10.0.1 --param rc=0

@github-actions
Copy link

Revision: a6eabc2

Submitted crossbow builds: ursacomputing/crossbow @ actions-96463f290f

Task Status
verify-rc-binaries-apt-linux-amd64 Github Actions
verify-rc-binaries-binary-linux-amd64 Github Actions
verify-rc-binaries-jars-linux-almalinux-8-amd64 Github Actions
verify-rc-binaries-jars-linux-conda-latest-amd64 Github Actions
verify-rc-binaries-jars-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-binaries-jars-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-binaries-jars-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-binaries-wheels-linux-almalinux-8-amd64 Github Actions
verify-rc-binaries-wheels-linux-conda-latest-amd64 Github Actions
verify-rc-binaries-wheels-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-binaries-wheels-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-binaries-wheels-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-binaries-wheels-macos-11-amd64 Github Actions
verify-rc-binaries-wheels-macos-11-arm64 Github Actions
verify-rc-binaries-wheels-macos-12-amd64 Github Actions
verify-rc-binaries-wheels-windows Github Actions
verify-rc-binaries-yum-linux-amd64 Github Actions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants