[CI][R][Homebrew] Nightly CI jobs aren't maintained

[kou]

Describe the bug, including details regarding any error messages, version, and platform.

homebrew-r-autobrew and homebrew-r-brew nightly CI jobs are failing in past 3 months:

Can we maintain them? If we can't maintain them, can we remove them? Are they useful to maintain something?

http://crossbow.voltrondata.com/

Task Name	Since Last Successful Build	Last Successful Commit	Last Successful Build	First Failure	9 Days Ago	8 Days Ago	7 Days Ago	6 Days Ago	5 Days Ago	4 Days Ago	3 Days Ago	2 Days Ago	Most Recent Failure
homebrew-r-autobrew	94 days	cf27001	pass	fail	fail	fail	fail	fail	fail	fail	fail	fail	fail
homebrew-r-brew	83 days	a63e60b	pass	fail	fail	fail	fail	fail	fail	fail	fail	fail	fail

• [R][C++][CI] Homebrew can't install 3.11 on GHA runners #15114
• [R][CI] pyarrow tests fail on macos 10.13 due to missing pyarrow wheel #15115
• ARROW-18240: [R] head() is crashing on some nightly builds #14582
• [CI][R] mutex lock failed: Invalid argument on MacOS 10.13 on process shutdown #15189

Component(s)

Continuous Integration, R

[assignUser]

We have been working on them and they have failed for different reasons iirc @paleolimbot is working on them currently

[paleolimbot]

Yes - see ARROW-18240 and #14582. Those nightly failures probably have the same cause as the valgrind job (which has been failing for around the same amount of time), and are indicative of real bugs. The bugs have not resulted in any user reports with the same failure mode (deadlock or crash when using head() on a dataset), so our maintenance effort has gone elsewhere. Last week I was finally able to reproduce something similar locally (see the PR/Jira) and I'm confident we can get this fixed for 11.0.0.

[paleolimbot]

#14582 will fix the segfaults that were occurring in those jobs; however, there is still:

══ Failed tests ════════════════════════════════════════════════════════════════
── Failure ('test-python.R:29'): install_pyarrow ───────────────────────────────
`install_pyarrow("arrow-test", nightly = TRUE)` threw an unexpected error.
Message: Error installing package(s): "'pyarrow'"
Class:   simpleError/error/condition
Backtrace:
     ▆
  1. ├─testthat::expect_error(...) at test-python.R:29:2
  2. │ └─testthat:::expect_condition_matching(...)
  3. │   └─testthat:::quasi_capture(...)
  4. │     ├─testthat (local) .capture(...)
  5. │     │ └─base::withCallingHandlers(...)
  6. │     └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
  7. └─arrow::install_pyarrow("arrow-test", nightly = TRUE)
  8.   └─reticulate::py_install(...)
  9.     └─reticulate::virtualenv_install(...)
 10.       └─reticulate:::pip_install(...)

( https://github.com/ursacomputing/crossbow/actions/runs/3715553353/jobs/6300858130#step:7:24012 )

I suspect that something like "python installed without pip" or "pip version is insufficient" or "the virtualenv that we create to install it already exists and is invalid" is happening but I don't know the details of those runners.

[assignUser]

I will check if this is an issue with the runners...

[kou]

Can we improve arrow::install_pyarrow() implementation (or our tests) to show the error message from pip install on error?

[jonkeane]

We have some output in the test output, here's the error (and this is all for the 10.13 runners):

https://github.com/ursacomputing/crossbow/actions/runs/3799336658/jobs/6461765126#step:7:10697

  Start test: install_pyarrow
  virtualenv: arrow-test
  Using virtual environment 'arrow-test' ...
  + '/Users/voltrondata/.virtualenvs/arrow-test/bin/python' -m pip install --upgrade --no-user --extra-index-url https://repo.fury.io/arrow-nightlies/ --pre --upgrade 'pyarrow'
  Looking in indexes: https://pypi.org/simple, https://repo.fury.io/arrow-nightlies/
  Requirement already satisfied: pyarrow in /Users/voltrondata/.virtualenvs/arrow-test/lib/python3.9/site-packages (10.0.0.dev211)
  Collecting pyarrow
    Downloading https://repo.fury.io/arrow-nightlies/-/ver_FFekQ/pyarrow-11.0.0.dev336.tar.gz (1.0 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 9.9 MB/s eta 0:00:00
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Discarding https://repo.fury.io/arrow-nightlies/-/ver_FFekQ/pyarrow-11.0.0.dev336.tar.gz#md5=0ef59391d75d28053f5e899f824b3842 (from https://repo.fury.io/arrow-nightlies/pyarrow/): Requested pyarrow from https://repo.fury.io/arrow-nightlies/-/ver_FFekQ/pyarrow-11.0.0.dev336.tar.gz#md5=0ef59391d75d28053f5e899f824b3842 has inconsistent version: filename has '11.0.0.dev336', but metadata has '11.0.0a0'
    Downloading https://repo.fury.io/arrow-nightlies/-/ver_1ASXRV/pyarrow-11.0.0.dev334.tar.gz (1.0 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 7.6 MB/s eta 0:00:00
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Discarding https://repo.fury.io/arrow-nightlies/-/ver_1ASXRV/pyarrow-11.0.0.dev334.tar.gz#md5=1c4676db5c0ef74a27bc307213c40b82 (from https://repo.fury.io/arrow-nightlies/pyarrow/): Requested pyarrow from https://repo.fury.io/arrow-nightlies/-/ver_1ASXRV/pyarrow-11.0.0.dev334.tar.gz#md5=1c4676db5c0ef74a27bc307213c40b82 has inconsistent version: filename has '11.0.0.dev334', but metadata has '11.0.0a0'
  ...

I tried in #15112 to remove the virtual env that's being used (it looks like there's an already-installed version of pyarrow there), but that fails still with a similar error. I'm suspicious about has inconsistent version: filename has '11.0.0.dev334', but metadata has '11.0.0a0' I can't seem to find where that 11.0.0a0 is coming from (it's not in the tar.gz and it's not in our source as far as I can tell. Eventually after looking at all of the hosted sources, it falls back to 10.0.1 (https://github.com/ursacomputing/crossbow/actions/runs/3799336658/jobs/6461765126#step:9:10737) and then that fails to build from source.

Ultimately, I suspect what's happening is that because we don't build pyarrow nightly binaries for 10.13 anymore (since 10.0.0dev222 if I'm looking at https://pypi.fury.io/arrow-nightlies/pyarrow correctly), this job won't work easily. Maybe we should skip our pyarrow tests for 10.13 (they will be skipped on cran already...)?

[assignUser]

Maybe we should skip our pyarrow tests for 10.13 (they will be skipped on cran already...)?

As we run on 10.13 for CRAN's sake only I think this is the way forward...

[jonkeane]

And just looked, the last 10.13 pyarrow binary was on 15 september 2022, which is a bit after the job generally started failing (121 days ago 30 August 2022). I'm trying to double-check this with a crossbow build from 10 september — before we stopped pyarrow 10.13 wheels, but after the failures started which might be the head issue that Dewey is looking at)

[assignUser]

I have created separate issues (and PRs) and added them as tracked issues to the issue description 👍