GH-39958: [Python][CI] Remove upper pin on pytest

[AlenkaF]

Rationale for this change

The latest version of pytest (8.0.0) is breaking our CI:

• S3 fixture from out test suite fails
• doctest-cython check fails

What changes are included in this PR?

• added allow_bucket_creation=True to the s3_example_fs fixture
• removed the pin on pytest, except for the doc builds

Are these changes tested?

Yes.

Are there any user-facing changes?

No

Closes:

• Closes [Python][CI] Remove upper pin on pytest #39958
• Closes [Python][CI] S3 fixtures fail with Pytest 8.0.0 #39957

• GitHub Issue: [Python][CI] Remove upper pin on pytest #39958

[github-actions]

⚠️ GitHub issue #39958 has been automatically assigned in GitHub to PR creator.

[kou]

@github-actions crossbow submit -g python

[kou]

+1 if CI is green

BTW, could you use our PR template instead of just listing related issue links?

[github-actions]

Revision: 701807a

Submitted crossbow builds: ursacomputing/crossbow @ actions-2dab689a54

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3-amd64
test-debian-11-python-3-i386
test-fedora-39-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

[AlenkaF]

BTW, could you use our PR template instead of just listing related issue links?

Will do, thank you for reminding me!

[AlenkaF]

@jorisvandenbossche do you mind giving a quick look at this changes?

[jorisvandenbossche]

I would personally prefer keeping doctest-cython running on our CI. Is it urgent to remove the pytest pin? Or we could also leave the pin just for the docs build where the doctests are ran?

[jorisvandenbossche]

added allow_bucket_creation=True to the s3_example_fs fixture

Do we have any understanding about why pytest 8 needs this change?

[AlenkaF]

I would personally prefer keeping doctest-cython running on our CI. Is it urgent to remove the pytest pin? Or we could also leave the pin just for the docs build where the doctests are ran?

We can also wait with merging this PR till lgpage/pytest-cython#58 is merged. I am OK with any of the proposed options.

added allow_bucket_creation=True to the s3_example_fs fixture

Do we have any understanding about why pytest 8 needs this change?

No 😊 Will research.

[AlenkaF]

I have kept the doctest-cython command on our CI (conda-python-docs) and pinned pytest to lower than 8 in this case.

As for the allow_bucket_creation=True in the s3_example_fs fixture, we have agreed there is no point in spending too much time on researching why it is needed. Looking at the PR that added this argument in other cases where S3FileSystem is created from a URI (#13206) it makes sense to also add it to the s3_example_fs fixture.

[jorisvandenbossche]

@github-actions crossbow submit -g python

[github-actions]

Revision: d5b4d40

Submitted crossbow builds: ursacomputing/crossbow @ actions-3ecfe9f5ae

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3-amd64
test-debian-11-python-3-i386
test-fedora-39-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

[jorisvandenbossche]

Thanks!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6d5cfb2.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6d5cfb2.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.