GH-36634: [Dev] Ensure merge script goes over all pages when requesting info from GitHub

[raulcd]

Rationale for this change

We currently were missing maintenance branches due to pagination on GH API.

What changes are included in this PR?

Check whether the API is returning a paginated view and extend the list returned.

Are these changes tested?

I have tested locally:

(Pdb) pr.maintenance_branches
['maint-0.11.x', 'maint-0.12.x', 'maint-0.14.x', 'maint-0.15.x', 'maint-0.17.x', 'maint-1.0.x', 'maint-3.0.x', 'maint-4.0.x', 'maint-6.0.x', 'maint-7.0.x', 'maint-7.0.1', 'maint-8.0.x', 'maint-9.0.0', 'maint-10.0.x', 'maint-10.0.0', 'maint-10.0.1', 'maint-11.0.0', 'maint-12.0.x', 'maint-12.0.0', 'maint-12.0.1', 'maint-13.0.0']
(Pdb) c
Enter fix version [14.0.0]: 

Are there any user-facing changes?

No

• Closes: [Dev] Merge PR script defaults to the wrong milestone when merging #36634

[raulcd]

There are currently no tests on the merge_pr script around neither GH issues or any interaction with GH. We should add them but this is tracked on this PR: #14802

[raulcd]

Just for the record, other libraries like pygithub use the link header for the pagination and do a similar parsing: https://github.com/PyGithub/PyGithub/blob/de80ff4b914b9ffd47347d60b14cbf270e07f346/github/PaginatedList.py#L241

[assignUser]

Ah good catch! Is this something we need to add to archery somewhere too? Or not as that is using an api wrapper that handles it?

[kou]

+1

[raulcd]

Is this something we need to add to archery somewhere too? Or not as that is using an api wrapper that handles it?

The pygithub and github3.py libraries used on archery handle pagination. I am not sure if there's any place where we should explicitly call to get next pages, etcetera but I don't think so as from what I've seen they just return iterators and handle that for us. Would be good though to try and refactor archery to use a single library so we are sure around those: #33241

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit a4384d9.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.