feature: display progress bar when downloading multi-page responses

[hydrosquall]

Motivation

For a long running command (longer than 1 minute) for a big table (like pull requests or commits), it can be tricky to know if the script is still running, or if a rate limit/error was encountered

We know how many pages there are, so it may be possible to indicate how many remain.

github-to-sqlite/github_to_sqlite/utils.py

Line 367 in a6e237f

url = "https://api.github.com/repos/{}/pulls?state=all&filter=all".format(repo)

Resources

• Using the existing Click API:
  • https://click.palletsprojects.com/en/5.x/utils/#showing-progress-bars
• Loading spinner: https://github.com/pavdmyt/yaspin
• Progress bar: https://github.com/tqdm/tqdm

[simonw]

That's interesting - yeah it looks like the number of pages can be derived from the Link header, which is enough information to show a progress bar, probably using Click just to avoid adding another dependency.

https://docs.github.com/en/rest/guides/traversing-with-pagination