`pycurl` update causing "500 : Internal Server Error" after logging into newly created TLJHs using Google Authenticator

[jrdnbradford]

I originally posted this issue at jupyterhub/the-littlest-jupyterhub#964 but it may belong here instead.

Bug description

Installing TLJH and configuring the Google Authenticator results in 500 : Internal Server Error after login.

Bug resolution

I resolved this by comparing the hub environment on a working TLJH with the hub environment on a newly created TLJH and downgrading pycurl from 7.45.3 to 7.45.2:

sudo su
source /opt/tljh/hub/bin/activate
pip install --force-reinstall pycurl==7.45.2
tljh-config reload

I see requirements-hub-env.txt is setup with pycurl>=7.45.2,<8 . Not sure if it should be pinned to 7.45.2 in that repo or if it should be pinned here.

How to reproduce

1. Install TLJH and configure Google Authenticator:

from oauthenticator.google import GoogleOAuthenticator
c.JupyterHub.authenticator_class = GoogleOAuthenticator

c.GoogleOAuthenticator.google_service_account_keys = {DOMAIN: GROUP_ADMIN_SECRET_PATH}
c.GoogleOAuthenticator.gsuite_administrator = {DOMAIN: GSUITE_ADMINISTRATOR}
c.GoogleOAuthenticator.allowed_google_groups = {DOMAIN: ALLOWED_GOOGLE_GROUPS}
c.GoogleOAuthenticator.admin_google_groups = {DOMAIN: ADMIN_GOOGLE_GROUPS}
c.GoogleOAuthenticator.client_id = OAUTH_CLIENT_ID
c.GoogleOAuthenticator.client_secret = OAUTH_CLIENT_SECRET
c.GoogleOAuthenticator.hosted_domain = DOMAIN
c.GoogleOAuthenticator.login_service = DOMAIN_NAME
c.GoogleOAuthenticator.oauth_callback_url = 'https://DNS_NAME/hub/oauth_callback'

2. Attempt to sign-in with an allowed Google account

Expected behaviour

User should be able to login.

Actual behaviour

500 : Internal Server Error occurs after user attempts to login with allowed account.

Logs reveal a tornado.curl_httpclient.CurlError. See Logs section.

Your personal set up

Ubuntu 22.04
Littlest JupyterHub 1.0.0

Full environment

Here's the offending environment:

source /opt/tljh/hub/bin/activate
pip list
Package                          Version
-------------------------------- ---------
aiohttp                          3.9.3
aiosignal                        1.3.1
alembic                          1.13.1
async-generator                  1.10
async-timeout                    4.0.3
attrs                            23.2.0
backoff                          2.2.1
bcrypt                           4.1.2
cachetools                       5.3.2
certifi                          2024.2.2
certipy                          0.1.3
cffi                             1.16.0
charset-normalizer               3.3.2
cryptography                     42.0.3
escapism                         1.0.1
frozenlist                       1.4.1
google-api-core                  2.17.1
google-api-python-client         2.118.0
google-auth                      2.28.0
google-auth-httplib2             0.2.0
google-auth-oauthlib             1.2.0
googleapis-common-protos         1.62.0
greenlet                         3.0.3
httplib2                         0.22.0
idna                             3.6
Jinja2                           3.1.3
jsonschema                       4.21.1
jsonschema-specifications        2023.12.1
jupyter-telemetry                0.1.0
jupyterhub                       4.0.2
jupyterhub-firstuseauthenticator 1.0.0
jupyterhub-idle-culler           1.2.1
jupyterhub-ldapauthenticator     1.3.2
jupyterhub-nativeauthenticator   1.2.0
jupyterhub-systemdspawner        1.0.1
jupyterhub-tmpauthenticator      1.0.0
jupyterhub-traefik-proxy         1.1.0
ldap3                            2.9.1
Mako                             1.3.2
MarkupSafe                       2.1.5
multidict                        6.0.5
oauthenticator                   16.2.1
oauthlib                         3.2.2
onetimepass                      1.0.1
packaging                        23.2
pamela                           1.1.0
passlib                          1.7.4
pip                              24.0
pluggy                           1.4.0
prometheus_client                0.20.0
protobuf                         4.25.3
pyasn1                           0.5.1
pyasn1-modules                   0.3.0
pycparser                        2.21
pycurl                           7.45.3
pyOpenSSL                        24.0.0
pyparsing                        3.1.1
python-dateutil                  2.8.2
python-json-logger               2.0.7
referencing                      0.33.0
requests                         2.31.0
requests-oauthlib                1.3.1
rpds-py                          0.18.0
rsa                              4.9
ruamel.yaml                      0.17.40
ruamel.yaml.clib                 0.2.8
setuptools                       59.6.0
six                              1.16.0
SQLAlchemy                       2.0.27
the-littlest-jupyterhub          1.0.0
toml                             0.10.2
tornado                          6.4
traitlets                        5.14.1
typing_extensions                4.9.0
uritemplate                      4.1.1
urllib3                          2.2.1
yarl                             1.9.4

Logs

Error logs:

Feb 19 03:02:55 $VM_NAME python3[9285]: [E 2024-02-19 03:02:55.548 JupyterHub oauth2:626] Error fetching 599 POST https://www.googleapis.com/oauth2/v4/token: HTTP 599: error setting certificate verify locations:
Feb 19 03:02:55 $VM_NAME python3[9285]:       CAfile: /etc/pki/tls/certs/ca-bundle.crt
Feb 19 03:02:55 $VM_NAME python3[9285]:       CApath: none
...
...
Feb 19 03:02:55 $VM_NAME python3[9285]:     Traceback (most recent call last):
Feb 19 03:02:55 $VM_NAME python3[9285]:       File "/opt/tljh/hub/lib/python3.10/site-packages/tornado/web.py", line 1790, in _execute
Feb 19 03:02:55 $VM_NAME python3[9285]:         result = await result
Feb 19 03:02:55 $VM_NAME python3[9285]:       File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 208, in get
Feb 19 03:02:55 $VM_NAME python3[9285]:         user = await self.login_user()
Feb 19 03:02:55 $VM_NAME python3[9285]:       File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 828, in login_user
Feb 19 03:02:55 $VM_NAME python3[9285]:         authenticated = await self.authenticate(data)
Feb 19 03:02:55 $VM_NAME python3[9285]:       File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/auth.py", line 492, in get_authenticated_user
Feb 19 03:02:55 $VM_NAME python3[9285]:         authenticated = await maybe_future(self.authenticate(handler, data))
Feb 19 03:02:55 $VM_NAME python3[9285]:       File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 969, in authenticate
Feb 19 03:02:55 $VM_NAME python3[9285]:         token_info = await self.get_token_info(handler, access_token_params)
Feb 19 03:02:55 $VM_NAME python3[9285]:       File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 843, in get_token_info
Feb 19 03:02:55 $VM_NAME python3[9285]:         token_info = await self.httpfetch(
Feb 19 03:02:55 $VM_NAME python3[9285]:       File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 661, in httpfetch
Feb 19 03:02:55 $VM_NAME python3[9285]:         return await self.fetch(
Feb 19 03:02:55 $VM_NAME python3[9285]:       File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 627, in fetch
Feb 19 03:02:55 $VM_NAME python3[9285]:         raise e
Feb 19 03:02:55 $VM_NAME python3[9285]:       File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 606, in fetch
Feb 19 03:02:55 $VM_NAME python3[9285]:         resp = await self.http_client.fetch(req, **kwargs)
Feb 19 03:02:55 $VM_NAME python3[9285]:     tornado.curl_httpclient.CurlError: HTTP 599: error setting certificate verify locations:
Feb 19 03:02:55 $VM_NAME python3[9285]:       CAfile: /etc/pki/tls/certs/ca-bundle.crt
Feb 19 03:02:55 $VM_NAME python3[9285]:       CApath: none
Feb 19 03:02:55 $VM_NAME python3[9285]:  
...
...
Feb 19 03:02:55 $VM_NAME python3[9285]: [E 2024-02-19 03:02:55.554 JupyterHub log:191] 500 GET /hub/oauth_callback?state=[secret]&code=[secret]&scope=email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=[secret]&hd=bc.edu&prompt=none (@$IP_ADDRESS) 15.03ms

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[bl-ey]

Confirming this is unrelated to TLJH/Google Auth - I had this error with Z2JH+Azure.

I would also like to confirm the resolution of pycurl downgrading from 7.45.3 -> 7.45.2 resolved this issue for me as well. I changed in my requirements.txt for jupyterhub to include pycurl==7.45.2.

Pycurl appears to be an indirect dependency (only used in certain cases/not installed by default) of jupyterhub itself, so the issue may be worth mentioning there for a dependency update to resolve.

I haven't been able to dig into what changed/actually broke on pycurl, but if someone can point to it it may be worth reporting in the pycurl github.

[consideRatio]

Tracked by pycurl/pycurl#834

[consideRatio]

Thank you for an amazing issue writeup @jrdnbradford!! And thanl you @pastram-i for connecting the dots on what this is about! Nice work!!

[jrdnbradford]

Awesome, I somehow missed that filed issue over at pycurl/pycurl#834.

Thanks for the assist!