HTTP Error 500 for /accounts/login/, resolved by setting DEBUG=1

[rjhenry]

Tandoor Version

2.0.0-beta-2

Setup

Kubernetes

Reverse Proxy

Others (please state below)

Other

ingress-nginx

Bug description

After updating to beta-2, my liveness and readiness probes (HTTP request to /) are failing.

In the logs of the recipes container, I can see the kube-probe request coming in, hitting /, and being redirected to /accounts/login, which then gives a HTTP 500 error (which fails the liveness probe). Port-forwarding to the recipes container and querying / gives the same result.
Interestingly, running the same Deployment manifest with the env-var DEBUG set to "1" results in the query for /accounts/login succeeding. There's nothing immediately apparent in the output of the recipes container. Changing the value of --log-level passed to the gunicorn command makes no difference to the behaviour of the probes.

Deployment spec looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: recipes
  labels:
    app: recipes
    environment: production
    tier: frontend
spec:
  replicas: 2
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: recipes
      environment: production
  template:
    metadata:
      labels:
        app: recipes
        tier: frontend
        environment: production
    spec:
      restartPolicy: Always
      serviceAccount: recipes
      serviceAccountName: recipes
      initContainers:
        - name: init-chmod-data
          # Configure options for S3
          envFrom:
            - secretRef:
                name: tandoor-s3-storage
          env:
            - name: DB_ENGINE
              value: django.db.backends.postgresql
            - name: ENABLE_PDF_EXPORT
              value: "1"
            - name: POSTGRES_HOST
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: host
            - name: POSTGRES_PORT
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: port
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: password
            - name: POSTGRES_DB
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: dbname
            # Actually set from tandoor-s3-storage, because abusing envFrom is
            # easier than setting up a dedicated secret.
            # - name: SECRET_KEY
            - name: SOCIAL_DEFAULT_ACCESS
              value: "1"
            - name: SOCIAL_DEFAULT_GROUP
              value: users
            - name: SOCIAL_PROVIDERS
              value: allauth.socialaccount.providers.openid_connect
            - name: SOCIALACCOUNT_PROVIDERS
              valueFrom:
                secretKeyRef:
                  name: tandoor-keycloak
                  key: provider-config
            - name: TZ
              value: Europe/London
          # yamllint disable rule:comments rule:line-length
          # Rules are disabled thanks to how Flux applies changes; the space
          # is stripped between the end of the "real" line and the imagepolicy
          # tag.
          # !!! DO NOT UPDATE - MANAGED BY FLUX !!!
          image: ghcr.io/tandoorrecipes/recipes:2.0.0-beta-2 # {"$imagepolicy": "tandoor:tandoor"}
          # yamllint enable rule:comments rule:line-length
          resources:
            requests:
              cpu: 250m
              memory: 64Mi
          command:
            - sh
            - -c
            - |
              set -e
              source venv/bin/activate
              echo "Updating database"
              python manage.py migrate
              python manage.py collectstatic --noinput
              echo "Done"
          securityContext:
            runAsUser: 0
          volumeMounts:
            - mountPath: /opt/recipes/staticfiles
              name: static
              # mount as subPath due to lost+found on ext4 pvc
              subPath: files
      containers:
        - name: recipes-nginx
          image: nginx:mainline
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
              protocol: TCP
              name: http
            - containerPort: 8080
              protocol: TCP
              name: gunicorn
          resources:
            requests:
              cpu: 250m
              memory: 64Mi
          volumeMounts:
            - mountPath: /static
              name: static
              # mount as subPath due to lost+found on ext4 pvc
              subPath: files
            - name: nginx-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx-config
              readOnly: true
        - name: recipes
          image: ghcr.io/tandoorrecipes/recipes:2.0.0-beta-2
          imagePullPolicy: IfNotPresent
          command:
            - /opt/recipes/venv/bin/gunicorn
            - -b
            - :8080
            - --access-logfile
            - "-"
            - --error-logfile
            - "-"
            - --log-level
            - DEBUG
            - recipes.wsgi
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /
              port: 8080
              scheme: HTTP
            periodSeconds: 30
            timeoutSeconds: 5
          readinessProbe:
            httpGet:
              path: /
              port: 8080
              scheme: HTTP
            periodSeconds: 30
            timeoutSeconds: 5
          resources:
            requests:
              cpu: 250m
              memory: 128Mi
          volumeMounts:
            - mountPath: /opt/recipes/staticfiles
              name: static
              # mount as subPath due to lost+found on ext4 pvc
              subPath: files
          # Configure options for S3
          envFrom:
            - secretRef:
                name: tandoor-s3-storage
          env:
            - name: ALLOWED_HOSTS
              value: '*'
            - name: DEBUG
              value: "0"
            - name: ENABLE_METRICS
              value: "1"
            - name: ENABLE_SIGNUP
              value: "1"
            - name: GUNICORN_MEDIA
              value: "0"
            - name: DB_ENGINE
              value: django.db.backends.postgresql
            - name: ENABLE_PDF_EXPORT
              value: "1"
            - name: POSTGRES_HOST
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: host
            - name: POSTGRES_PORT
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: port
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: password
            # Actually set from tandoor-s3-storage, because abusing envFrom is
            # easier than setting up a dedicated secret.
            # - name: SECRET_KEY
            - name: POSTGRES_DB
              valueFrom:
                secretKeyRef:
                  name: tandoor-pg-app
                  key: dbname
            # Actually set from tandoor-s3-storage, because abusing envFrom is
            # easier than setting up a dedicated secret.
            # - name: SECRET_KEY
            - name: SOCIAL_DEFAULT_ACCESS
              value: "1"
            - name: SOCIAL_DEFAULT_GROUP
              value: users
            - name: SOCIAL_PROVIDERS
              value: allauth.socialaccount.providers.openid_connect
            - name: SOCIALACCOUNT_PROVIDERS
              valueFrom:
                secretKeyRef:
                  name: tandoor-keycloak
                  key: provider-config
            - name: TZ
              value: Europe/London
          securityContext:
            runAsUser: 65534
      volumes:
        - name: static
          persistentVolumeClaim:
            claimName: recipes-static
        - name: nginx-config
          configMap:
            name: recipes-nginx-config

Relevant logs

Non-debug:

[2025-07-07 13:33:50 +0100] [7] [DEBUG] GET /
172.16.0.1 - - [07/Jul/2025:13:33:50 +0100] "GET / HTTP/1.1" 302 0 "-" "kube-probe/1.33"
[2025-07-07 13:33:50 +0100] [7] [DEBUG] GET /accounts/login/
172.16.0.1 - - [07/Jul/2025:13:33:50 +0100] "GET /accounts/login/?next=/ HTTP/1.1" 500 145 "http://172.16.0.107:8080/" "kube-probe/1.33"
[2025-07-07 13:34:00 +0100] [7] [DEBUG] GET /
172.16.0.1 - - [07/Jul/2025:13:34:00 +0100] "GET / HTTP/1.1" 302 0 "-" "kube-probe/1.33"
[2025-07-07 13:34:00 +0100] [7] [DEBUG] GET /accounts/login/
172.16.0.1 - - [07/Jul/2025:13:34:00 +0100] "GET /accounts/login/?next=/ HTTP/1.1" 500 145 "http://172.16.0.107:8080/" "kube-probe/1.33"


Debug:

[2025-07-07 13:35:33 +0100] [7] [DEBUG] GET /
172.16.0.1 - - [07/Jul/2025:13:35:33 +0100] "GET / HTTP/1.1" 302 0 "-" "kube-probe/1.33"
[2025-07-07 13:35:33 +0100] [7] [DEBUG] GET /accounts/login/
172.16.0.1 - - [07/Jul/2025:13:35:33 +0100] "GET /accounts/login/?next=/ HTTP/1.1" 200 6229 "http://172.16.0.111:8080/" "kube-probe/1.33"
[2025-07-07 13:35:36 +0100] [7] [DEBUG] GET /
172.16.0.1 - - [07/Jul/2025:13:35:36 +0100] "GET / HTTP/1.1" 302 0 "-" "kube-probe/1.33"
[2025-07-07 13:35:36 +0100] [7] [DEBUG] GET /accounts/login/
172.16.0.1 - - [07/Jul/2025:13:35:36 +0100] "GET /accounts/login/?next=/ HTTP/1.1" 200 6229 "http://172.16.0.111:8080/" "kube-probe/1.33"

[vabene1111]

fixed

[rjhenry]

Sorry to pester, but using beta-3 has the same result. Enabling DEBUG fixes the issue, and non-debug instances return HTTP/500s on /accounts/login/. Logs look near identical:

172.16.0.1 - - [08/Jul/2025:17:43:45 +0100] "GET / HTTP/1.1" 302 0 "-" "kube-probe/1.33"
172.16.0.1 - - [08/Jul/2025:17:43:45 +0100] "GET / HTTP/1.1" 302 0 "-" "kube-probe/1.33"
172.16.0.1 - - [08/Jul/2025:17:43:45 +0100] "GET /accounts/login/?next=/ HTTP/1.1" 500 145 "http://172.16.0.139:8080/" "kube-probe/1.33"
172.16.0.183 - - [08/Jul/2025:17:43:59 +0100] "GET /metrics HTTP/1.1" 200 30096 "-" "Prometheus/3.4.2"
172.16.0.1 - - [08/Jul/2025:17:44:01 +0100] "GET / HTTP/1.1" 302 0 "-" "kube-probe/1.33"
172.16.0.1 - - [08/Jul/2025:17:44:01 +0100] "GET /accounts/login/?next=/ HTTP/1.1" 500 145 "http://172.16.0.139:8080/" "kube-probe/1.33"
172.16.0.1 - - [08/Jul/2025:17:44:11 +0100] "GET / HTTP/1.1" 302 0 "-" "kube-probe/1.33"
172.16.0.1 - - [08/Jul/2025:17:44:11 +0100] "GET /accounts/login/?next=/ HTTP/1.1" 500 145 "http://172.16.0.139:8080/" "kube-probe/1.33"

[vabene1111]

interesting, I tought I tested on my instance but maybe there is another issue. Will look into it

[rjhenry]

I stood up a quick Docker Compose stack to make sure it's not a quirk of my environment with the following config files and have the same issue, so I don't think it's just me. The difference I'm noting here is that / redirects to /setup/ rather than /accounts/login/ and still fails with the HTTP/500 error. As before, setting the env var DEBUG to a value of '1' and the redirect works.

Logs:

web_recipes-1    | Running django-vite in production mode (no HMR)
nginx_recipes-1  | 192.168.65.1 - - [10/Jul/2025:07:32:31 +0100] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0" "-"
web_recipes-1    | ::ffff:172.18.0.4 - - [10/Jul/2025:07:32:31 +0100] "GET / HTTP/1.0" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0"
nginx_recipes-1  | 192.168.65.1 - - [10/Jul/2025:07:32:32 +0100] "GET /setup/ HTTP/1.1" 500 145 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0" "-"
web_recipes-1    | ::ffff:172.18.0.4 - - [10/Jul/2025:07:32:32 +0100] "GET /setup/ HTTP/1.0" 500 145 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0"
web_recipes-1    | ::ffff:172.18.0.4 - - [10/Jul/2025:07:32:35 +0100] "GET /setup/ HTTP/1.0" 500 145 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0"
nginx_recipes-1  | 192.168.65.1 - - [10/Jul/2025:07:32:35 +0100] "GET /setup/ HTTP/1.1" 500 145 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0" "-"
nginx_recipes-1  | 192.168.65.1 - - [10/Jul/2025:07:33:47 +0100] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0" "-"
web_recipes-1    | ::ffff:172.18.0.4 - - [10/Jul/2025:07:33:47 +0100] "GET / HTTP/1.0" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0"
nginx_recipes-1  | 192.168.65.1 - - [10/Jul/2025:07:33:47 +0100] "GET /setup/ HTTP/1.1" 500 145 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0" "-"
web_recipes-1    | ::ffff:172.18.0.4 - - [10/Jul/2025:07:33:47 +0100] "GET /setup/ HTTP/1.0" 500 145 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0"
nginx_recipes-1  | 192.168.65.1 - - [10/Jul/2025:07:33:48 +0100] "GET /favicon.ico HTTP/1.1" 499 0 "http://localhost:8080/setup/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0" "-"
web_recipes-1    | ::ffff:172.18.0.4 - - [10/Jul/2025:07:33:48 +0100] "GET /favicon.ico HTTP/1.0" 302 0 "http://localhost:8080/setup/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0"

.env:

SECRET_KEY=12345678901234567890123456789012345678901234567890
TZ=Europe/London
DB_ENGINE=django.db.backends.postgresql
POSTGRES_HOST=db_recipes
POSTGRES_DB=djangodb
POSTGRES_PORT=5432
POSTGRES_USER=djangouser
POSTGRES_PASSWORD=supersecret

docker-compose.yml:

services:
  db_recipes:
    restart: always
    image: postgres:16-alpine
    platform: linux/amd64
    volumes:
      - ./postgresql:/var/lib/postgresql/data
    env_file:
      - ./.env

  web_recipes:
    restart: always
    image: ghcr.io/tandoorrecipes/recipes:2.0.0-beta-3
    platform: linux/amd64
    env_file:
      - ./.env
    volumes:
      - staticfiles:/opt/recipes/staticfiles
      # Do not make this a bind mount, see https://docs.tandoor.dev/install/docker/#volumes-vs-bind-mounts
      - nginx_config:/opt/recipes/nginx/conf.d
      - ./mediafiles:/opt/recipes/mediafiles
    depends_on:
      - db_recipes

  nginx_recipes:
    image: nginx:mainline-alpine
    restart: always
    ports:
      - 8080:80
    env_file:
      - ./.env
    depends_on:
      - web_recipes
    volumes:
      # Do not make this a bind mount, see https://docs.tandoor.dev/install/docker/#volumes-vs-bind-mounts
      - nginx_config:/etc/nginx/conf.d:ro
      - staticfiles:/static:ro
      - ./mediafiles:/media:ro

volumes:
  nginx_config:
  staticfiles:

[vabene1111]

thanks for the feedback, i think i fixed it for real now 🙈