Logging in fails with 403 after upgrading to v2.1.0

[nmaggioni]

Past Issues Searched

• I have searched open and closed issues to make sure that the bug has not yet been reported

Issue is a Bug Report

• This is a bug report and not a feature request, nor asking for self-hosted support

Using official Plausible Cloud hosting or self-hosting?

Self-hosting

Describe the bug

After upgrading from v2.0.0 I am having the same issue that was reported for v2.1.0-rc0: when trying to log in I get a 403 and no logs.

As already mentioned there by others, I can reproduce the problem by reverting back to v2.0.0 and then switching again to the v2.1.0 image. Postgres and Clickhouse images have been upgraded (respectively to 16-alpine and 24.3.3.102-alpine), the TOTP_VAULT_KEY env var has been added, and SECRET_KEY_BASE is untouched.

Expected behavior

Login should be possible.

Screenshots

Environment

- OS: Linux
- Browser: Chromium
- Browser Version: 125.0.6422.60

[ruslandoga]

👋 @nmaggioni

Would you be able to verify that SECRET_KEY_BASE env var is the same in both containers (even if untouched in plausible-conf.env)? Also would you be able to decode the contents of the _plausible_key cookie? Please see #3762 (comment)

[nmaggioni]

Thanks, the contents of the cookie seem to be correct (the token matches [a-zA-Z0-9_]+ as usual):

%{
  "_csrf_token" => "<TOKEN_HERE>",
  "current_user_id" => 1,
  "last_seen" => 1716469476,
  "login_dest" => nil,
  "session_timeout_at" => 1717679076
}

This cookie was set by v2.0.0 and, I guess, ignored or rejected by v2.1.0 since I am prompted to login again. When using the new version deleting it and/or using a temporary browser profile still gets me a 403 on login and no new cookie is set.

The SECRET_KEY_BASE env var has indeed the same expected value in both images.

If that can be of interest I am deploying Plausible on Kubernetes and my manifests have been more or less the same since v1.4.4. The only changes made while upgrading to v2.1.0 were bumping PostgreSQL, ClickHouse and Plausible image versions to the recommended values and setting the new TOTP_VAULT_KEY env var.

[ruslandoga]

The cookie data seems OK. I guess we'll need to do some tracing similar to #2767 (reply in thread)

iex> :recon_trace.calls({PlausibleWeb.AuthPlug, :call, 2}, 10)

Do you get any traces if you execute this command and then visit the dashboard?

---

Also would you be able to share more information on your setup? Configs, manifests, proxies, env vars, logs, etc.

[ledermann]

The only changes made while upgrading to v2.1.0 were bumping PostgreSQL, [...] image versions to the recommended values [...]

@nmaggioni: Besides all other things: Bumping PostgreSQL to a new major version will not work. It requires a backup/restore. Check the logs of the PostgreSQL container.

[nmaggioni]

@ruslandoga Here are the traces, recorded using a fresh browser profile.

Visiting the root and being served the Welcome dialog

'Elixir.PlausibleWeb.AuthPlug':call(#{owner=><0.4407.0>, port=>INTERNAL_PORT_HERE, private=>#{'Elixir.PlausibleWeb.Router'=>[], plug_session_fetch=>done, plug_session=>#{}, before_send=>[#Fun<Elixir.Phoenix.Controller.4.96996861>,
 #Fun<Elixir.Plug.Session.0.76384852>,#Fun<Elixir.Plug.Telemetry.0.54455629>], phoenix_endpoint=>'Elixir.PlausibleWeb.Endpoint', phoenix_format=><<"html">>, phoenix_router=>'Elixir.PlausibleWeb.Router', robots=><<"noindex, nofollow">>}, scheme=>http, status=>nil, script_name=>[], state=>unset, host=><<"INTERNAL_HOSTNAME_HERE">>, cookies=>#{}, params=>#{}, '__struct__'=>'Elixir.Plug.Conn', halted=>false, assigns=>#{flash=>#{}}, method=><<"GET">>, path_info=>[], adapter=>{'Elixir.Plug.Cowboy.Conn',#{pid => <0.4406.0>,port => INTERNAL_PORT_HERE,
                             scheme => <<"http">>,version => 'HTTP/1.1',
                             path => <<"/">>,host => <<"INTERNAL_HOSTNAME_HERE">>,
                             peer => {{CLIENT_IP_HERE},44704},
                             sock => {{10,42,1,88},8000},
                             bindings => #{},
                             ref => 'Elixir.PlausibleWeb.Endpoint.HTTP',
                             cert => undefined,
                             headers =>
                                 #{<<"accept">> =>
                                       <<"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7">>,
                                   <<"accept-encoding">> =>
                                       <<"gzip, deflate">>,
                                   <<"accept-language">> =>
                                       <<"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7">>,
                                   <<"connection">> => <<"keep-alive">>,
                                   <<"host">> => <<"INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>,
                                   <<"upgrade-insecure-requests">> => <<"1">>,
                                   <<"user-agent">> =>
                                       <<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36">>},
                             method => <<"GET">>,qs => <<>>,
                             path_info => undefined,body_length => 0,
                             has_body => false,streamid => 1,
                             host_info => undefined}}, secret_key_base=><<"SECRET_KEY_BASE_ENV_VAR_HERE">>, req_headers=>[{<<"accept">>,
  <<"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7">>},
 {<<"accept-encoding">>,<<"gzip, deflate">>},
 {<<"accept-language">>,<<"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7">>},
 {<<"connection">>,<<"keep-alive">>},
 {<<"host">>,<<"INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>},
 {<<"upgrade-insecure-requests">>,<<"1">>},
 {<<"user-agent">>,
  <<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36">>}], remote_ip=>{CLIENT_IP_HERE}, body_params=>#{}, query_params=>#{}, query_string=><<>>, request_path=><<"/">>, path_params=>#{}, req_cookies=>#{}, resp_body=>nil, resp_cookies=>#{}, resp_headers=>[{<<"cache-control">>,<<"max-age=0, private, must-revalidate">>},
 {<<"x-request-id">>,<<"F9IvVG98mZw3SeEAACVh">>},
 {<<"access-control-allow-credentials">>,<<"true">>},
 {<<"access-control-allow-origin">>,<<"*">>},
 {<<"access-control-expose-headers">>,<<>>},
 {<<"referrer-policy">>,<<"strict-origin-when-cross-origin">>},
 {<<"x-content-type-options">>,<<"nosniff">>},
 {<<"x-download-options">>,<<"noopen">>},
 {<<"x-frame-options">>,<<"SAMEORIGIN">>},
 {<<"x-permitted-cross-domain-policies">>,<<"none">>},
 {<<"x-robots-tag">>,<<"noindex, nofollow">>}]}, [])

Visiting the login page

'Elixir.PlausibleWeb.AuthPlug':call(#{owner=><0.4416.0>, port=>INTERNAL_PORT_HERE, private=>#{'Elixir.PlausibleWeb.Router'=>[], plug_session_fetch=>done, plug_session=>#{}, before_send=>[#Fun<Elixir.Phoenix.Controller.4.96996861>,
 #Fun<Elixir.Plug.Session.0.76384852>,#Fun<Elixir.Plug.Telemetry.0.54455629>], phoenix_endpoint=>'Elixir.PlausibleWeb.Endpoint', phoenix_format=><<"html">>, phoenix_router=>'Elixir.PlausibleWeb.Router', robots=><<"noindex, nofollow">>}, scheme=>http, status=>nil, script_name=>[], state=>unset, host=><<"INTERNAL_HOSTNAME_HERE">>, cookies=>#{}, params=>#{}, '__struct__'=>'Elixir.Plug.Conn', halted=>false, assigns=>#{flash=>#{}}, method=><<"GET">>, path_info=>[<<"login">>], adapter=>{'Elixir.Plug.Cowboy.Conn',#{pid => <0.4406.0>,port => INTERNAL_PORT_HERE,
                             scheme => <<"http">>,version => 'HTTP/1.1',
                             path => <<"/login">>,
                             host => <<"INTERNAL_HOSTNAME_HERE">>,
                             peer => {{CLIENT_IP_HERE},44704},
                             sock => {{10,42,1,88},8000},
                             bindings => #{},
                             ref => 'Elixir.PlausibleWeb.Endpoint.HTTP',
                             cert => undefined,
                             headers =>
                                 #{<<"accept">> =>
                                       <<"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7">>,
                                   <<"accept-encoding">> =>
                                       <<"gzip, deflate">>,
                                   <<"accept-language">> =>
                                       <<"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7">>,
                                   <<"connection">> => <<"keep-alive">>,
                                   <<"host">> => <<"INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>,
                                   <<"referer">> =>
                                       <<"http://INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE/">>,
                                   <<"upgrade-insecure-requests">> => <<"1">>,
                                   <<"user-agent">> =>
                                       <<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36">>},
                             method => <<"GET">>,qs => <<>>,
                             path_info => undefined,body_length => 0,
                             has_body => false,streamid => 3,
                             host_info => undefined}}, secret_key_base=><<"SECRET_KEY_BASE_ENV_VAR_HERE">>, req_headers=>[{<<"accept">>,
  <<"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7">>},
 {<<"accept-encoding">>,<<"gzip, deflate">>},
 {<<"accept-language">>,<<"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7">>},
 {<<"connection">>,<<"keep-alive">>},
 {<<"host">>,<<"INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>},
 {<<"referer">>,<<"http://INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE/">>},
 {<<"upgrade-insecure-requests">>,<<"1">>},
 {<<"user-agent">>,
  <<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36">>}], remote_ip=>{CLIENT_IP_HERE}, body_params=>#{}, query_params=>#{}, query_string=><<>>, request_path=><<"/login">>, path_params=>#{}, req_cookies=>#{}, resp_body=>nil, resp_cookies=>#{}, resp_headers=>[{<<"cache-control">>,<<"max-age=0, private, must-revalidate">>},
 {<<"x-request-id">>,<<"F9IvXDuFqRl3FxYAACYh">>},
 {<<"access-control-allow-credentials">>,<<"true">>},
 {<<"access-control-allow-origin">>,<<"*">>},
 {<<"access-control-expose-headers">>,<<>>},
 {<<"referrer-policy">>,<<"strict-origin-when-cross-origin">>},
 {<<"x-content-type-options">>,<<"nosniff">>},
 {<<"x-download-options">>,<<"noopen">>},
 {<<"x-frame-options">>,<<"SAMEORIGIN">>},
 {<<"x-permitted-cross-domain-policies">>,<<"none">>},
 {<<"x-robots-tag">>,<<"noindex, nofollow">>}]}, [])

Logging in and receiving the customized 403 page

'Elixir.PlausibleWeb.AuthPlug':call(#{owner=><0.4427.0>, port=>INTERNAL_PORT_HERE, private=>#{'Elixir.PlausibleWeb.Router'=>[], plug_session_fetch=>done, plug_session=>#{}, before_send=>[#Fun<Elixir.Phoenix.Controller.4.96996861>,
 #Fun<Elixir.Plug.Session.0.76384852>,#Fun<Elixir.Plug.Telemetry.0.54455629>], phoenix_endpoint=>'Elixir.PlausibleWeb.Endpoint', phoenix_format=><<"html">>, phoenix_router=>'Elixir.PlausibleWeb.Router', robots=><<"noindex, nofollow">>}, scheme=>http, status=>nil, script_name=>[], state=>unset, host=><<"INTERNAL_HOSTNAME_HERE">>, cookies=>#{}, params=>#{<<"_csrf_token">>=><<"DiN8JgMHIjwJPXUpGjoPZyAHFEQYE0wYAgNkaeHqsYGLQJe4WIFtwaxH">>, <<"email">>=><<"USERNAME_HERE">>, <<"password">>=><<"PASSWORD_HERE">>}, '__struct__'=>'Elixir.Plug.Conn', halted=>false, assigns=>#{flash=>#{}}, method=><<"POST">>, path_info=>[<<"login">>], adapter=>{'Elixir.Plug.Cowboy.Conn',#{pid => <0.4406.0>,port => INTERNAL_PORT_HERE,
                             scheme => <<"http">>,version => 'HTTP/1.1',
                             path => <<"/login">>,
                             host => <<"INTERNAL_HOSTNAME_HERE">>,
                             peer => {{CLIENT_IP_HERE},44704},
                             sock => {{10,42,1,88},8000},
                             bindings => #{},
                             ref => 'Elixir.PlausibleWeb.Endpoint.HTTP',
                             cert => undefined,
                             headers =>
                                 #{<<"accept">> =>
                                       <<"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7">>,
                                   <<"accept-encoding">> =>
                                       <<"gzip, deflate">>,
                                   <<"accept-language">> =>
                                       <<"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7">>,
                                   <<"cache-control">> => <<"max-age=0">>,
                                   <<"connection">> => <<"keep-alive">>,
                                   <<"content-length">> => <<"133">>,
                                   <<"content-type">> =>
                                       <<"application/x-www-form-urlencoded">>,
                                   <<"host">> => <<"INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>,
                                   <<"origin">> =>
                                       <<"http://INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>,
                                   <<"referer">> =>
                                       <<"http://INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE/login">>,
                                   <<"upgrade-insecure-requests">> => <<"1">>,
                                   <<"user-agent">> =>
                                       <<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36">>},
                             method => <<"POST">>,qs => <<>>,
                             path_info => undefined,body_length => 133,
                             has_body => true,streamid => 4,
                             host_info => undefined,has_read_body => true}}, secret_key_base=><<"SECRET_KEY_BASE_ENV_VAR_HERE">>, req_headers=>[{<<"accept">>,
  <<"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7">>},
 {<<"accept-encoding">>,<<"gzip, deflate">>},
 {<<"accept-language">>,<<"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7">>},
 {<<"cache-control">>,<<"max-age=0">>},
 {<<"connection">>,<<"keep-alive">>},
 {<<"content-length">>,<<"133">>},
 {<<"content-type">>,<<"application/x-www-form-urlencoded">>},
 {<<"host">>,<<"INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>},
 {<<"origin">>,<<"http://INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>},
 {<<"referer">>,<<"http://INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE/login">>},
 {<<"upgrade-insecure-requests">>,<<"1">>},
 {<<"user-agent">>,
  <<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36">>}], remote_ip=>{CLIENT_IP_HERE}, body_params=>#{<<"_csrf_token">>=><<"DiN8JgMHIjwJPXUpGjoPZyAHFEQYE0wYAgNkaeHqsYGLQJe4WIFtwaxH">>, <<"email">>=><<"USERNAME_HERE">>, <<"password">>=><<"PASSWORD_HERE">>}, query_params=>#{}, query_string=><<>>, request_path=><<"/login">>, path_params=>#{}, req_cookies=>#{}, resp_body=>nil, resp_cookies=>#{}, resp_headers=>[{<<"cache-control">>,<<"max-age=0, private, must-revalidate">>},
 {<<"x-request-id">>,<<"F9IvZhFTQP3eEl8AACdh">>},
 {<<"access-control-allow-credentials">>,<<"true">>},
 {<<"access-control-allow-origin">>,<<"*">>},
 {<<"access-control-expose-headers">>,<<>>},
 {<<"referrer-policy">>,<<"strict-origin-when-cross-origin">>},
 {<<"x-content-type-options">>,<<"nosniff">>},
 {<<"x-download-options">>,<<"noopen">>},
 {<<"x-frame-options">>,<<"SAMEORIGIN">>},
 {<<"x-permitted-cross-domain-policies">>,<<"none">>},
 {<<"x-robots-tag">>,<<"noindex, nofollow">>}]}, [])

Navigating back the root and being served the Welcome dialog once again

'Elixir.PlausibleWeb.AuthPlug':call(#{owner=><0.4403.0>, port=>INTERNAL_PORT_HERE, private=>#{'Elixir.PlausibleWeb.Router'=>[], plug_session_fetch=>done, plug_session=>#{}, before_send=>[#Fun<Elixir.Phoenix.Controller.4.96996861>,
 #Fun<Elixir.Plug.Session.0.76384852>,#Fun<Elixir.Plug.Telemetry.0.54455629>], phoenix_endpoint=>'Elixir.PlausibleWeb.Endpoint', phoenix_format=><<"html">>, phoenix_router=>'Elixir.PlausibleWeb.Router', robots=><<"noindex, nofollow">>}, scheme=>http, status=>nil, script_name=>[], state=>unset, host=><<"INTERNAL_HOSTNAME_HERE">>, cookies=>#{}, params=>#{}, '__struct__'=>'Elixir.Plug.Conn', halted=>false, assigns=>#{flash=>#{}}, method=><<"GET">>, path_info=>[], req_headers=>[{<<"accept">>,
  <<"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7">>},
 {<<"accept-encoding">>,<<"gzip, deflate">>},
 {<<"accept-language">>,<<"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7">>},
 {<<"connection">>,<<"keep-alive">>},
 {<<"host">>,<<"INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>},
 {<<"upgrade-insecure-requests">>,<<"1">>},
 {<<"user-agent">>,
  <<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36">>}], remote_ip=>{CLIENT_IP_HERE}, adapter=>{'Elixir.Plug.Cowboy.Conn',#{pid => <0.4387.0>,port => INTERNAL_PORT_HERE,
                             scheme => <<"http">>,version => 'HTTP/1.1',
                             path => <<"/">>,host => <<"INTERNAL_HOSTNAME_HERE">>,
                             peer => {{CLIENT_IP_HERE},48096},
                             sock => {{10,42,1,90},8000},
                             bindings => #{},
                             ref => 'Elixir.PlausibleWeb.Endpoint.HTTP',
                             cert => undefined,
                             headers =>
                                 #{<<"accept">> =>
                                       <<"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7">>,
                                   <<"accept-encoding">> =>
                                       <<"gzip, deflate">>,
                                   <<"accept-language">> =>
                                       <<"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7">>,
                                   <<"connection">> => <<"keep-alive">>,
                                   <<"host">> => <<"INTERNAL_HOSTNAME_HERE:INTERNAL_PORT_HERE">>,
                                   <<"upgrade-insecure-requests">> => <<"1">>,
                                   <<"user-agent">> =>
                                       <<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36">>},
                             method => <<"GET">>,qs => <<>>,
                             path_info => undefined,streamid => 7,
                             body_length => 0,has_body => false,
                             host_info => undefined}}, body_params=>#{}, query_params=>#{}, query_string=><<>>, request_path=><<"/">>, secret_key_base=><<"SECRET_KEY_BASE_ENV_VAR_HERE">>, resp_body=>nil, resp_cookies=>#{}, resp_headers=>[{<<"cache-control">>,<<"max-age=0, private, must-revalidate">>},
 {<<"x-request-id">>,<<"F9Iw2qjk2SXhwd8AACYB">>},
 {<<"access-control-allow-credentials">>,<<"true">>},
 {<<"access-control-allow-origin">>,<<"*">>},
 {<<"access-control-expose-headers">>,<<>>},
 {<<"referrer-policy">>,<<"strict-origin-when-cross-origin">>},
 {<<"x-content-type-options">>,<<"nosniff">>},
 {<<"x-download-options">>,<<"noopen">>},
 {<<"x-frame-options">>,<<"SAMEORIGIN">>},
 {<<"x-permitted-cross-domain-policies">>,<<"none">>},
 {<<"x-robots-tag">>,<<"noindex, nofollow">>}], req_cookies=>#{}, path_params=>#{}}, [])

By enabling query logging on Postgres through the -c log_statement=all flag I saw that (on top of the background noise generated by Oban jobs) only a single query was being made when serving each of the above requests: SELECT TRUE FROM "users" AS u0 LIMIT 1. Running this query manually returns t as expected.

I have no Elixir knowledge but while I was at it I have also tried tracing other Plugs out of curiosity and, while FirstLaunchPlug, LastSeenPlug, NoRobots, SessionTimeoutPlug, Tracker produced some output when POSTing the login data, none of them seemed to log anything that caught my eye.

Here are the relevant portions of my K8S manifest:

K8S deployment

# PostgreSQL
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: plausible
spec:
  type: ClusterIP
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432
      protocol: TCP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: plausible
spec:
  replicas: 1
  serviceName: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          env: # Auth vars
      volumes:
        - name: data
          hostPath:
            path: # Unix path
            type: Directory

# Plausible
---
apiVersion: v1
kind: Secret
metadata:
  name: plausible
  namespace: plausible
type: Opaque
data:
  secretKeyBase: # SECRET_KEY_BASE value re-encoded to base64
  totpVaultKey: # TOTP_VAULT_KEY value re-encoded to base64
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: plausible
  namespace: plausible
spec:
  replicas: 1
    spec:
      initContainers:
        - name: wait-postgres
          image: docker.io/library/alpine:3.17
          command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 postgres 5432 && exit 0 || sleep 3; done; exit 1"]
        - name: wait-clickhouse
          image: docker.io/library/alpine:3.17
          command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 clickhouse 8123 && exit 0 || sleep 3; done; exit 1"]
      containers:
        - name: plausible
          image: plausible/analytics:v2.0.0  # ghcr.io/plausible/community-edition:v2.1.0
          env:
            - name: BASE_URL
              value: # EXTERNAL URL
            - name: SECRET_KEY_BASE
              valueFrom:
                secretKeyRef:
                  name: plausible
                  key: secretKeyBase
                  optional: false
            - name: TOTP_VAULT_KEY
              valueFrom:
                secretKeyRef:
                  name: plausible
                  key: totpVaultKey
                  optional: false
            - name: GEOLITE2_COUNTRY_DB
              value: /geoip/GeoLite2-Country.mmdb
            - name: IP_GEOLOCATION_DB
              value: /geoip/GeoLite2-City.mmdb
            - name: GEONAMES_SOURCE_FILE
              value: /geoip/geonames.lite.csv
            - name: DISABLE_REGISTRATION
              value: "true"
            - name: MAILER_EMAIL
              value: # Sender address
            - name: CRON_ENABLED
              value: "true"
            - name: DATABASE_URL
              value: "postgres://<USERNAME>:<PASSWORD>@postgres:5432/plausible_db"
            - name: CLICKHOUSE_DATABASE_URL
              value: "http://clickhouse:8123/plausible_events_db"
            - name: SMTP_HOST_ADDR
              value: "mailer"
            - name: SMTP_HOST_PORT
              value: "25"
          ports:
            - protocol: TCP
              containerPort: 8000
              hostPort: # Host port
          livenessProbe:
            httpGet:
              path: /api/health
              port: 8000
          readinessProbe:
            httpGet:
              path: /api/health
              port: 8000
          command: ["/bin/sh"]
          args: ["-c", "/entrypoint.sh db createdb && /entrypoint.sh db migrate && /entrypoint.sh run"]
          volumeMounts:
            - mountPath: /geoip
              name: geoip
              readOnly: true
      volumes:
        - name: geoip
          hostPath:
            path: # Unix path
            type: Directory

For these tests I am accessing the instance locally through its exposed port.

---

@ledermann My bad, I took that for granted and did not elaborate for the sake of brevity. The upgrade was indeed done as recommended and Postgres shows no errors at startup or when trying to log in. ClickHouse seems happy too, just in case.

[nunombispo]

Not sure if it helps, but in my case I managed to log in successfully after changing the BASE_URL.

Since I am using Plausible behind Nginx, I had the BASE_URL defined to the local IP address. Changing this to the real domain name allowed me to log in.

But now I get a websocket error and on the dashboard, I don't see graphs but entering the details of a site all is correct:

[ruslandoga]

👋 @nunombispo you probably need to enable websockets in nginx: https://github.com/plausible/community-edition/blob/c5f3386338057878648fd30af9c0462d6de78425/reverse-proxy/nginx/plausible#L12-L17

[ruslandoga]

@nmaggioni the traces show that there are no cookies being passed into the requests. Please check that your BASE_URL is the same as the cookie domain and please share your reverse-proxy setup. I've tried to reproduce the problem by starting docker compose with plausible:v2.0.0, logging in, and then upgrading to v2.1.0 but it all works as expected. So it seems to be a configuration issue.

[nmaggioni]

@ruslandoga Sorry for not proving enough info from the start, I am trying to pinpoint the issue as we go. I am running a split-horizon setup, in which the necessary tracking scripts are accessible from public networks while the management interface itself is not; when clients request the tracking snippet they ultimately hit a Caddy proxy configured as such:

@tracking_paths path /js/plausible.js /api/event /api/error

handle @tracking_paths {
  reverse_proxy <INTERNAL_HOST>:<INTERNAL_PORT> {
    header_down Access-Control-Allow-Origin [...]
  }
}

handle {
  abort
}

Basically all Caddy is doing here is passing on the requests to /js/plausible.js, /api/event and /api/error to the actual Plausible instance, adding a necessary Access-Control-Allow-Origin header on the reply, and denying everything else. When I need to check the tracking stats I head to <INTERNAL_HOST>:<INTERNAL_PORT> from the local network and do my business.

This flow has been working for quite some time and a behavioral change in v2.1.0 has broken it, so I don't think this is a configuration issue. Obligatory xkcd, I know.

Again, I'm no Elixir expert, but could the issue stem from the CSRF tokens being tied to the session and in turn it now being tied to the BASE_URL instead of the domain the login request is received at? Could this commit explain the new behavior? Am I relying on an edge case of Plug.Session's default configuration?
I have spun up a local development environment to confirm this suspicion and indeed, with BASE_URL=http://localhost:8000 (and LISTEN_IP=0.0.0.0 since I'm juggling development containers) I can login just fine through localhost but CSRF validation fails when I try to do the same through my machine's IP address. The port is 8888 instead of 8000 due to the devcontainer's port forwarding.

If I login through localhost, copy the session cookie and manually add it in the browser for the bare IP "domain" too, I can get to the dashboard without triggering CSRF errors. It then breaks because the session is not valid and the WS connection can't be stabilished, but that's obvious. Am I on the right track, could the (I'm guessing here!) new BASE_URL usage be the curlprit?

[ruslandoga]

@nmaggioni thank you for a comprehensive reply! I'll try reproducing this later following your split setup.

I have one question though, do sessions work if BASE_URL is set to the machine's IP address + 8888 port? The listening port can be configured separately via PORT env var.

[nunombispo]

👋 @nunombispo you probably need to enable websockets in nginx: https://github.com/plausible/community-edition/blob/c5f3386338057878648fd30af9c0462d6de78425/reverse-proxy/nginx/plausible#L12-L17

@ruslandoga thanks for the info. Since I am using Nginx proxy manager it was just setting a flag for websocket support.

So for me, this worked (behind Nginx):

• Setting the BASE_URL for the domain and not the internal IP address
• Enabling websocket support

[nmaggioni]

do sessions work if BASE_URL is set to the machine's IP address + 8888 port? The listening port can be configured separately via PORT env var.

@ruslandoga They indeed do and connections through other hostnames/IPs break as expected. The port is not taken into account as far as I can see.

I'm struggling to grasp why my setup was/is working at all then: I suppose that commenting this line is the equivalent to minimally reverting the commit I mentioned above in the current codebase and doing that in fact solves the CSRF errors (but not the WS ones), but that doesn't explain much.

[ruslandoga]

They indeed do and connections through other hostnames/IPs break as expected.

@nmaggioni So it works? Or is there a problem still?

I've tried running a CE build locally with BASE_URL=http://127.0.0.1:8000 to simulate your internal URL and login (forms), sites (websockets), dashboards (ajax), etc. seem to work fine.

Posting events to an "external" URL like http://localhost:8000/api/event also works and the events show up on the dashboard. http://localhost:8000/js/plausible.js works too.

[ZackBoe]

I'm seeing this when attempting to login via multiple domains. Events are still coming in on both domains, but I can only login from whichever is set as the BASE_URL.