Envoy tls_inspector: parseClientHello failed (WRONG_VERSION_NUMBER) for PostgreSQL clients using native SSLRequest; Gateway TLS listener rejects connection (no matching filter chain)

[AnindyaKrDas]

Issue Description

When an Istio Gateway is configured with protocol: TLS and tls.mode: SIMPLE (TLS termination + re-encrypt), Envoy’s tls_inspector rejects PostgreSQL client connections that use PostgreSQL’s native SSL negotiation (the SSLRequest preamble) because the first bytes are not a TLS ClientHello. Envoy logs tls_inspector: parseClientHello failed: ... WRONG_VERSION_NUMBER and closing connection ... no matching filter chain found. openssl s_client works (it sends a TLS ClientHello), and Envoy -> Postgres upstream TLS works, so the control plane, secrets and certificates are fine. The issue is Envoy’s inability to accept PostgreSQL’s initial SSLRequest and then let the client initiate TLS.

This results in client connections failing at the Gateway even though the overall TLS re-encrypt architecture is desired.

what i expct:
When a Gateway server is declared with protocol: TLS and hosts: ["*.mypostgres.com"] and a client connects with PostgreSQL client configured for SSL (e.g. psql ... sslmode=verify-full and valid client certs), the Istio Gateway should accept the client upgrade to TLS and proceed with TLS termination; Envoy should not immediately drop connections that begin with PostgreSQL’s SSLRequest preamble, or there should be an official supported knob to allow PostgreSQL’s handshake to be handled.

what happens actually:

psql client using native PostgreSQL SSL negotiation fails: connection closed with message from client psql:

server closed the connection unexpectedly
This probably means the server terminated abnormally before or while processing the request.

Envoy logs show:

tls_inspector: parseClientHello failed: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
closing connection from <client-ip>:<port>: no matching filter chain found

openssl s_client -connect :5432 -servername succeeds — because it sends a TLS ClientHello immediately.
The upstream Envoy → Postgres TLS test (via openssl s_client from the gateway) also succeeds — upstream TLS config is OK.

Setup:

• all the necessary root ca, certs and keys are setup correctly.

1. Istio Gateway (in istio-system) — TLS termination on 5432:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: postgres-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 5432
      name: tls-postgres
      protocol: TLS
    tls:
      mode: SIMPLE
      credentialName: mypostgres-wildcard-cert
    hosts:
    - "*.mypostgres.com"

2. Virtual Service

# postgres-virtualservice.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: pg5-vs
  namespace: default
spec:
  hosts:
  - "pg5.mypostgres.com"
  gateways:
  - istio-system/postgres-gateway
  tcp:
  - match:
    - port: 5432
    route:
    - destination:
        host: pg5-primary.default.svc.cluster.local
        port:
          number: 5432

3. destionation rule

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: pg5-dr
  namespace: default
spec:
  host: pg5-primary.default.svc.cluster.local
  trafficPolicy:
    tls:
      mode: SIMPLE
      caCertificates: /etc/certs/pg-ca.crt

4. crunchy operator based pg cluster

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: pg5
  namespace: default
  annotations:
    postgres-operator.crunchydata.com/autoCreateUserSchema: "true"
spec:
  postgresVersion: 17

  users:
    - name: testuser
      databases:
        - postgres

  instances:
    - name: pg5-instance
      replicas: 1
      dataVolumeClaimSpec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

  customTLSSecret:
    name: pg.tls
    items:
      - key: tls.crt
        path: tls.crt
      - key: tls.key
        path: tls.key
      - key: ca.crt
        path: ca.crt

  proxy:
    pgBouncer:
      replicas: 1

  backups:
    pgbackrest:
      repos:
        - name: repo1
          volume:
            volumeClaimSpec:
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 1Gi

From a pod inside the cluster, attempt to connect using psql that uses PostgreSQL’s built-in SSLRequest (i.e. psql host=... sslmode=require or even verify-full if certs are present).

2025-10-29T08:38:55.321730Z debug envoy filter ... tls_inspector.cc:143 tls:onServerName(), requestedServerName: pg5.mypostgres.com
2025-10-29T08:38:55.339087Z debug envoy connection ... SSL shutdown: rc=0
...
2025-10-29T09:06:06.491550Z debug envoy filter ... tls_inspector.cc:227 tls inspector: parseClientHello failed: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER thread=25
2025-10-29T09:06:06.491601Z debug envoy conn_handler ... closing connection from 10.180.0.2:15835: no matching filter chain found thread=25

Config Dump(envoy listeners)

"name": "outbound|5432||pg5-primary.default.svc.cluster.local",
      "type": "EDS",
      "eds_cluster_config": {
       "eds_config": {
        "ads": {},
        "initial_fetch_timeout": "0s",
        "resource_api_version": "V3"
       },
       "service_name": "outbound|5432||pg5-primary.default.svc.cluster.local"
      },
      "connect_timeout": "10s",
      "lb_policy": "LEAST_REQUEST",
      "circuit_breakers": {
       "thresholds": [
        {
         "max_connections": 4294967295,
         "max_pending_requests": 4294967295,
         "max_requests": 4294967295,
         "max_retries": 4294967295,
         "track_remaining": true
        }
       ]
      },
      "transport_socket": {
       "name": "envoy.transport_sockets.tls",
       "typed_config": {
        "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
        "common_tls_context": {
         "tls_params": {
          "tls_minimum_protocol_version": "TLSv1_2",
          "tls_maximum_protocol_version": "TLSv1_3"
         },
         "combined_validation_context": {
          "default_validation_context": {},
          "validation_context_sds_secret_config": {
           "name": "file-root:/etc/certs/pg-ca.crt",
           "sds_config": {
            "api_config_source": {
             "api_type": "GRPC",
             "grpc_services": [
              {
               "envoy_grpc": {
                "cluster_name": "sds-grpc"
               }
              }
             ],
             "set_node_on_first_message_only": true,
             "transport_api_version": "V3"
            },
--
             "host": "pg5-primary.default.svc.cluster.local",
           "namespace": "default",
           "name": "pg5-primary"
          }
         ]
        }
       }
      },
      "common_lb_config": {},
      "alt_stat_name": "outbound|5432||pg5-primary.default.svc.cluster.local;",
      "typed_extension_protocol_options": {
       "envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
        "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
        "upstream_http_protocol_options": {
         "auto_sni": true,
         "auto_san_validation": true
        },
        "explicit_http_config": {
         "http_protocol_options": {}
        }
       }
      },
      "filters": [
       {
        "name": "istio.metadata_exchange",
        "typed_config": {
         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
         "type_url": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
         "value": {
          "protocol": "istio-peer-exchange",
          "enable_discovery": true
         }
        }
       }
      ]
     },
     "last_updated": "2025-10-27T14:19:28.128Z"
    },
    {
     "cluster": {
      "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name": "outbound|5432||pg5-replicas.default.svc.cluster.local",
      "type": "EDS",
      "eds_cluster_config": {
       "eds_config": {
        "ads": {},
        "initial_fetch_timeout": "0s",
        "resource_api_version": "V3"
       },
       "service_name": "outbound|5432||pg5-replicas.default.svc.cluster.local"
--
              "stat_prefix": "outbound|5432||pg5-primary.default.svc.cluster.local",
            "cluster": "outbound|5432||pg5-primary.default.svc.cluster.local"
           }
          }
         ],
         "transport_socket": {
          "name": "envoy.transport_sockets.tls",
          "typed_config": {
           "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
           "common_tls_context": {
            "alpn_protocols": [
             "h2",
             "http/1.1"
            ],
            "tls_certificate_sds_secret_configs": [
             {
              "name": "kubernetes://mypostgres-wildcard-cert",
              "sds_config": {
               "ads": {},
               "resource_api_version": "V3"
              }
             }
            ]
           },
           "require_client_certificate": false
          }
         },
         "transport_socket_connect_timeout": "15s"
        }
       ],
       "listener_filters": [
        {
         "name": "envoy.filters.listener.tls_inspector",
         "typed_config": {
          "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector",
          "initial_read_buffer_size": 512
         }
        }
       ],
       "traffic_direction": "OUTBOUND",
       "max_connections_to_accept_per_socket_event": 1
      },


Analysis
• Envoy’s tls_inspector correctly attempts to parse a TLS ClientHello. If it receives a PostgreSQL SSLRequest preamble (4–8 bytes), that input is not a TLS record header and triggers WRONG_VERSION_NUMBER. Envoy cannot extract SNI or TLS protocol version, so it cannot match the listener filter chain declared with transport_protocol: "tls". Without a matching filter chain, Envoy closes the connection.
• This is a protocol mismatch between PostgreSQL native SSL negotiation (SSLRequest → server ‘S’/‘N’ → TLS handshake) and Envoy’s expectation that TLS ClientHello is the very first bytes on the socket.
• ServiceEntry and DestinationRule are unrelated — the problem occurs at listener filter chain matching time (pre-routing), so changes in ServiceEntry do not address this.

Suggested path

Document the behavior clearly in Istio docs: explain that when protocol: TLS is used on Gateway servers, Envoy expects TLS ClientHello as first bytes and will drop connections that use protocol-level preambles (like PostgreSQL SSLRequest). Suggest recommended patterns: use passthrough for PostgreSQL, use a protocol translator (stunnel/HAProxy), or dual-mode listener.

Feature request: Add an upstream configuration knob to allow a “Postgres-compatibility mode” on TLS listener: a small listener extension that recognizes PostgreSQL SSLRequest and responds 'S' (or proxies to a TLS handshake) before invoking tls_inspector, so SNI is later provided in the real TLS handshake. This might be implemented as:
• A new listener filter (Postgres SSLRequest detector) that optionally responds and then allows TLS to proceed; or
• A documented EnvoyFilter example that installs a custom filter to allow Postgres preamble handling.

Version

istioctl version
client version: 1.27.2
control plane version: 1.27.2
data plane version: 1.27.2 (1 proxies)

kubectl version
Client Version: v1.33.3
Kustomize Version: v5.6.0
Server Version: v1.33.4

Additional Information

No response

[ggreenway]

For any issues with istio or istio docs, you'll need to file a separate issue with istio.

By default, the tls_inspector does not terminate a connection if it isn't TLS, but it is logging. I think the actual connection failure is probably in the TLS transport socket, again because it isn't TLS.

It would be possible to write another transport socket that handles whatever postgres is doing here; I'm unfamiliar with the protocol. But someone would have to sign up to do that work.

[AnindyaKrDas]

@ggreenway are you aware of any open issues with postgres wrt envoy filters?

Some references whcih i found were:

istio/istio#42380.
istio/istio#47895
istio/proxy#4328
istio/istio#47895
istio/istio#33345

[AnindyaKrDas]

Any insight on this: #14634 and #13112

[ggreenway]

StartTLS support was implemented I believe. I don't know much about postgres. Maybe @fabriziomello @cpakulski know more? They're the owners of the postgres extension in envoy.

[AnindyaKrDas]

Appreciate your swift response @ggreenway :)

[ggreenway]

I also just noticed #40803, which may be related to what you need.

[AnindyaKrDas]

@ggreenway yea , its still ongoing though.
Also, i came across this : https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_protocols/postgres#arch-overview-postgres

Here, it says:

The filter currently offers the following features:

Decode non SSL traffic, **ignore SSL traffic.**

[AnindyaKrDas]

@fabriziomello @cpakulski any insight on this issue which i am facing?