read from workConn for sudp error: EOF; all visitors/proxies "stuck"

[ByteCrunch]

Bug Description

We have issues for multiple frpc instances at different locations using different internet connections.

The symptoms are that all the proxies/visitors are broken after the following output in the frpc logs:

[visitor/stcp.go:112] [c3265bd915ac1b76] [prdkeycloak] get newVisitorConnRespMsg error: EOF
[proxy/sudp.go:143] [c3265bd915ac1b76] [dns] read from workConn for sudp error: EOF
[proxy/sudp.go:160] [c3265bd915ac1b76] [dns] writer goroutine for sudp work connection closed

In the frps logs at the same time, all proxies are closed:

proxy/proxy.go:204] [5d25de26cc7b4706] [prdeksd] get a user connection [117.240.119.149:54188]
[proxy/proxy.go:204] [5d25de26cc7b4706] [prdkeycloak] get a user connection [117.240.119.149:54188]
[proxy/proxy.go:115] [c3265bd915ac1b76] [zzdynprt-50084] proxy closing
[proxy/proxy.go:115] [c3265bd915ac1b76] [zzdynprt-49783] proxy closing
[...]
[proxy/proxy.go:201] [c3265bd915ac1b76] [zzdynprt-50084] listener is closed: listener closed
[...]

In the frps metrics the client count is decreased by 1 in this case.

Restarting frpc always recovers the connections until the same symptoms will show up again after some hours or days.

frpc never recovers, there are also no further log lines or attempts to reconnect to the server. Basically frpc is stuck until restarted.

As a mitigation we have implemented a wrapper which, when frpc log lines like "read from workConn for sudp error: EOF" appear, will check the general internet connectivity and a service reachable via frp itself and then will restart the frpc process.

We always see that there is no general internet outage as at the same time. Just the services provided via frp are not reachable.

frpc Version

v0.65.0

frps Version

v0.64.0

System Architecture

linux/amd64

Configurations

frpc.toml

serverAddr = "x.x.x.x"
serverPort = 9000
transport.tcpMux = true # multiplex connections

loginFailExit = false

# webserver needed for hot reload feature
webServer.addr = "127.0.0.1"
webServer.port = 7400
webServer.user = "admin"
webServer.password = "admin"

auth.method = "token"
auth.token = "..."

log.level = "debug"

[[proxies]]
name = "dns"
type = "sudp"
secretKey = "..."
localIP = "172.16.153.1"
localPort = 53

[[visitors]]
name = "prdkeycloak"
type = "stcp"
secretKey = "..."
serverName = "prdkeycloak"
bindAddr = "172.16.199.1"
bindPort = 5443

[...]

Logs

2025-11-27 11:13:11 [info     ] performing periodic checks                                                       filename=main.py func_name=periodic_check lineno=78
2025-11-27 11:13:12 [info     ] system health: {'cpu_usage': '0.0%', 'memory_usage': '17.5%', 'disk_usage': '13.6%', 'open_socket_count': 254} filename=main.py func_name=periodic_check lineno=79
2025-11-27 11:13:15 [info     ] connectivity check: internet (https://www.google.com) succeeded.               filename=main.py func_name=check_connectivity lineno=47
2025-11-27 11:13:20 [warning  ] connectivity check: visitor service (https://172.16.199.1:5443) failed: <urlopen error _ssl.c:993: The handshake operation timed out> filename=main.py func_name=check_connectivity lineno=53
2025-11-27 11:13:20 [info     ] [FRPC] 2025-11-27 11:13:15.310 [D] [visitor/stcp.go:86] [c3265bd915ac1b76] [prdkeycloak] get a new stcp user connection
 filename=main.py func_name=handle_frpc_output lineno=94
2025-11-27 11:13:21 [info     ] [FRPC] 2025-11-27 11:13:21.081 [W] [visitor/stcp.go:112] [c3265bd915ac1b76] [prdkeycloak] get newVisitorConnRespMsg error: EOF
2025-11-27 11:13:21.082 [W] [proxy/sudp.go:143] [c3265bd915ac1b76] [dns] read from workConn for sudp error: EOF
2025-11-27 11:13:21.082 [I] [proxy/sudp.go:160] [c3265bd915ac1b76] [dns] writer goroutine for sudp work connection closed
 filename=main.py func_name=handle_frpc_output lineno=94
2025-11-27 11:13:21 [warning  ] Potential issue detected in frpc logs, performing connectivity checks            filename=main.py func_name=handle_frpc_output lineno=132
2025-11-27 11:13:22 [info     ] system health: {'cpu_usage': '0.0%', 'memory_usage': '17.5%', 'disk_usage': '13.6%', 'open_socket_count': 253} filename=main.py func_name=handle_frpc_output lineno=133
2025-11-27 11:13:22 [warning  ] connectivity check: visitor service (https://172.16.199.1:5443) failed: <urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1010)> filename=main.py func_name=check_connectivity lineno=53
2025-11-27 11:13:22 [info     ] restarting frpc process                                                          filename=main.py func_name=restart_frpc lineno=186
2025-11-27 11:13:22 [error    ] frpc process exited: (CalledProcessError(1, 'frpc process terminated due to failed connectivity check'),) filename=main.py func_name=start lineno=162
2025-11-27 11:13:22 [info     ] Restarting frpc in 5 seconds                                                     filename=main.py func_name=restart_message_and_wait lineno=179
2025-11-27 11:13:27 [info     ] Starting SecretsFS                                                               filename=main.py func_name=start lineno=147
2025-11-27 11:13:27 [info     ] Starting frpc                                                                    filename=main.py func_name=start lineno=153
2025-11-27 11:13:27 [info     ] frpc process running                                                             pid=69 filename=main.py func_name=start lineno=157
2025-11-27 11:13:27 [info     ] [FRPC] 2025-11-27 11:13:27.262 [I] [sub/root.go:149] start frpc service for config file [/opt/frp/etc/frpc.toml]
 filename=main.py func_name=handle_frpc_output lineno=94
2025-11-27 11:13:27 [info     ] [FRPC] 2025-11-27 11:13:27.263 [I] [client/service.go:325] try to connect to server...
 filename=main.py func_name=handle_frpc_output lineno=94
2025-11-27 11:13:27 [info     ] frpc is trying to connect to its FRP server, checking internet connectivity      filename=main.py func_name=handle_frpc_output lineno=99
2025-11-27 11:13:30 [info     ] connectivity check: internet (https://www.google.com) succeeded.               filename=main.py func_name=check_connectivity lineno=47
2025-11-27 11:13:30 [info     ] [FRPC] 2025-11-27 11:13:27.263 [I] [client/service.go:204] admin server listen on 127.0.0.1:7400
2025-11-27 11:13:27.572 [I] [client/service.go:317] [80eca8832f0346c8] login to server success, get run id [80eca8832f0346c8]

Steps to reproduce

Unfortunately we haven't found a way to reproduce the error.
It occurs sporadically after hours or days of using frp tunneling without issues. But it only happens when actual traffic is handled by frp - not when frpc is only connected to a frps without actually using any proxies/visitors.

Affected area

• Docs
• Installation
• Performance and Scalability
• Security
• User Experience
• Test and Release
• Developer Infrastructure
• Client Plugin
• Server Plugin
• Extensions
• Others

[fatedier]

You can also upgrade frps to version v0.65 and observe it for a while. I remember that version updated the underlying yamux library and fixed a possibly related issue.