Socks: Fix buffer full panic when encoding large UDP packets

[vemneyy]

bug

Description:
panic: common/buf: buffer is full
This issue occurred in the SOCKS inbound proxy when processing large UDP packets (e.g., QUIC handshake packets >8KB).
The regression was introduced after the upstream change around v25.8.29, when the behavior of (*buf.Buffer).Write was modified:
Before v25.8.29 (<= v25.8.3) – Write silently truncated data exceeding buffer capacity (causing silent corruption).
After v25.8.29 – Write now correctly returns ErrBufferFull, preventing silent data loss but exposing logic errors.

EncodeUDPPacket used a fixed-size buffer via buf.New() and common.Must2(b.Write(data)), causing a panic when large packets exceeded the buffer capacity.

Fix:

• Dynamically allocate buffer size based on total payload length.
• Replace common.Must2 with proper error handling.
• Add unit test TestEncodeUDPPacketLargePayload to ensure stability for large datagrams.

Error log:

panic: common/buf: buffer is full
goroutine 230 [running]:
github.com/xtls/xray-core/common.Must(...)
	github.com/xtls/xray-core/common/common.go:22
github.com/xtls/xray-core/common.Must2[...](...)
	github.com/xtls/xray-core/common/common.go:30
github.com/xtls/xray-core/proxy/socks.EncodeUDPPacket(0xc00091ac90, {0xc000fdc000, 0x1ff8, 0x0?})
	github.com/xtls/xray-core/proxy/socks/protocol.go:356 +0x265
github.com/xtls/xray-core/proxy/socks.(*Server).handleUDPPayload.func1({0x7ff64e8a4478, 0xc000930550}, 0x0?)
	github.com/xtls/xray-core/proxy/socks/server.go:211 +0x1fb
github.com/xtls/xray-core/transport/internet/udp.handleInput({0x7ff64e8a4478, 0xc000930550}, 0x0?, {{0x7ff64e8a49b8?, 0xc000bf8934?}, 0x0?, 0x0?}, 0xc0005e4eb8, 0x0?)
	github.com/xtls/xray-core/transport/internet/udp/dispatcher.go:157 +0x227
created by github.com/xtls/xray-core/transport/internet/udp.(*Dispatcher).getInboundRay in goroutine 228
	github.com/xtls/xray-core/transport/internet/udp/dispatcher.go:104 +0x6bb

[Fangliding]

我觉得可能静默丢弃才是对的 我们不能迁就无限大的data(

[vemneyy]

我觉得可能静默丢弃才是对的 我们不能迁就无限大的data(

The silent truncation was eliminated by the maintainers in version v25.8.29, when [(*buf.Buffer).Write] started returning ErrBufferFull instead of quietly discarding excess data. This isn’t about “accommodating infinitely large packets”; it’s about fixing a hidden logic flaw that used to silently corrupt UDP datagrams. The proper solution is to allocate the buffer dynamically based on the actual payload length and handle the write error explicitly. Keeping silent truncation would mean knowingly preserving data corruption and hidden state loss unacceptable behavior, especially for large QUIC packets exceeding 8 KB

[Fangliding]

以前也会看情况丢弃UDP包 大多数UDP程序可能都需要自己探测MTU 8KB已经够大了 QUIC的最低不低于1200字节就行了

[vemneyy]

In this case (look panic log) the panic wasnt triggered by a typical oversized or malformed UDP datagram it occurred when Firefox attempted to establish a QUIC connection to a website. During the initial handshake the QUIC packets exceeded 8 KB, which caused the EncodeUDPPacket function to hit the fixed-size buffer limit and panic, effectively halting the core. This regression doesnt occur in kernel version v25.8.3, where the buffer behavior still tolerated such payloads (albeit with silent truncation). The newer versions correctly surface the buffer overflow but exposed the underlying logic flaw in the SOCKS UDP handler

[Fangliding]

一个8KB甚至大于8KB的UDP包跑到公网上基本也会被丢弃 哪怕是分片也是极其不好的 这绝对是错误的 (如果说的是真的)Firefox很可能是读取了loopback接口的MTU 65536才发出这样的包