Weird port change

[mkyybx]

What version of shadowsocks-libev are you using?

shadowsocks-libev 3.1.3
I got this from official OpenWrt repo and installed on my EA3500 router.

What operating system are you using?

Linux EA3500 4.14.95 #0 Mon Jan 28 08:54:32 2019 armv5tel GNU/Linux

What did you do?

I configured the ss-server as the following:

/var/etc/shadowsocks-libev/ss_server.cfg07eca1.json:

{
"use_syslog": true,
"ipv6_first": false,
"fast_open": false,
"reuse_port": false,
"mode": "tcp_and_udp",
"server": "0.0.0.0",
"server_port": 52500,
"method": "chacha20-ietf-poly1305",
"password": "********"
}
And the OpenWrt will run ss-server with the following parameters:
/usr/bin/ss-server -c /var/etc/shadowsocks-libev/ss_server.cfg07eca1.json -b 192.168.1.1

What did you expect to see?

I could connect my router using shadowsocks client from another host and use the socks5 proxy to browse Internet.

What did you see instead?

Some webpages could be loaded normally like google.com. But some pages couldn't like bilibili.com. Through the browser I found it was stuck by the request of gstatic.com or akamai***.net.

I then turn ss-server into verbose mode and use curl to do the test. I found sometimes ss-server will try to connect to 5xxxx port instead of 443 port. The log is as following:

Client side:

$ tsocks curl https://www.google.com:443 -o /dev/null -v
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.217.5.196...

• TCP_NODELAY set
• Connected to www.google.com (127.0.0.1) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• successfully set certificate verify locations:
• CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
  } [512 bytes data]
  0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0^C

Server(router) side:

22:56:35 daemon.info /usr/bin/ss-server[16974]: accept a connection
22:56:35 daemon.info /usr/bin/ss-server[16974]: connect to 172.217.5.196:50177
22:56:53 daemon.info /usr/bin/ss-server[16974]: TCP connection timeout

Sometimes I will also get the following message:

getpeername: Socket not connected

Thanks

[madeye]

I don't think the log on your server has anything to do with your local curl command.

Your problem looks a typical DNS pollution issue, which means you need to setup a non-poisoned local DNS server first.

[mkyybx]

Thanks for your reply.
I don't think my DNS is not polluted because I'm Charter's subscriber. curl resolves www.google.com as 172.217.5.196 and tries to access 172.217.5.196:443. But the server log showes ss-server tries to access 172.217.5.196:50177 instead. I don't know where does the ss-server derive port 50177 and it doesn't seem like the endian-problem.

BTW, I can visit 172.217.5.196 and get the Google home page.

Thank you.

[madeye]

Try curl --socks5-hostname instead of tsocks

[mkyybx]

Thanks for your reply.

Client log:

$ curl --socks5-hostname 127.0.0.1:1086 https://start.firefoxchina.cn/ -o /dev/null -v
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.0.1...

• TCP_NODELAY set
• SOCKS5 communication to start.firefoxchina.cn:443
• SOCKS5 request granted.
• Connected to 127.0.0.1 (127.0.0.1) port 1086 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@strength
• successfully set certificate verify locations:
• CAfile: /etc/ssl/cert.pem
  CApath: none
• TLSv1.2 (OUT), TLS handshake, Client hello (1):
  } [227 bytes data]
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to start.firefoxchina.cn:443
• stopped the pause stream!
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
• Closing connection 0
  curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to start.firefoxchina.cn:443

Server log:

05:35:23 daemon.info /usr/bin/ss-server[14214]: connect to start.firefoxchina.cn:28161
05:35:23 daemon.info /usr/bin/ss-server[14214]: found address name v4 address start.firefoxchina.cn.wscdns.com
05:35:23 daemon.info /usr/bin/ss-server[14214]: found address name v6 address start.firefoxchina.cn.wscdns.com
05:35:23 daemon.info /usr/bin/ss-server[14214]: successfully resolved start.firefoxchina.cn
05:35:23 daemon.err /usr/bin/ss-server[14214]: getpeername: Socket not connected
05:35:23 daemon.info /usr/bin/ss-server[14214]: current remote connection: 9
05:35:23 daemon.info /usr/bin/ss-server[14214]: current server connection: 9
05:35:27 daemon.info /usr/bin/ss-server[14214]: TCP connection timeout

This time I changed another website because I can curl https://www.google.com with no problem.

[mkyybx]

Another example for static:

Client log:

$ curl --socks5-hostname 127.0.0.1:1086 https://www.gstatic.com/ -o /dev/null -v
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.0.1...

• TCP_NODELAY set
• SOCKS5 communication to www.gstatic.com:443
• SOCKS5 request granted.
• Connected to 127.0.0.1 (127.0.0.1) port 1086 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@strength
• successfully set certificate verify locations:
• CAfile: /etc/ssl/cert.pem
  CApath: none
• TLSv1.2 (OUT), TLS handshake, Client hello (1):
  } [221 bytes data]
  0 0 0 0 0 0 0 0 --:--:-- 0:00:38 --:--:-- 0^C

Server log:

05:43:05 daemon.info /usr/bin/ss-server[14214]: accept a connection
05:43:05 daemon.info /usr/bin/ss-server[14214]: connect to www.gstatic.com:27905
05:43:05 daemon.info /usr/bin/ss-server[14214]: found address name v4 address www.gstatic.com
05:43:05 daemon.info /usr/bin/ss-server[14214]: found address name v6 address www.gstatic.com
05:43:05 daemon.info /usr/bin/ss-server[14214]: successfully resolved www.gstatic.com

[madeye]

ss-server is installed on your router? If so, it looks a porting issue on your platform.

Given the version is maintained by third-party, you have to report it to them instead.

You can also try our porting here: https://github.com/shadowsocks/openwrt-shadowsocks/releases

[mkyybx]

Thanks for your reply.
I installed the newest(3.2.5) ss-server from the link you provided, but the issue isn't solved.
I'll appreciate it if you could help me solve or pin point where the bug is. At the same time I will try compile from the source and debug it myself.

[madeye]

Also try change the cipher to rc4-md5, to see if it's a library issue.

[mkyybx]

Thanks for your reply.

• It seems like the hardware bug where the CPU does not execute ldrh instruction correctly. The same binary will run into error in my EA3500.
• Usually ldrh r0, [r1] means loading bytes in the memory location r1 and r1+1 into r0, but my CPU loads r1 and r1-1 into r0.
• The bug happens at src/server.c:897: port = (*(uint16_t *)(server->buf->data + offset));
• This code aims to load store the port number in a variable where my CPU translates it into
  ldrh r11, [r0, r3]. r0 stores the address of server->buf->data and r3 is 0x11 which is probably the offset variable in my case. Therefore the CPU loads the first bytes of the port and the last byte of domain name into register r0 and mixes them together as the "port number". (First byte means the lowest memory address.)
• The hex form of decimal 443 in big endian is 0x01bb and the ascii of 'm' is 0x6d (the last letter of www.gstatic.com). The mixing result of r0 is 0x016d and after calling ntohs(port) it, the port would turn out to be 27905 (0x6d01).

• I think there are two ways to solve this issue: compile the source code with no optimizations or patch the instruction to ldr instead of ldrh.

This may also related to alignment issue because some times the website could be visited. In the case above, the offset of the port number of server->buf->data is 0x11 which may reason why ldrh read the memory location of r1 and r1-1.

I am not sure if my findings is reasonable and questions and suggestions are welcome!

I remember someone else has posted the same question and I am not sure if my findings is helpful to him. But I cannot reason about why you got 27904 port. #2221 @mailshuxin

[madeye]

Cool! @mkyybx Thanks for the finding!

I think it's actually a bug of the compiler, as a result, the unaligned ldrh caused the problem here.

[madeye]

According to ARM's ISA, 4.10.4, http://vision.gel.ulaval.ca/~jflalonde/cours/1001/h17/docs/arm-instructionset.pdf

A halfword load (LDRSH or LDRH) expects data on data bus inputs 15 through to 0 if
the supplied address is on a word boundary and on data bus inputs 31 through to 16 if
it is a halfword boundary, (A[1]=1).The supplied address should always be on a
halfword boundary.

Given the offset of port depends on the length of hostname, it's possible that the address is not on a halfword boundary, which mean not halfword aligned.

Someone from OpenWRT toolchain team should take a look at it.