iperf3: error - unable to receive parameters from client: Bad file descriptor

[coinhu1995]

Context

Got error when measuring bandwidth between macos (local) and ubuntu (AWS EC2 with public IP)

• Version of iperf3:
  Client and server are the same: iperf 3.17.1+

• Hardware:

• Client: Apple macmini M1 - macos Sonoma 14.2.1
• Server: AWS EC2 - Ubuntu 18.04.5 LTS

Bug Report

• Steps to Reproduce
  Server log

ubuntu@ip-172-31-0-5:~/tmp/iperf-master$ iperf3 -s -B 0.0.0.0 -V -d
iperf 3.17.1+
Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)


State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
warning: JSON size of data read does not correspond to offered length
All threads stopped
iperf3: error - unable to receive parameters from client: Bad file descriptor
iperf 3.17.1+
Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

Client log

user@name iperf-master % iperf3 -c 52.77.179.243 -V -d
iperf 3.17.1+
Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Reading new State from the Server - current state is 0-Test reset
State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.17.1+"
}
Time: Thu, 12 Dec 2024 07:58:14 UTC
Connecting to host 52.77.179.243, port 5201
      Cookie: jnfff5macsu2es3oawqxbguci47gdwncu576
      TCP MSS: 1388 (default)
Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
All threads stopped
State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
iperf3: error - unable to send control message - port may not be available, the other side may have stopped running, etc.: Broken pipe

• Possible Solution
  N/A

[davidBar-On]

warning: JSON size of data read does not correspond to offered length

To better understand this issue, I created a branch in my private iperf3 fork which print a debug message when this problem happens. The message starts with "*** TEMP DEBUG - Parameters JSON: ..." and shows the expected Parameters JSON length and the actual length received.

Can you rebuild iperf3 for the server from this branch, run it and send the debug message output?

[coinhu1995]

@davidBar-On Here is the log

iperf-issue-1808-json-parameters-length-mismatch % iperf3 -c 52.77.179.243 -V -d
iperf 3.17.1+
Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Reading new State from the Server - current state is 0-Test reset
State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.17.1+"
}
Time: Fri, 13 Dec 2024 02:05:28 UTC
Connecting to host 52.77.179.243, port 5201
      Cookie: udgeliqdmeiobpcu7jub7d573lmmogzpeg5j
      TCP MSS: 1388 (default)
Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
All threads stopped
State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
iperf3: error - unable to send control message - port may not be available, the other side may have stopped running, etc.: Broken pipe

~/tmp/iperf-issue-1808-json-parameters-length-mismatch$ iperf3 -s -B 0.0.0.0 -V -d
iperf 3.17.1+
Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)
State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
*** TEMP DEBUG - Parameters JSON: expected 127 bytes but received 0 bytes;
warning: JSON size of data read does not correspond to offered length
All threads stopped
iperf3: error - unable to receive parameters from client: Bad file descriptor
iperf 3.17.1+
Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

[coinhu1995]

@davidBar-On Additional information: I'm on the corporate network. Are there any chances that the corporate network could prevent the client from sending packets to the server?

[davidBar-On]

I'm on the corporate network. Are there any chances that the corporate network could prevent the client from sending packets to the server?

I don't know. Usually different UDP/TCP ports are blocked, but in this case some messages were already exchanged between the client and server before the error happened. If there is a content filter (which I am not sure there is), then maybe the JSON contents were filtered.

*** TEMP DEBUG - Parameters JSON: expected 127 bytes but received 0 bytes;

Receiving 0 bytes can happen either because of timeout or because the the session was closed (by the corporate network)? To better understand what happens, I added few additional debug messages (same branch on my repository). Can you build the server from the new code and send the results?

Please add the --timestamps option to both Client and Server. If the error is because of timeout, probably because the network filters the message, then the log messages timestamps may indicate that.

[coinhu1995]

@davidBar-On Here is the log

Server

$ iperf3 -s -B 0.0.0.0 -V -d --timestamps
Mon Dec 16 02:50:12 2024 iperf 3.17.1+
Mon Dec 16 02:50:12 2024 Mon Dec 16 02:50:12 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Mon Dec 16 02:50:12 2024 -----------------------------------------------------------
Mon Dec 16 02:50:12 2024 Server listening on 5201 (test #1)
Mon Dec 16 02:50:12 2024 -----------------------------------------------------------
Mon Dec 16 02:50:12 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)
*** TEMP DEBUG: Nread() read() returned 37, errno=0;
Mon Dec 16 02:50:49 2024 State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
*** TEMP DEBUG: Nread() read() returned 4, errno=0;
*** TEMP DEBUG: Nread() select() 1 returned 0 (timeout), errno=0;
warning: JSON size of data read does not correspond to offered length - expected 127 bytes but received 0 bytes; errno=0
Mon Dec 16 02:50:59 2024 All threads stopped
Mon Dec 16 02:50:59 2024 iperf3: error - unable to receive parameters from client: Bad file descriptor
Mon Dec 16 02:50:59 2024 iperf 3.17.1+
Mon Dec 16 02:50:59 2024 Mon Dec 16 02:50:59 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Mon Dec 16 02:50:59 2024 -----------------------------------------------------------
Mon Dec 16 02:50:59 2024 Server listening on 5201 (test #2)
Mon Dec 16 02:50:59 2024 -----------------------------------------------------------
Mon Dec 16 02:50:59 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

Client

iperf3 -c 52.77.179.243 -V -d --timestamps
Mon Dec 16 09:50:48 2024 iperf 3.17.1+
Mon Dec 16 09:50:48 2024 Mon Dec 16 09:50:48 2024 Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Mon Dec 16 09:50:49 2024 Reading new State from the Server - current state is 0-Test reset
Mon Dec 16 09:50:49 2024 State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.17.1+"
}
Mon Dec 16 09:50:49 2024 Time: Mon, 16 Dec 2024 02:50:49 UTC
Mon Dec 16 09:50:49 2024 Connecting to host 52.77.179.243, port 5201
Mon Dec 16 09:50:49 2024       Cookie: 6t5wnbqf3avcmt6kjmprzox65c2dghxrdbjp
Mon Dec 16 09:50:49 2024       TCP MSS: 1388 (default)

Mon Dec 16 09:57:23 2024 Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
Mon Dec 16 09:57:23 2024 All threads stopped
Mon Dec 16 09:57:23 2024 State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
Mon Dec 16 09:57:23 2024 iperf3: error - unable to send control message - port may not be available, the other side may have stopped running, etc.: Broken pipe

[davidBar-On]

Somehow the JSON parameters message does not arrive to the server, so the read times out (note the 10 seconds difference between the client send time 50:49 and the server error at 50:59). It is probably because of some filtering in the corporate network.

I don't know what this filtering is, but one guess is that it filters text/JSON. To check that I have changed the sent JSON to zeros. If JSON is filtered, then the message will arrive to the server (but it will fail since this is not JSON). Can you rebuild and run with the new version in the branch to check?

[coinhu1995]

@davidBar-On Here is the update

Server

iperf3 -s -B 0.0.0.0 -V -d --timestamps
Tue Dec 17 02:16:12 2024 iperf 3.18
Tue Dec 17 02:16:12 2024 Tue Dec 17 02:16:12 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Tue Dec 17 02:16:12 2024 -----------------------------------------------------------
Tue Dec 17 02:16:12 2024 Server listening on 5201 (test #1)
Tue Dec 17 02:16:12 2024 -----------------------------------------------------------
Tue Dec 17 02:16:12 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)
*** TEMP DEBUG: Nread() read() returned 37, errno=0;
Tue Dec 17 02:16:24 2024 State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
*** TEMP DEBUG: Nread() read() returned 4, errno=0;
*** TEMP DEBUG: Nread() select() 1 returned 0 (timeout), errno=0;
warning: JSON size of data read does not correspond to offered length - expected 124 bytes but received 0 bytes; errno=0
Tue Dec 17 02:16:34 2024 All threads stopped
Tue Dec 17 02:16:34 2024 iperf3: error - unable to receive parameters from client: Bad file descriptor
Tue Dec 17 02:16:34 2024 iperf 3.18
Tue Dec 17 02:16:34 2024 Tue Dec 17 02:16:34 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Tue Dec 17 02:16:34 2024 -----------------------------------------------------------
Tue Dec 17 02:16:34 2024 Server listening on 5201 (test #2)
Tue Dec 17 02:16:34 2024 -----------------------------------------------------------
Tue Dec 17 02:16:34 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

Client

iperf3 -c 52.77.179.243 -V -d --timestamps
Tue Dec 17 09:16:24 2024 iperf 3.18
Tue Dec 17 09:16:24 2024 Tue Dec 17 09:16:24 2024 Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Tue Dec 17 09:16:24 2024 Reading new State from the Server - current state is 0-Test reset
Tue Dec 17 09:16:24 2024 State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.18"
}
Tue Dec 17 09:16:24 2024 Time: Tue, 17 Dec 2024 02:16:24 UTC
Tue Dec 17 09:16:24 2024 Connecting to host 52.77.179.243, port 5201
Tue Dec 17 09:16:24 2024       Cookie: pqiaqxlgn7eajr6tbif6ozrb737khtbbokhn
Tue Dec 17 09:16:24 2024       TCP MSS: 1388 (default)
Tue Dec 17 09:17:17 2024 Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
Tue Dec 17 09:17:17 2024 All threads stopped
Tue Dec 17 09:17:17 2024 State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
Tue Dec 17 09:17:17 2024 iperf3: error - unable to send control message - port may not be available, the other side may have stopped running, etc.: Broken pipe

[davidBar-On]

@coinhu1995, sorry, I somehow didn't push the last debug changes to github. I did it now. Can you rebuild and run again?

[coinhu1995]

@davidBar-On

Server

iperf3 -s -B 0.0.0.0 -V -d --timestamps
Tue Dec 17 10:08:18 2024 iperf 3.18
Tue Dec 17 10:08:18 2024 Tue Dec 17 10:08:18 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Tue Dec 17 10:08:18 2024 -----------------------------------------------------------
Tue Dec 17 10:08:18 2024 Server listening on 5201 (test #1)
Tue Dec 17 10:08:18 2024 -----------------------------------------------------------
Tue Dec 17 10:08:18 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)
*** TEMP DEBUG: Nread() read() returned 37, errno=0;
Tue Dec 17 10:08:29 2024 State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
*** TEMP DEBUG: Nread() read() returned 4, errno=0;
*** TEMP DEBUG: Nread() read() returned 124, errno=0;
Tue Dec 17 10:08:29 2024 All threads stopped
Tue Dec 17 10:08:29 2024 iperf3: error - unable to receive parameters from client: Bad file descriptor
Tue Dec 17 10:08:29 2024 iperf 3.18
Tue Dec 17 10:08:29 2024 Tue Dec 17 10:08:29 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Tue Dec 17 10:08:29 2024 -----------------------------------------------------------
Tue Dec 17 10:08:29 2024 Server listening on 5201 (test #2)
Tue Dec 17 10:08:29 2024 -----------------------------------------------------------
Tue Dec 17 10:08:29 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

Client

iperf3 -c 52.77.179.243 -V -d --timestamps                                                                           
Tue Dec 17 17:08:29 2024 iperf 3.18
Tue Dec 17 17:08:29 2024 Tue Dec 17 17:08:29 2024 Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Tue Dec 17 17:08:29 2024 Reading new State from the Server - current state is 0-Test reset
Tue Dec 17 17:08:29 2024 State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.18"
}
*** TEMP DEBUG: set JSON string to zeros - strlen(str)=0;
*** TEMP DEBUG: JSON_write() length=124, return code=0;
Tue Dec 17 17:08:29 2024 Time: Tue, 17 Dec 2024 10:08:29 UTC
Tue Dec 17 17:08:29 2024 Connecting to host 52.77.179.243, port 5201
Tue Dec 17 17:08:29 2024       Cookie: jlkp5fpm6zywxzcuyjdjcna2q7rmljrhebzn
Tue Dec 17 17:08:29 2024       TCP MSS: 1388 (default)
Tue Dec 17 17:08:29 2024 Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
Tue Dec 17 17:08:29 2024 All threads stopped
Tue Dec 17 17:08:29 2024 State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
Tue Dec 17 17:08:29 2024 iperf3: error - control socket has closed unexpectedly

[davidBar-On]

@coinhu1995, thanks for re-running. It is almost clear now that this is a corporate network issue. Either it filters JSON, filters text messages or something else. This is because in the last test there was no timeout and all the full 124 bytes message was received by the server. There is still small probability that this is an iperf3 issue, although I cannot guess what that might be.

I suggest to ask corporate network administrators about this. If you want to evaluate this further, you can try sending different messages instead of the parameters string, to see if they are received by the server. This can be done with my branch code, in the JSON_write() function in iperf_api.c. I suggest either or both:

• Try to change the message contents to see its effect, by changing the memset(str, 0, hsize) I inserted to set the message to zeroes. E.g. change it to memset(str, 'X', hsize) to see if they are received by the server.
• Try sending different JSON or textual messages to see if they are received by the server (print str after str = cJSON_PrintUnformatted(json) to see how the sent JSON looks like). This is by replaceing the Nwrite(fd, str, hsize, Ptcp). For example, send emty JSON by Nwrite(fd, "{}", strlen("{}"), Ptcp), or send subset of the JSON to see if and which filed(s) may cause the problem - e.g. str="{\"tcp\":true,\"omit\":0,\"client_version\":\"3.18\"}"; Nwrite(fd, str, strlen(str), Ptcp).

If you get any interesting results, either from the tests or about the corporate network, please share.

[coinhu1995]

@davidBar-On thank you. I will check with network admin.

[lmq8267]

The error I'm encountering is: Unable to create a new stream. I'm on Linux and also encountered this error. Simply setting export TMPDIR=/tmp fixed it.

[davidBar-On]

The error I'm encountering is: Unable to create a new stream. I'm on Linux and also encountered this error. Simply setting export TMPDIR=/tmp fixed it.

Looking at the iperf3 code, the only possibility I see is that either TMP, TEMP or TEMPDIR were already defined with a value different than /tmp. Is this the case? (By the way, I believe that the root cause here is different than the original cause for the issue discussed above.)

[lmq8267]

Looking at the iperf3 code, the only possibility I see is that either TMP, TEMP or TEMPDIR were already defined with a value different than /tmp. Is this the case? (By the way, I believe that the root cause here is different than the original cause for the issue discussed above.)

Yes, my error is different from the discussions above.
I've examined the code, and in my system environment, variables like TMP, TEMP, etc., are all empty. However, iperf3 does not default to /tmp either. It only works when I explicitly set TMPDIR=/tmp. Setting other variables like TMP or TEMP has no effect.