DoS on sending invalid length in iperf_api.c+2684

[someusername123]

NOTE: The iperf3 issue tracker is for registering bugs, enhancement
requests, or submissions of code. It is not a means for asking
questions about building or using iperf3. Those are best directed
towards the Discussions section for this project at
https://github.com/esnet/iperf/discussions
or to the iperf3 mailing list at iperf-dev@googlegroups.com.
A list of frequently-asked questions
regarding iperf3 can be found at http://software.es.net/iperf/faq.html.

Context

In doing some testing on iperf I found a bug that could effect all versions that use "str = (char ) calloc(sizeof(char), hsize+1); / +1 for trailing null */" in iperf_api.c+2684. By sending a value of ffffffff at length 37 in the TCP packet (where the size is set) the code will add one to it, rolling the hex value to 0 and calling calloc(1,0) which ends in a malloc(): corrupted top size error and crashes the program.
If needed I can provide a proof of concept python code, just let me know if that is required.

• Version of iperf3: iperf 3.13 (cJSON 1.7.15)

• Hardware:virtual machine

• Operating system (and distribution, if any):Linux kali 6.1.0-kali-9-amd64

Please note: iperf3 is supported on Linux, FreeBSD, and macOS.
Support may be provided on a best-effort basis to other UNIX-like
platforms. We cannot provide support for building and/or running
iperf3 on Windows, iOS, or Android.

• Other relevant information (for example, non-default compilers,
  libraries, cross-compiling, etc.): Not that I'm aware of. Installed using apt-get iperf3.

Please fill out one of the "Bug Report" or "Enhancement Request"
sections, as appropriate. Note that submissions of bug fixes, new
features, etc. should be done as a pull request at
https://github.com/esnet/iperf/pulls

Bug Report

DoS on sending invalid length in iperf_api.c+2684, program crashes.

• Expected Behavior
  The application handles a overly large user controlled hsize value.
• Actual Behavior
  There are no bounds checking in place for hsize.
• Steps to Reproduce
  Send two TCP 37 padded bytes to the iperf server application running as "iperf -s" or "iperf -s -d 4", plus an additional 4 hex '\xff' values, and some additional padding at the end (say 50 bytes) of 'A's.
  Python encoded string to send- print(b'\x41'*37+b'\xff'*4+b'\x41'*50)
• Possible Solution
  The length of hsize should be limited to ffffffff-1 if the expectation is that there are possibilities that the size could reach that value to prevent the improper calling of calloc with a 0 byte size and putting a single character in it.

Enhancement Request

• Current behavior

• Desired behavior

• Implementation notes

[bmah888]

Thanks for the bug report! We'll look into this.