Undefined-behavior (left shift of negative value) in `rdpcm_h_fallback()`

[skorpion98]

Summary

We found an undefined behavior while testing the encoder_heic_fuzzer fuzzing harness for ImageMagick. Upon further inspection, this issue seems to be related to an external dependency (libde265) rather than to ImageMagick itself. For this reason, we decided to report it directly here instead of ImageMagick. It is still possible that this originates from an incorrect use of libde265; if you believe this is the case, we will issue a report to ImageMagick as well.

More specifically, we discovered a left shift of negative value in function rdpcm_h_fallback():

libde265/libde265/fallback-dct.cc

Line 208 in 0a9999e

int c = coeffs[x+y*nT] << tsShift;

The issue appears to be caused by a negative value being stored in the coeff array which is then left-shifted. By analyzing both the stacktrace, and the source code, the coeff array seems to be given as an argument to the “current” thread. It is not clear to us whether the function rdpcm_h_fallback() (or, most likely, some function before that) should check for the presence of such an ill-formed input, or if this input should not be provided to this working thread at all (e.g. as per some policy in the documentation).

Steps to Reproduce

In the following archive you will find:

• the input file that caused the bug
• the output of UBSan confirming our finding

Additionally, you will find at this link the compiled fuzzing harness we used to perform the test (we had to share it from outside github as its size exceeds the size limit for this issue).

To reproduce the error, simply run the given binary by providing the testcase as input, with a command similar to ./encoder_heic_fuzzer /path_to_testcases/input

The program has been tested on the standard Docker image provided on OSS-Fuzz using Ubuntu 20.04, providing AFL++ as fuzzing engine and build flag --sanitizer=undefined.

The libde265 hash commit used to perform the tests is 0a9999e, the latest at the time of testing.
The ImageMagick hash commit used to perform the tests is 6299383.

Environment

• OS: Linux
• Version/Distribution: Ubuntu 20.04
• Architecture: x86_64