Internal UTF / Python error in fuzzer

[kripken]

Testcase:

(module
 (type $0 (func (result stringref)))
 (export "execute" (func $0))
 (func $0 (result stringref)
  (string.const "\ed\a0\80\e2\82\ac\c2\a3")
 )
)

STR:

$ bin/wasm-opt out/test/w.wasm -all --fuzz-exec-before &> x
$ cat x
[fuzz-exec] calling execute
[fuzz-exec] note result: execute => string("€£")
warning: no passes specified, not doing any work
warning: no output file specified, not emitting output
$ python3
Python 3.11.7
Type "help", "copyright", "credits" or "license" for more information.
>>> open('x').read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 72: invalid continuation byte

When running in the fuzzer, this is the full stack trace:

!
-----------------------------------------
Exception:
  File "binaryen/./scripts/fuzz_opt.py", line 1733, in <module>
    total_wasm_size += test_one(raw_input_data, given_wasm)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "binaryen/./scripts/fuzz_opt.py", line 1470, in test_one
    testcase_handler.handle_pair(input=random_input, before_wasm=abspath('a.wasm'), after_wasm=abspath('b.wasm'), opts=opts + FEATURE_OPTS)
  File "binaryen/./scripts/fuzz_opt.py", line 760, in handle_pair
    run([in_bin('wasm-opt'), before_wasm] + opts + ['--fuzz-exec'])
  File "binaryen/./scripts/fuzz_opt.py", line 97, in run
    return subprocess.check_output(cmd, stderr=stderr, text=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1196, in communicate
    stdout = self.stdout.read()
             ^^^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
-----------------------------------------
!
utf-8
b'[fuzz-exec] calling execute\n[fuzz-exec] note result: execute => string("\xed\xa0\x80\xe2\x82\xac\xc2\xa3")\n[fuzz-exec] calling execute\n[fuzz-exec] note result: execute => string("\xed\xa0\x80\xe2\x82\xac\xc2\xa3")\n[fuzz-exec] comparing execute\n'
72
73
invalid continuation byte

So it looks like running subprocess.check_output(cmd, stderr=stderr, text=True) is the issue. Adding , encoding='utf-8' to that command (what a quick google suggested) does not help. Removing , text=True also does not help (if it did we could convert manually afterwards, was my idea).

Is that not UTF-8 @tlively ?

[tlively]

It's valid WTF-8, but it's not valid UTF-8 because the first code point is a surrogate.

[kripken]

Should we avoid emitting that in the fuzzer, then? Or if we do want to fuzz it, should we escape it somehow when printing the string?

[tlively]

It would be good to escape it in the fuzz output, I think. In general it's important to be fuzzing with strings that are not valid unicode.

[kripken]

I'm not sure how to fix this in the fuzzer specifically, because the fuzzer isn't able to read the output from wasm-opt --fuzz-exec. We could do it in the Literal printing logic for strings, which is what the fuzzer is reading, though I'm not sure if that would affect roundtripping - what do you think?

Btw I see this constantly now when I fuzz - several times a day at least. I'm reverting #6430 locally in order to be able to continue to fuzz for now.

[tlively]

There's no way to round trip the fuzz exec output anyway, so I think any kind of escaping we can do both in C++ and in the JS fuzz wrapper should be fine, right?

[kripken]

Fuzz exec output is just showing the output of printing a Literal. We do use the printed output of Literals in the text format as well, which we do need to roundtrip. Are you suggesting that --fuzz-exec mode should have custom printing code instead, something like that?

[tlively]

For strings, at least, the way we print them in Print.cpp is unrelated to how we print them when they are Literals. We essentially already have that custom printing code.

[kripken]

Am I misreading the code, then?

binaryen/src/passes/Print.cpp

Lines 878 to 880 in 47bcca6

	void visitConst(Const* curr) {
	o << curr->value.type << ".const " << curr->value;
	}

That does o << curr->value using a std::ostream and a Literal, so I think it uses the same Literal printing as --fuzz-exec, which is here:

binaryen/src/tools/execution-results.h

Line 152 in 47bcca6

std::cout << value << '\n';

?

[tlively]

StringConst != Const!

[tlively]

I'm working on a fix, btw.