[CI][Python] Conda Python 3.10 jobs fail with UnicodeDecodeError due to gdb issue

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

The CI job AMD64 Conda Python 3.10 Without Pandas has started failing on test_gdb.py with several UnicodeDecodeError invalid continuation byte.
Example of job failures from different PRs and main:

• https://github.com/apache/arrow/actions/runs/14878658706/job/41781302772?pr=46261
• https://github.com/apache/arrow/actions/runs/14878500696/job/41780789650
• https://github.com/apache/arrow/actions/runs/14876650269/job/41775137379

The log is quite long so I am adding only one of the test failures but there are several so please check the output of the jobs:

______________________________ test_scalars_heap _______________________________

gdb_arrow = <pyarrow.tests.test_gdb.GdbSession object at 0x7f722c8add10>

    def test_scalars_heap(gdb_arrow):
>       check_heap_repr(gdb_arrow, "heap_null_scalar", "arrow::NullScalar")

opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_gdb.py:728: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_gdb.py:242: in check_heap_repr
    s = gdb.print_value(f"*{expr}")
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_gdb.py:143: in print_value
    out = self.run_command(f"p {expr}")
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_gdb.py:137: in run_command
    return self.wait_until_ready()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyarrow.tests.test_gdb.GdbSession object at 0x7f722c8add10>

    def wait_until_ready(self):
        """
        Record output until the gdb prompt displays.  Return recorded output.
        """
        # TODO: add timeout?
        while (not self.last_stdout_line.startswith(b"(gdb) ") and
               self.proc.poll() is None):
            block = self.proc.stdout.read(4096)
            if self.verbose:
                sys.stdout.buffer.write(block)
                sys.stdout.buffer.flush()
            block, sep, last_line = block.rpartition(b"\n")
            if sep:
                self.last_stdout.append(self.last_stdout_line)
                self.last_stdout.append(block + sep)
                self.last_stdout_line = last_line
            else:
                assert block == b""
                self.last_stdout_line += last_line
    
        if self.proc.poll() is not None:
            raise IOError("gdb session terminated unexpectedly")
    
>       out = b"".join(self.last_stdout).decode('utf-8')
E       UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 46: invalid continuation byte

opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_gdb.py:122: UnicodeDecodeError
----------------------------- Captured stdout call -----------------------------
p *heap_null_scalar
_______________________________ test_array_data ________________________________

Component(s)

Continuous Integration, Python

[raulcd]

This seems to also fail on the nightlies and weirdly enough it seems to only fail on the conda jobs for Python 3.10:

• test-conda-python-3.10
• test-conda-python-3.10-pandas-latest-numpy-latest

@pitrou any idea what could be the issue?

[pitrou]

I'll have to look at it, but it's probably a change in how some gdb versions display non-ASCII string values...

[pitrou]

Someone else could try to debug of course. Perhaps launch the Docker container, run a bash session inside it, and try to reproduce the gdb steps from the tests to understand what happens.

[kapoisu]

Could anyone please elaborate on the standard process for testing in this scenario?

I am currently using export PYTHON=3.10 && archery docker run compose run conda-python bash to build the image and enter interactive mode. After manually running /arrow/ci/scripts/cpp_build.sh /arrow /build, it seems that during the configure step, the current directory is not correctly switched to /build/cpp, which results in the CMake error: Error: could not load cache.

[raulcd]

I've been able to reproduce with:

PYTHON=3.10 archery docker run conda-python bash
$ /arrow/ci/scripts/cpp_build.sh /arrow /build
...
$ /arrow/ci/scripts/python_build.sh /arrow /build
...
$ /arrow/ci/scripts/python_test.sh /arrow
...
    
        if self.proc.poll() is not None:
            raise IOError("gdb session terminated unexpectedly")
    
>       out = b"".join(self.last_stdout).decode('utf-8')
E       UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 46: invalid continuation byte

opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_gdb.py:122: UnicodeDecodeError
...

Not entirely sure what is the error you were facing, could you share more details?

[amoeba]

It looks like the failing job is using gdb 16.3. In the 16.3 NEWS file, I see only one change that looks related,

• GDB now has support for multi-byte and wide character sets on the target. Strings whose character type is wchar_t, char16_t, or char32_t are now correctly printed. GDB supports wide- and unicode- literals in C, that is, L'x', L"string", u'x', u"string", U'x', and U"string" syntax. And, GDB allows the "%ls" and "%lc" formats in `printf'. This feature requires iconv to work properly; if your system does not have a working iconv, GDB can use GNU libiconv. See the installation instructions for more information.

From @bkietz's comment in #46180, is what this patch does technically more correct and worth testing?

diff --git i/python/pyarrow/src/arrow/python/gdb.cc w/python/pyarrow/src/arrow/python/gdb.cc
index 7c58bae334..9251dedfdd 100644
--- i/python/pyarrow/src/arrow/python/gdb.cc
+++ w/python/pyarrow/src/arrow/python/gdb.cc
@@ -307,12 +307,12 @@ void TestSession() {
   StringScalar string_scalar_null{};
   StringScalar string_scalar_unallocated{std::shared_ptr<Buffer>{nullptr}};
   StringScalar string_scalar_empty{Buffer::FromString("")};
-  StringScalar string_scalar_hehe{Buffer::FromString("héhé")};
+  StringScalar string_scalar_hehe{Buffer::FromString(u8"héhé")};
   StringScalar string_scalar_invalid_chars{
       Buffer::FromString(std::string("abc") + '\x00' + "def\xffghi")};

   LargeBinaryScalar large_binary_scalar_abc{Buffer::FromString("abc")};
-  LargeStringScalar large_string_scalar_hehe{Buffer::FromString("héhé")};
+  LargeStringScalar large_string_scalar_hehe{Buffer::FromString(u8"héhé")};

   FixedSizeBinaryScalar fixed_size_binary_scalar{Buffer::FromString("abc"),
                                                  fixed_size_binary(3)};
@@ -458,7 +458,7 @@ void TestSession() {
   const char* json_binary_array = "[null, \"abcd\", \"\\u0000\\u001f\xff\"]";
   auto heap_binary_array = SliceArrayFromJSON(binary(), json_binary_array);
   auto heap_large_binary_array = SliceArrayFromJSON(large_binary(), json_binary_array);
-  const char* json_string_array = "[null, \"héhé\", \"invalid \xff char\"]";
+  const char* json_string_array = u8"[null, \"héhé\", \"invalid \xff char\"]";
   auto heap_string_array = SliceArrayFromJSON(utf8(), json_string_array);
   auto heap_large_string_array = SliceArrayFromJSON(large_utf8(), json_string_array);
   auto heap_binary_array_sliced = heap_binary_array->Slice(1, 1);

[amoeba]

I am wondering now if something else is going on: I get different gdb print behavior between system (debian sid) and conda. Both versions are 16.3. For this source file,

#include <iostream>
#include <string>

int main() {
	std::string mystr = "héhé";
	std::cout << mystr << std::endl;

	return 0;
}

with system gdb 16.3-1, I get,

(gdb) print mystr
$1 = "héhé"

with conda gdb 16.3, I get,

(gdb) print mystr
$1 = {_M_dataplus = {<std::allocator<char>> = {<std::__new_allocator<char>> = {<No data fields>}, <No data fields>}, 
    _M_p = 0x7fffffffdf50 "h\303\251h\303\251"}, _M_string_length = 6, {
    _M_local_buf = "h\303\251h\303\251\000l\230C\345\367\377\177\000", _M_allocated_capacity = 7782406812835890024}}

[amoeba]

My two comments above might be unrelated to the issue at hand. When I jump into pdb after a test failure, I get,

(Pdb) self.last_stdout
[b'', b'$138 = arrow::StringScalar of size 6, value "h\xe9h\xe9"\n']

which just looks like latin-1 encoding.

[kapoisu]

Not entirely sure what is the error you were facing, could you share more details?

Never mind... I think I might have messed up some steps in the process. I followed your steps exactly and managed to reproduced it.

I am wondering now if something else is going on: I get different gdb print behavior between system (debian sid) and conda. Both versions are 16.3. For this source file,

This is due to the libstdcxx printer (comes along with gcc) not being registered in the conda environment's gdb.

[amoeba]

Where I'm at with this now is that my the container appears to have a minimal UTF-8 locale,

root@7cf632713c34:/# locale -a
C
C.utf8
POSIX

but conda gdb only has Latin-1 registered:

(gdb) set host-charset 
ISO-8859-1  auto 

[amoeba]

If I roll conda-provided gdb from 16.3 back one release to 15.1, the tests pass and show host-charset in gdb shows a full list of available encodings. I don't see an issue already filed on the feedstock so I'll go see if something might have changed with that.

One possible fix here is just to pin gdb<16.3.