[C++] Parquet reader is unable to read LargeString columns

[nicki-dese]

Describe the bug, including details regarding any error messages, version, and platform.

read_parquet() is giving the following error with large parquet files:

Capacity error: array cannot contain more than 2147483646 bytes, have 2147489180

Versions etc from sessionInfo:

• arrow 14.0.0.2
• R version 4.3.0 (2023-04-21 ucrt)
• Platform: x86_64-w64-ming32/x64
• Windows 11 x64 (build 22621)

Descriptive info on example problematic table, with two columns:

• 140 million rows.
• id: large_string, 4.2 Gb
• state: int_32, 0.5 Gb

The id is a hashed string, 24 characters long. It is not practical to change it, as it's the joining key.

Note, the data above is stored as a data.table in R and left that way when saving it with write_parquet(). But I've converted it to an arrow table for the above descriptive stats, because I thought they'd be more useful to you!

Other relevant information:

• The large parquet files were created with arrow::write_parquet()
• The same files previously opened with an earlier version of read_parquet()
  (unfortunately I'm not sure which version, but it was working late November/early December, we work in a closed environment and use Posit Package manager, VMs rebuild every 30 days, so it would have been a fairly recent version)
• I've duplicated the error, and it still occurs with newly created large parquet files, such as the one described above
• Loading the same files with open_dataset() works. However, our team uses targets, which implicitly calls read_parquet, so this bug has unfortunately efffected many of our workflows.

Note: I haven't been able to roll back to an earlier version of arrow - because we only have earlier source versions and not binaries and I'm using windows, I get libarrow errors. If there is a work around for this please let me know.

UPDATE

With the help of IT, I have been able to install earlier versions of arrow in my environment, and have shown that:

• Bug was introduced in arrow version 14.0.0, and it's the read_parquet function that is the issue, not write_parquet.
• With version 13.0.0.1 I was able to open the above problematic file, which was created with write_parquet version 14.0.0.2.

Component(s)

Parquet, R

[assignUser]

Note: I haven't been able to roll back to an earlier version of arrow - because we only have earlier source versions and not binaries and I'm using windows, I get libarrow errors. If there is a work around for this please let me know.

devtools::install_version('arrow', '13.0.0.1') works for me on windows. It does need to download a binary so that might get blocked? If that's the case you can get the binary manually here and then set the envvar RWINLIB_LOCAL to point to the zip file.

[nicki-dese]

@assignUser - thank you! unfortunately the closed environment does not connect to the internet. I will point the IT team who I've asked for help with this internally at that link and see if they'd be willing to test or upload it for me so I can test.

[nicki-dese]

Given that the bug was introduced in version 14.0.0, I am wondering if it relates to #37274

[thisisnic]

@felipecrv Mind giving this a quick skim over? I'm not sure whether this is at the R or C++ layer, but one of the places in the codebase when I grep for the array cannot contain more than error was updated in #35345 which appears to be have merged after this error appeared, so I figure you might be more familar with that area of the codebase than us, even if it's not related to this?

[felipecrv]

@thisisnic sure. I'm going to take a look now.

[felipecrv]

Capacity error: array cannot contain more than 2147483646 bytes, have 2147489180

log(2147483646 + 2, 2) is 31, so this is a check coming from BaseBinaryBuilder<T> which uses sizeof<offset_type> in the memory_limit() method. FixedSizeBinaryBuilder can produce the same message, but it always uses int64_t as these arrays don't need an offsets buffer.

The regular string builder is based on BaseBinaryBuilder<T> and it uses 32-bits for the offsets array, so it requires all the concatenated strings to be addressable with 31 bits (~2GB). But the bug report says the column type is LargeString which uses 64-bit offsets! It shouldn't be a problem.

Digging into the Parquet reader code, I find...

      // XXX: if a LargeBinary chunk is larger than 2GB, the MSBs of offsets
      // will be lost because they are first created as int32 and then cast to int64.

...leading to a commit from November:

commit b4a0751effe316aee1a0fd80fb1c444ecd6842c5
Author:     Antoine Pitrou <...>
AuthorDate: Mon Nov 23 14:53:56 2020 +0100

    ARROW-10426: [C++] Allow writing large strings to Parquet

    Large strings are still read back as regular strings.

    Closes #8632 from pitrou/ARROW-10426-parquet-large-binary

    Authored-by: Antoine Pitrou <antoine@python.org>

This is from PR #8632 that fixed Issue #26405 which is about writing LargeString into Arrow, but doesn't fix the reading them back part. I suppose the value in this is that C++/R/Python scripts can produce files that the Java Parquet reader can read without problems.

Next steps:

• Change this issue to "[C++] Parquet reader is unable to read LargeString columns"
• Get help/tips from @pitrou @mapleFU on how to fix this
• Prepare a fix PR

[thisisnic]

Thanks for figuring that one out @felipecrv. I've updated the title to reflect that it's a C++ bug, though don't have the capacity to implement the fix myself.

[mapleFU]

arrow 14.0.0.2

Do you mean 14.0.2 or other? I did a bug fix here on 14.0.2 ( #38784 ), not sure whether this is related

[thisisnic]

It could be related, we haven't been able to release 14.0.2 to CRAN, so it'll be 14.0.0.2, which is 14.0.0 but with some R patches.