Reading parquet file behavior change from 13.0.0 to 14.0.0

[jhwang7628]

Describe the bug, including details regarding any error messages, version, and platform.

Hi,

We have a parquet that used to read fine in 13.0.0, but now I got an error when calling via pandas.read_parquet using 14.0.0. The relevant error is:

  File "/opt/venv/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 3003, in read_table
    return dataset.read(columns=columns, use_threads=use_threads,   
  File "/opt/venv/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 2631, in read
    table = self._dataset.to_table(  
  File "pyarrow/_dataset.pyx", line 556, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 3713, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2148480400

Is this an intended behavior? I skimmed through the changelog but did not find this. Thanks.

Component(s)

Python

[mapleFU]

Hmmm would you mind provide the file or a minimal reproduce way? A bit hard to check the scanner change without the data or logging..

[alexeyche]

Also noticed this, though hard to reproduce it without sharing data as of now, I might find time in future. In short it's just big dataset with nested arrays of floats

[mapleFU]

#38621

A bug is found about reading parquet in python read_table calls more requests than expected. But I don't now if this is related to the issue.

@alexeyche Would you mind provide some info about how to reproduce the issue here?

[jhwang7628]

I cannot provide the data blob as it is company internal data. The issue is consistent and deterministic on my side, so I think maybe any large parquet will do?

[mapleFU]

@jhwang7628 After go though the code, I think this might related to #38437

I'll try to check this in this week. This might trable when you read a large binary column, after this patch it might reserve more data than expected.

[mapleFU]

@jhwang7628 @alexeyche Would you mind check that whether you're using dictionary encoding for string/binary column?

[jorisvandenbossche]

I cannot provide the data blob as it is company internal data. The issue is consistent and deterministic on my side, so I think maybe any large parquet will do?

Would you be able to try to create a file that has the same characteristics as your internal data file but with random data? (eg approximately the same types, and number of rows and size)
That would help a lot in evaluating if we are actually fixing this issue with #38784

Or otherwise, giving some more information about the data types being used, the encoding and row group sizes in the Parquet file, etc could also help to reproduce it on our side.

[jorisvandenbossche]

From seeing the potential fix in #38784, I could manage to create a simple reproducer:

Creating this file with pyarrow 13.0 reads fine with that version:

import string
import numpy as np
import pyarrow as pa

# column with >2GB data
data = ["".join(np.random.choice(list(string.ascii_letters), n)) for n in np.random.randint(10, 500, size=10_000)]
table = pa.table({'a': pa.array(data*1000)})

import pyarrow.parquet as pq
pq.write_table(table, "test_capacity.parquet")

but reading with pyarrow 14:

import pyarrow.parquet as pq
pf = pq.ParquetFile("test_capacity.parquet")

In [6]: pf.read()
...
ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2148282365
/home/joris/scipy/repos/arrow/cpp/src/arrow/array/builder_binary.h:332  ValidateOverflow(elements)
/home/joris/scipy/repos/arrow/cpp/src/parquet/encoding.cc:1202  acc_->builder->ReserveData( std::min<int64_t>(*estimated_data_length, ::arrow::kBinaryMemoryLimit))
/home/joris/scipy/repos/arrow/cpp/src/parquet/encoding.cc:1407  helper.Prepare(len_)
/home/joris/scipy/repos/arrow/cpp/src/parquet/arrow/reader.cc:109  LoadBatch(batch_size)
/home/joris/scipy/repos/arrow/cpp/src/parquet/arrow/reader.cc:1252  ReadColumn(static_cast<int>(i), row_groups, reader.get(), &column)
/home/joris/scipy/repos/arrow/cpp/src/parquet/arrow/reader.cc:1233  fut.MoveResult()