[C++][Parquet] Parquet FileMetaData key_value_metadata not always mapped to Arrow Schema metadata

[asfimport]

Context: I ran into this issue when reading Parquet files created by GDAL (using the Arrow C++ APIs, OSGeo/gdal#5477), which writes files that have custom key_value_metadata, but without storing ARROW:schema in those metadata (cc @paleolimbot)

—

Both in reading and writing files, I expected that we would map Arrow Schema::metadata with Parquet FileMetaData::key_value_metadata. But apparently this doesn't (always) happen out of the box, and only happens through the "ARROW:schema" field (which stores the original Arrow schema, and thus the metadata stored in this schema).

For example, when writing a Table with schema metadata, this is not stored directly in the Parquet FileMetaData (code below is using branch from ARROW-16337 to have the store_schema keyword):

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({'a': [1, 2, 3]}, metadata={"key": "value"})
pq.write_table(table, "test_metadata_with_arrow_schema.parquet")
pq.write_table(table, "test_metadata_without_arrow_schema.parquet", store_schema=False)

# original schema has metadata
>>> table.schema
a: int64
-- schema metadata --
key: 'value'

# reading back only has the metadata in case we stored ARROW:schema
>>> pq.read_table("test_metadata_with_arrow_schema.parquet").schema
a: int64
-- schema metadata --
key: 'value'
# and not if ARROW:schema is absent
>>> pq.read_table("test_metadata_without_arrow_schema.parquet").schema
a: int64

It seems that if we store the ARROW:schema, we also store the schema metadata separately. But if store_schema is False, we also stop writing those metadata (not fully sure if this is the intended behaviour, and that's the reason for the above output):

# when storing the ARROW:schema, we ALSO store key:value metadata
>>> pq.read_metadata("test_metadata_with_arrow_schema.parquet").metadata
{b'ARROW:schema': b'/////7AAAAAQAAAAAAAKAA4ABgAFAA...',
 b'key': b'value'}
# when not storing the schema, we also don't store the key:value
>>> pq.read_metadata("test_metadata_without_arrow_schema.parquet").metadata is None
True

On the reading side, it seems that we generally do read custom key/value metadata into schema metadata. We don't have the pyarrow APIs at the moment to create such a file (given the above), but with a small patch I could create such a file:

# a Parquet file with ParquetFileMetaData::metadata that ONLY has a custom key
>>> pq.read_metadata("test_metadata_without_arrow_schema2.parquet").metadata
{b'key': b'value'}

# this metadata is now correctly mapped to the Arrow schema metadata
>>> pq.read_schema("test_metadata_without_arrow_schema2.parquet")
a: int64
-- schema metadata --
key: 'value'

But if you have a file that has both custom key/value metadata and an "ARROW:schema" key, we actually ignore the custom keys, and only look at the "ARROW:schema" one.
This was the case that I ran into with GDAL, where I have a file with both keys, but where the custom "geo" key is not also included in the serialized arrow schema in the "ARROW:schema" key:

# includes both keys in the Parquet file
>>> pq.read_metadata("test_gdal.parquet").metadata
{b'geo': b'{"version":"0.1.0","...',
 b'ARROW:schema': b'/////3gBAAAQ...'}
# the "geo" key is lost in the Arrow schema
>>> pq.read_table("test_gdal.parquet").schema.metadata is None
True

Reporter: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [C++][Parquet] Field-level metadata are not supported? (ColumnMetadata.key_value_metadata) (is related to)

_{Note: This issue was originally created as ARROW-16339. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Note: I recently created an issue about field-level metadata (ARROW-15548), but so this issue is about schema-level (for Arrow) / file-level (for Parquet) metadata.

The above is a long description with examples, but trying to summarize the findings, and questions to answer:

• Do we generally want to map the schema-level metadata from Arrow with Parquet file-level metadata? (I think the answer is yes?)

• When reading, and the file metadata does not contain a "ARROW:schema" key, we actually already do map the Parquet file metadata to resulting Arrow schema metadata (this is OK)

• When writing, the store_schema flag seems to also influence whether we store schema metadata key/values in the Parquet file. This might be a bug? (or at least unintended behaviour?)

• When reading, and the file metadata does contain both an "ARROW:schema" key and other keys, we ignore the other keys. Should we merge the keys from the metadata in the serialized "ARROW:schema" schema with the other keys in the Parquet FileMetaData key_value_metadata? (those could of course be duplicative / conflicting)

  cc @pitrou @emkornfield
   

[asfimport]

Micah Kornfield / @emkornfield:
Q1: I also think the answer is yes.

Q2: Yes, that seems correct to me as well.  I think the assumption is Arrow didn't produce this but KV metadata is still useful.

Q3: Yes, it seems like the two things should be separate flags.

Q4: Not sure on this one, it is a little surprising they aren't already covered with the Arrow Schema?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@emkornfield Thanks for the input!

Q4: Not sure on this one, it is a little surprising they aren't already covered with the Arrow Schema?

Yes, for data written using the standard Arrow mechanisms, that typically won't happen. But so this is exactly the case that I ran into while reading Parquet files created by GDAL (using the Arrow C++ APIs, OSGeo/gdal#5477). By using the Arrow C++ APIs, they do create a Parquet file with an ARROW:schema metadata field, but they also add additional (geospatial) metadata, and add this directly to the Parquet file metadata, instead of adding it to the Arrow schema that is getting written.
From GDAL's point of view, I think it makes sense that they add it directly to the Parquet file metadata (since this metadata is also meant to be read by other, non-arrow readers).

I think for this case if the serialized "ARROW:schema" schema does not contain custom metadata fields, but the Parquet file metadata does, I would say we can clearly preserve the ones in the Parquet file metadata.
It is only when both contain key/value metadata that there might be conflicts, and that it is less clear what to do.

—

That also raises yet another question, related to writing:

Q5: If we are writing data with schema-level metadata (and we decided to map this to Parquet file-level metadata, Q1 above), do we then drop this metadata fields from the original schema as how it is serialized into "ARROW:schema"?
Because writing both would exactly give such a case of duplicated metadata keys (but not writing both (only in the actual parquet file metadata fields, dropping from the "ARROW:schema" field) would result in files that don't read any metadata with existing versions of Arrow, because we currently ignore other keys if an "ARROW:schema" key is present, i.e. Q4 above)

[asfimport]

Antoine Pitrou / @pitrou:
Q1, Q2: agreed with @emkornfield

Q3: no idea about this

Q4: it would sound reasonable to merge metadata, with Arrow keys taking precedence over Parquet keys (if Parquet defines "key": "foo" and Arrow defines "key": "bar", keep only "key": "bar").