ARROW-661: [C++] Add LargeRecordBatch metadata type, IPC support, associated refactoring

[wesm]

This patch enables the following code for writing record batches exceeding 2^31 - 1

RETURN_NOT_OK(WriteLargeRecordBatch(
    batch, buffer_offset, mmap_.get(), &metadata_length, &body_length, pool_));
return ReadLargeRecordBatch(batch.schema(), 0, mmap_.get(), result);

This also does a fair amount of refactoring and code consolidation related to ongoing code cleaning in arrow_ipc.

These APIs are marked experimental. This does add LargeRecordBatch flatbuffer type to the Message union, but I've indicated that Arrow implementations (e.g. Java) are not required to implement this type. It's strictly to enable C++ users to write very large datasets that have been embedded for convenience in Arrow's structured data model.

cc @pcmoritz @robertnishihara

[wesm]

Java builds are failing intermittently due to Maven Central flakiness

[xhochy]

Green build: https://travis-ci.org/wesm/arrow/builds/212816101

[xhochy]

+1, LGTM

[wesm]

@xhochy thanks for merging this patch. Since this changed Message.fbs, want to make sure that @julienledem takes a look and understands the issue.

It would be good to provide a reasonable guarantee that data stored in a RecordBatch can be read by all Arrow implementations. So the two ways to solve this issue could have been:

• Change lengths in FieldNode and RecordBatch from int to long
• OR add the LargeFieldNode and LargeRecordBatch types

Since this is marked experimental we aren't making any forward or API compatibility guarantees on thsi functionality

[julienledem]

This looks fine. If the feature becomes real we should consider just changing the length field to long in FIeldNode and specify the restriction in the metadata that supporting length > 2^31 - 1 is optional.

[wesm]

If others feel that would be acceptable, maintaining less code is always preferable from my perspective. I believe the requirement to store vectors with more than INT32_MAX elements on the C++/Python side is not going to go away.

The downside is that if you encounter a RecordBatch in the wild in Java, you may get an exception if it's too big. I'm not sure how concerning that is.

[julienledem]

I'd recommend that the C++ side does not allow writing vectors with more than INT32_MAX entries by default. You'd have to explicitly enable it. This way people don't inadvertently create things that won't be cross-language compatible.

[wesm]

I'm OK with that. I will open a JIRA about changing the RecordBatch types from int to long

[wesm]

see ARROW-679