ARROW-1364: [C++] IPC support machinery for record batch roundtrips to GPU device memory

[wesm]

This additionally does a few things:

• Change libarrow_gpu to use CUDA driver API instead of runtime API
• Adds code for exporting buffers using CUDA IPC on Linux, but this is not yet tested

[wesm]

cc @m1mc @asuhan @seibert @sklam

This patch enables writing record batches to GPU device memory with code like:

std::shared_ptr<CudaBuffer> device_serialized;
ASSERT_OK(arrow::gpu::SerializeRecordBatch(*batch, context_.get(),
                                           &device_serialized));

You can also do zero-copy read from device memory:

Status ReadRecordBatch(const std::shared_ptr<Schema>& schema,
                       const std::shared_ptr<CudaBuffer>& buffer,
                       MemoryPool* pool, std::shared_ptr<RecordBatch>* out);

The returned record batch in this case will have device pointers in its buffers.

We'll need to do a tiny bit of work so that libarrow_gpu can "attach" to existing CUcontext created by others (see ARROW-1423), but I wanted to get this reviewed and merged first.

Additionally, we have the new functions for schema serialization:

Status SerializeSchema(const Schema& schema, MemoryPool* pool,
                       std::shared_ptr<Buffer>* out);

Status ReadSchema(io::InputStream* stream, std::shared_ptr<Schema>* out);

[pcmoritz]

newline

[pcmoritz]

Thanks, this is very exciting!

[xhochy]

+1, LGTM

[wesm]

Thanks. I will resolve the outstanding issues around shared CUDA contexts in the course of making a PR into MapD to simplify their Arrow serialization code

[asuhan]

Thanks, I forward to using this in our product.