Skip to content

ARROW-661: [C++] Add LargeRecordBatch metadata type, IPC support, associated refactoring #404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wesm wants to merge 9 commits into from
Closed

ARROW-661: [C++] Add LargeRecordBatch metadata type, IPC support, associated refactoring #404

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e8f8973
Split adapter.h/cc into reader.h/writer.h. Draft LargeRecordBatch type
wesm Mar 19, 2017
0f2722c
Consolidate metadata-internal.h into metadata.h. Use own Arrow struct…
wesm Mar 19, 2017
85d1a1c
Add (untested) metadata writer for LargeRecordBatch
wesm Mar 19, 2017
f4c8830
Consolidate ipc-metadata-test and ipc-read-write-test and draft large…
wesm Mar 19, 2017
4c1d08c
Refactoring, failing test fixture for large record batch
wesm Mar 19, 2017
36c3862
Get LargeRecordBatch round trip working. Add to Message union for now
wesm Mar 19, 2017
179a1e3
Add unit test for large record batches. Use bytewise comparisons with…
wesm Mar 19, 2017
d7811f2
cpplint
wesm Mar 19, 2017
9c18a95
Fix import ordering
wesm Mar 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cpp/src/arrow/allocator-test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
// under the License.

#include "gtest/gtest.h"

#include "arrow/allocator.h"
#include "arrow/test-util.h"

Expand Down
1 change: 1 addition & 0 deletions cpp/src/arrow/allocator.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
#include <cstddef>
#include <memory>
#include <utility>

#include "arrow/memory_pool.h"
#include "arrow/status.h"

Expand Down
18 changes: 18 additions & 0 deletions cpp/src/arrow/io/test-common.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,24 @@
namespace arrow {
namespace io {

static inline Status ZeroMemoryMap(MemoryMappedFile* file) {
constexpr int64_t kBufferSize = 512;
static constexpr uint8_t kZeroBytes[kBufferSize] = {0};

RETURN_NOT_OK(file->Seek(0));
int64_t position = 0;
int64_t file_size;
RETURN_NOT_OK(file->GetSize(&file_size));

int64_t chunksize;
while (position < file_size) {
chunksize = std::min(kBufferSize, file_size - position);
RETURN_NOT_OK(file->Write(kZeroBytes, chunksize));
position += chunksize;
}
return Status::OK();
}

class MemoryMapFixture {
public:
void TearDown() {
Expand Down
15 changes: 2 additions & 13 deletions cpp/src/arrow/ipc/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,10 @@ set(ARROW_IPC_TEST_LINK_LIBS
arrow_io_static)

set(ARROW_IPC_SRCS
adapter.cc
feather.cc
json.cc
json-internal.cc
metadata.cc
metadata-internal.cc
reader.cc
writer.cc
)
Expand Down Expand Up @@ -64,16 +62,8 @@ ADD_ARROW_TEST(feather-test)
ARROW_TEST_LINK_LIBRARIES(feather-test
${ARROW_IPC_TEST_LINK_LIBS})

ADD_ARROW_TEST(ipc-adapter-test)
ARROW_TEST_LINK_LIBRARIES(ipc-adapter-test
${ARROW_IPC_TEST_LINK_LIBS})

ADD_ARROW_TEST(ipc-file-test)
ARROW_TEST_LINK_LIBRARIES(ipc-file-test
${ARROW_IPC_TEST_LINK_LIBS})

ADD_ARROW_TEST(ipc-metadata-test)
ARROW_TEST_LINK_LIBRARIES(ipc-metadata-test
ADD_ARROW_TEST(ipc-read-write-test)
ARROW_TEST_LINK_LIBRARIES(ipc-read-write-test
${ARROW_IPC_TEST_LINK_LIBS})

ADD_ARROW_TEST(ipc-json-test)
Expand Down Expand Up @@ -148,7 +138,6 @@ add_dependencies(arrow_ipc_objlib metadata_fbs)

# Headers: top level
install(FILES
adapter.h
api.h
feather.h
json.h
Expand Down
Loading