PARQUET-712: Add library to read into Arrow memory

[xhochy]

At the moment this is just move of the existing code into the state where it compiles. Outstanding work includes:

• Understand the issues with ParquetException typeid matching in the tests on macOS. @wesm We already had this problem and you fixed it somewhere. Do you remember the solution?
• Understand why BoolenType tests break with a Thrift exception
• Add functions that read directly into Arrow memory and not intermediate structures.

[xhochy]

The failing BooleanType test is not because of arrow but there seems to be a general problem inside parquet-cpp. Will make a PR with a separate failing test.

[wesm]

@xhochy I thought I fixed the ParquetException issue in 616305c, but it is recurring?

[wesm]

I think we can leave item 3 for a follow up JIRA.

The output for OS X (all the Thrift linker warnings) looks worrisome, not sure how to fix that off hand

[wesm]

TODO: building static libparquet_arrow.a

[wesm]

@xhochy my understanding from google searches is that throwing a shared library's exception types outside its boundaries can be unreliable. So we may need to come up with a different way for Parquet library implementers to signal failure of an interface extension type's methods to parquet-cpp

[wesm]

@xhochy as a workaround you can add a protected API to the base classes that allow subclasses in other shared libraries to throw ParquetException without using it directly. e.g.

in subclass

if (... /* error */) {
  RaiseException("it failed");
}

in ParquetAllocator (etc.)

void RaiseException(const std::string& msg) {
  throw ParquetException(msg);
}

[wesm]

Or maybe add the exception thrower function to the public API

[wesm]

Why is the output of the unit tests not visible? We should fix the Travis CI here to be like Arrow so we can see which test failed.

Does my solution to the exception issue work? I think this is the only offending line that needs to be replaced with a function call from libparquet.so that actually throws the exception:

https://github.com/apache/parquet-cpp/pull/158/files#diff-92da47ba6e6cd44f28603464bbbe66d2R47

[wesm]

Just have to add -VV to the ctest call

https://github.com/apache/arrow/blob/master/ci/travis_script_cpp.sh#L19

[xhochy]

Locally I encounter #161 now with a static build for parquet_arrow enabled, thus that one should go in first.

[wesm]

I can merge this whenever the build is green

[xhochy]

There are still open issues I have to fix, so please wait until I remove the WIP prefix.

[xhochy]

With correct static linkage, we don't have the ParquetException problems. I'll open a JIRA for faster reads and a benchmarks for Arrow-Parquet I/O. This PR is so far done.

@wesm conda builds are broken. Not sure if we can fix them / we want to fix them as they probably involve a cyclic Arrow<->Parquet dependency.

[wesm]

@xhochy I'm going to set up conda builds in conda-forge with pinned git revisions -- the builds here were a stop gap so I'll remove them once conda-forge is all working

[wesm]

+1, thank you! let's wait to remove arrow_parquet from Arrow until ARROW-280 is merged

[pitrou]

@xhochy Can you explain why UINT_32 is not emitted for Parquet version 1.0? There's zero comment and I can't find any explanation otherwise.

[pitrou]

Ah, sorry, I saw the comment elsewhere:

        // Parquet 1.0 reader cannot read the UINT_32 logical type. Thus we need
        // to use the larger Int64Type to store them lossless.

[pitrou]

But it's still not very informative. Which "Parquet 1.0 reader" is it? parquet-mr? parquet-cpp? A particular version?

[wesm]

It seems it is unclear about when it's acceptable to use the unsigned integer types (whether the "1.0" format version versus "2.0" format version -- here 1.0 refers to the Parquet format version).