ARROW-3803: [C++/Python] Merge C++ builds and tests, run Python tests in separate CI entries

[wesm]

I found unfortunately that the conda-forge boost-cpp package is not fully compatible with Xcode 8.3, see https://issues.apache.org/jira/browse/ARROW-4056

We might have to build a vendored Boost in this CI entry to work around the problem (this is what the Ray project did when they also hit this issue)

[wesm]

We may be reaching the end of the road with Xcode 6.4. Travis is pulling support for the image in January 2019 according to messages in the build logs.

Gandiva does not run properly (due to a strptime bug?) on 6.4 so we might move everything to 8.3. We should see what conda-forge is planning exactly, since if they are remaining on 6.4 then we may not be able to have Gandiva in the Python wheels

cc @pravindra @praveenbingo

[pravindra]

@wesm current change lgtm i.e CI has gandiva with Xcode 8.3 but not Xcode 6.4.

If gandiva with Xcode 6.4 becomes important/required for any reason, we can spend some time on workarounds to make it work.

[wesm]

I opened https://issues.apache.org/jira/browse/ARROW-4068 about Xcode 6.4. If you know what problems there are, could you post them there? I'm working on getting a reliable macOS box where I can do development and testing

[wesm]

I ran into https://issues.apache.org/jira/browse/ARROW-4070 here when trying to use ARROW_BOOST_VENDORED with ninja

[fsaintjacques]

LD_LIBRARY_PATH is less invasive.

[wesm]

LD_LIBRARY_PATH is already set...

https://github.com/apache/arrow/blob/master/ci/travis_env_common.sh#L63

[wesm]

I think this is almost there. I'm just waiting on the Gandiva valgrind fixes to go in. If folks could take pity on me and try to not touch the CI scripts any more until this is merged that would be really great

[codecov-io]

Codecov Report

Merging #3208 into master will increase coverage by 20.58%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #3208       +/-   ##
==========================================
+ Coverage   67.62%   88.2%   +20.58%     
==========================================
  Files          58     550      +492     
  Lines        4012   73840    +69828     
==========================================
+ Hits         2713   65132    +62419     
- Misses       1192    8601     +7409     
  Partials      107     107

Impacted Files	Coverage Δ
python/pyarrow/ipc.pxi	67.53% <0%> (ø)
cpp/src/arrow/csv/chunker-test.cc	100% <0%> (ø)
cpp/src/parquet/column_page.h	89.28% <0%> (ø)
cpp/src/parquet/bloom_filter-test.cc	99.13% <0%> (ø)
cpp/src/arrow/array/builder_decimal.cc	100% <0%> (ø)
cpp/src/plasma/client.cc	96.02% <0%> (ø)
cpp/src/arrow/io/test-common.h	97.29% <0%> (ø)
cpp/src/gandiva/function_registry.h	100% <0%> (ø)
cpp/src/arrow/util/int-util-test.cc	100% <0%> (ø)
cpp/src/arrow/python/io.cc	95.72% <0%> (ø)
... and 482 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25b6a6c...32cffe0. Read the comment docs.

[pravindra]

I opened https://issues.apache.org/jira/browse/ARROW-4068 about Xcode 6.4. If you know what problems there are, could you post them there? I'm working on getting a reliable macOS box where I can do development and testing

will do. thanks.

[wesm]

Rebased. Will merge once the build runs

[wesm]

Running all the C++ unit tests under valgrind is taking about 23 minutes

Label Time Summary:
arrow-tests      = 369.38 sec*proc (46 tests)
gandiva-tests    = 575.75 sec*proc (27 tests)
parquet-tests    = 458.77 sec*proc (18 tests)
plasma-tests     =  14.58 sec*proc (2 tests)
unittest         = 1418.48 sec*proc (93 tests)

This is OK for now, but may not be scalable.

[wesm]

Overall this patch cuts about 10 minutes out of build times

Will merge as soon as Appveyor is looking ok

[wesm]

The Appveyor builds passed (Rust is still running -- I opened https://issues.apache.org/jira/browse/ARROW-4080)