ARROW-82: Initial IPC support for ListArray

[emkornfield]

This is a work in progress because I can't get clang-tidy to shut-up on parameterized test files (see last commit which I need to revert before merge). I'd like to make sure this is a clean build and make sure people are ok with these change. This PR also has a lot of collateral damage for small/large things I cleaned up my way to make this work. I tried to split the commits up logically but if people would prefer separate pull requests I can try to do that as well.

Open questions:

1. For supporting strings, binary, etc. I was thinking of changing thei4 type definitions to inherit from ListType, and to hard-code the child type. This would allow for simpler IPC code (all of the instantiation of types would happen in construct.h/.cc?) vs handling each of there types separately for IPC.
2. There are some TODOs I left sprinkled in the code and would like peoples thoughts on if they are urgent/worthwhile for following up on.

Open issues:

1. Supporting the rest of the List backed logical types
2. More unit tests for added functionality.

As part of this commit I also refactored the Builder interfaces a little bit for the following reasons:

1. It seems that if ArrayBuilder owns the bitmap it should be responsible for having methods to manipulate it.
2. This allows ListBuilder to use the parent class + a BufferBuilder instead of inheriting Int32Builder, which means it doesn't need to do strange length/capacity hacks.

Other misc things here:

1. Native popcount in test-util.h
2. Ability to build a new list on top of an existing by incrementally add offsets/sizes
3. Added missing types primitive types in construct.h for primitive.

[wesm]

Awesome, I will review in detail when I can.

Also pinging @jacques-n on code reviews -- can we figure out a code review solution that is less horrible than GitHub (I hate to be so negative but: really)? I'd love to use gerrithub.io but may require blessing from ASF Infra. I could ask Cloudera folks to set up a public Gerrit for us, too, but I'd rather figure out something that runs under the ASF umbrella.

[jacques-n]

Our options are really:

1. Reviewboard (official Apache solution)
2. Github (somewhat Apache supported)
3. External Gerrit run by someone (that pushes to mailing list) similar to what Kudu does.

See here for more sadness: https://issues.apache.org/jira/browse/INFRA-2205

I'm inclined towards (3) and fall back to (1). (2) is actually the worse in my experience.

[wesm]

Are you all (in particular @emkornfield) cool if I ask for a gerrit.cloudera.org to be set up? I asked ASF Infra for GerritHub but I'm not that optimistic.

If there is another way we can set up a Gerrit for Arrow that would be great too. The ultimate would be getting the ASF to manage it (unfortunately, people have been "losing karma" for using external code review tools).

[emkornfield]

I don't understand all the implications for moving to Gerrit. A few questions (this might be a better topic for the ML?):

1. Is the desire for Gerrit simply as a better UI, or does it improve workflows in some way (looking at the ticket Jacques referenced it seems like the answer is "Yes")?
2. Does gerrit.cloudera.org start handling the CI? Does it make it easier to do things, like tracking performance benchmarks (I remember the cribbed run-tests.sh, mentioning failed builds get uploaded someplace else)? Any notion of how much work it is to modify our CI to work with Gerrit? How much on going support is required from cloudera for modifying pre-commit hooks?
3. Does adding another tool into the mix discourage or add friction to contributions?

On a UI level, I have a preference for ReviewBoard over Gerrit but that might just be a matter of familiarity. I've never used Gerrit in a code review workflow only perused a few open source projects using it, so I suppose it might grow on me.

[emkornfield]

By the way I was thinking it would be nice to make Resize and Init virtual methods, I think it would reduce code repetition and potential bugs for classes the don't have any reason to override Reserve. Thoughts?

[wesm]

Sounds good to me, go ahead

[wesm]

To your questions, briefly:

1. it's a significantly improved workflow. Gerrit makes large code reviews transactional -- it's much more organized and things don't fall through the cracks like they do in GitHub.

2. it'd be easier in the short term to keep running builds with Travis CI -- so there would be some finagling to push updated patches to a PR branch as well as Gerrit (for reviews)

3. I don't think we would need to require Gerrit for anything right now, so patches would still start with a pull request and we could have reviews for larger patches (> 50 lines or whatever) go to Gerrit if it will be deemed helpful.

[wesm]

Let me open a ML discussion about this as I'm sure others will have opinions.

[wesm]

Reserve may return error status

[emkornfield]

Oops. I wonder how hard it would be write a clang-tidy rule to always ensure methods with a Status field are checked.

[wesm]

Kudu wrote an LLVM static analyzer for this https://github.com/apache/incubator-kudu/tree/master/build-support/tools/kudu-lint

[emkornfield]

OK, I think I addressed all of the suggestions here. I filed JIRAs for remaining issues. I accidentally ran clang-format-3.8 which re-arranged some of the headers to make them alphabetical. This lead to some header reordering which looks correct. some other stuff that disagrees with clang-format-3.7 so I had to push one additional commit to make the check-format pass. Sorry for the noisy code review.

[wesm]

Thank you for these thorough tests =)

[wesm]

Great, thank you for all of this additional work. At some point I figure we'll move to LLVM 3.8 -- for now I guess we can use include-what-you-use or other tools for deeper header cleanup.

+1, merging now