ARROW-786: [Format] In-memory format for 128-bit Decimals, handling of sign bit #981

cpcloud · 2017-08-20T18:33:17Z

Reimplement Decimal128 types to use the Int128 type as the underlying integer
representation, adapted from the Apache ORC project's C++ in memory format.
This enables us to write integration tests and results in an in-memory
Decimal128 format that is compatible with the Java implementation
Additionaly, this PR also fixes Decimal slice comparison and adds related
regression tests
Follow-ups include ARROW-695 (C++ Decimal integration tests), ARROW-696 (JSON
read/write support for decimals) and ARROW-1238 (Java Decimal integration
tests).

wesm · 2017-08-20T18:51:35Z

nice! I will review when I can

wesm · 2017-08-21T00:24:51Z

If you rebase on master and then fix bugs, you should be able to get to a passing build

wesm · 2017-08-22T17:02:46Z

cpp/src/arrow/array.cc

Make the default offset 0

It's set to 0 as the default in the constructor already.

wesm · 2017-08-22T17:04:00Z

cpp/src/arrow/compare.cc

Any way to share code with FixedSizeBinaryArray on this?

Possibly. Let me look into cleaning up DecimalArray even more.

I was able to do a pretty significant refactoring of DecimalArray and inherit most methods from FixedSizeBinaryArray, including reuse of existing comparison methods for both FixedSizeBinaryArray and DecimalArray.

wesm · 2017-08-22T17:04:13Z

cpp/src/arrow/python/arrow_to_pandas.cc

kMaximumPrecision

wesm · 2017-08-22T17:05:32Z

cpp/src/arrow/util/CMakeLists.txt

does this do anything?

wesm · 2017-08-22T17:05:54Z

cpp/src/arrow/util/decimal-test.cc

trailing underscores

wesm · 2017-08-22T17:09:44Z

cpp/src/arrow/util/int128.cc

wesm · 2017-08-22T17:10:20Z

cpp/src/arrow/util/int128.h

wesm · 2017-08-22T17:10:26Z

cpp/src/arrow/util/int128.h

wesm · 2017-08-22T17:10:41Z

cpp/src/arrow/util/int128.h

Add \brief here for doxygen

Can you use /// style comments for doxygen instead of C-style?

wesm · 2017-08-22T17:10:50Z

cpp/src/arrow/util/int128.h

\brief, and below

wesm · 2017-08-22T17:12:37Z

Here's my "Arrow preflight check" that helps with not checking in cpplint or flake8 fails

function arrow_preflight {
    ARROW_PREFLIGHT_DIR=$HOME/code/arrow/cpp/preflight
    mkdir -p $ARROW_PREFLIGHT_DIR
    pushd $ARROW_PREFLIGHT_DIR
    arrow_cmake
    ninja format
    ninja lint
    popd
    pushd $HOME/code/arrow/python
    flake8 pyarrow
    flake8 --config=.flake8.cython pyarrow
    popd
}

…f sign bit

wesm

+1, thanks for all this! onward to the integration tests

wesm · 2017-08-24T17:55:56Z

cpp/src/arrow/array.h

- private:
-  void SetData(const std::shared_ptr<internal::ArrayData>& data);
-  const uint8_t* raw_values_;
-  const uint8_t* sign_bitmap_data_;


cpcloud force-pushed the decimal-rewrite branch from f0fdc76 to f0ab2e7 Compare August 20, 2017 18:46

cpcloud force-pushed the decimal-rewrite branch 4 times, most recently from 5837cfe to 3ae50de Compare August 20, 2017 19:10

cpcloud force-pushed the decimal-rewrite branch 2 times, most recently from 2a18b76 to 82332b6 Compare August 22, 2017 12:58

wesm reviewed Aug 22, 2017

View reviewed changes

cpcloud force-pushed the decimal-rewrite branch from 82332b6 to ea930e1 Compare August 23, 2017 13:46

cpcloud added 14 commits August 23, 2017 17:00

ARROW-786: [Format] In-memory format for 128-bit Decimals, handling o…

349dc58

…f sign bit

MSVC fix

9af97d8

kMaximumPrecision

22eda4b

Remove unnecessary header from CMakeLists.txt

1eab5c4

Fix naming convention in decimal-test.cc

2930d7b

Fix doc style in int128.h and add const where possible

38c9b50

NOLINT for MSVC specific and necessary types

178d3f2

Formatting

8ecb315

More const

e4a1792

Fix more docs

628ce85

Remove unnecessary args to sto* function calls

8833904

Cast 18 to matching type size_t for msvc

5716010

Cast so we have enough space to contain the integer

be220c8

BUG: Export symbols

b98c894

cpcloud force-pushed the decimal-rewrite branch from a7e721e to b98c894 Compare August 23, 2017 21:06

cpcloud added 3 commits August 23, 2017 21:26

BUG: Double export

162eeeb

Share comparison code

b53d7cd

Include the correct header for _BitScanReverse

355fb24

cpcloud added 5 commits August 23, 2017 23:14

Use CHAR_BIT

1cc43ce

Use a macro for cases

535f9ff

Subclass from FixedSizeBinaryArray for code reuse

86db184

Remove redundant constructor

fe13ef3

Formatting

53ce04b

wesm approved these changes Aug 24, 2017

View reviewed changes

asfgit closed this in 750b77d Aug 24, 2017

ARROW-786: [Format] In-memory format for 128-bit Decimals, handling of sign bit #981

ARROW-786: [Format] In-memory format for 128-bit Decimals, handling of sign bit #981

Uh oh!

Conversation

cpcloud commented Aug 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wesm commented Aug 20, 2017

Uh oh!

wesm commented Aug 21, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wesm commented Aug 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wesm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cpcloud commented Aug 20, 2017 •

edited

Loading

wesm commented Aug 22, 2017 •

edited

Loading