[fix](orc) ignore null values when the literals of in_predicate contains #45104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

morningman merged 11 commits into apache:master from suxiaogang223:fix_orc_push_dwon

Dec 18, 2024

Contributor

suxiaogang223 commented Dec 6, 2024

What problem does this PR solve?

Related PR: #43255

Problem Summary:
Should ignore null values when the literals of in_predicate contains null value, like in (1, null)
For example, init table in hive:

CREATE TABLE sample_orc_table (
    id INT,
    name STRING,
    age INT
)
STORED AS ORC;
INSERT INTO TABLE sample_orc_table VALUES
    (1, 'Alice', 25),
    (2, NULL, NULL);

select result in Doris should be:

mysql> select * from sample_orc_table where age in (null,25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.30 sec)

mysql> select * from sample_orc_table where age in (25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.27 sec)

mysql> select * from sample_orc_table where age in (null);
Empty set (0.01 sec)

mysql> select * from sample_orc_table where age is null;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    2 | NULL | NULL |
+------+------+------+
1 row in set (0.11 sec)

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot commented Dec 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 force-pushed the fix_orc_push_dwon branch 3 times, most recently from b48f4f7 to d317351 Compare

December 6, 2024 16:46

Contributor Author

suxiaogang223 commented Dec 6, 2024

run buildall

morningman added the dev/3.0.x label

suxiaogang223 marked this pull request as draft

December 9, 2024 06:22

suxiaogang223 force-pushed the fix_orc_push_dwon branch from b4bb463 to 52d3b60 Compare

December 9, 2024 07:58

Contributor

github-actions bot commented Dec 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 force-pushed the fix_orc_push_dwon branch from 52d3b60 to ec9aded Compare

December 9, 2024 08:39

Contributor Author

suxiaogang223 commented Dec 9, 2024

run buildall

Contributor

github-actions bot commented Dec 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 force-pushed the fix_orc_push_dwon branch from ec9aded to 02754ce Compare

December 9, 2024 11:12

Contributor Author

suxiaogang223 commented Dec 9, 2024

run buildall

Contributor

github-actions bot commented Dec 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 marked this pull request as ready for review

December 10, 2024 03:33

Contributor Author

suxiaogang223 commented Dec 10, 2024

run buildall

Contributor

github-actions bot commented Dec 10, 2024

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 mentioned this pull request

ORC-1813: [C++] Fix has_null forward compatibility apache/orc#2082

Closed

Contributor Author

suxiaogang223 commented Dec 11, 2024

run buildall

Contributor

github-actions bot commented Dec 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

Contributor Author

suxiaogang223 commented Dec 11, 2024

run buildall

Contributor

github-actions bot commented Dec 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

doris-robot commented Dec 11, 2024

TeamCity be ut coverage result:
Function Coverage: 38.76% (10104/26069)
Line Coverage: 29.70% (84765/285368)
Region Coverage: 28.77% (43515/151228)
Branch Coverage: 25.33% (22107/87270)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a20e66db478b3b0ddcfbeb2ac4ad7e1dee45765a_a20e66db478b3b0ddcfbeb2ac4ad7e1dee45765a/report/index.html

suxiaogang223 added 6 commits

December 12, 2024 00:04


          fix bug when the literals of in_predicate contains null value, like i…

dd42ab9

…n (1, null)


          fix _build_filter_in

93e1cb6


          add cases


          add be ut

7bc3d26

fix

07822c2


          fix orc

5aa85f9

github-actions bot added the reviewed label

Contributor

github-actions bot commented Dec 12, 2024

PR approved by anyone and no changes requested.

dongjoon-hyun pushed a commit to apache/orc that referenced this pull request


          ORC-1813: [C++] Fix has_null forward compatibility

da5bda5

### What changes were proposed in this pull request?
close: #2079
relate pr: #2055
Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation.
The Java implementation includes the following logic:
```java
if (stats.hasHasNull()) {
    hasNull = stats.getHasNull();
} else {
    hasNull = true;
}
```
In contrast, the C++ implementation directly uses the has_null value without any fallback logic:
```c++
ColumnStatisticsImpl::ColumnStatisticsImpl(const proto::ColumnStatistics& pb) {
    stats_.setNumberOfValues(pb.number_of_values());
    stats_.setHasNull(pb.has_null());
}
```
### Why are the changes needed?
We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly.
**This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(**
### How was this patch tested?
I have tested this using [Doris](https://github.com/apache/doris) external pipeline:
apache/doris#45104
apache/doris-thirdparty#259

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #2082 from suxiaogang223/fix_has_null.

Authored-by: Socrates <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

suxiaogang223 added a commit to suxiaogang223/orc that referenced this pull request


          ORC-1813: [C++] Fix has_null forward compatibility

21aa4cd

close: apache#2079
relate pr: apache#2055
Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation.
The Java implementation includes the following logic:
```java
if (stats.hasHasNull()) {
    hasNull = stats.getHasNull();
} else {
    hasNull = true;
}
```
In contrast, the C++ implementation directly uses the has_null value without any fallback logic:
```c++
ColumnStatisticsImpl::ColumnStatisticsImpl(const proto::ColumnStatistics& pb) {
    stats_.setNumberOfValues(pb.number_of_values());
    stats_.setHasNull(pb.has_null());
}
```
We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly.
**This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(**
I have tested this using [Doris](https://github.com/apache/doris) external pipeline:
apache/doris#45104
apache/doris-thirdparty#259
No

Closes apache#2082 from suxiaogang223/fix_has_null.

Authored-by: Socrates <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

suxiaogang223 mentioned this pull request

ORC-1813: [C++] Fix has_null forward compatibility apache/orc#2086

Closed

kaka11chen approved these changes

View reviewed changes

Contributor

kaka11chen left a comment

LGTM

morningman merged commit 58f1df2 into apache:master

25 of 28 checks passed

github-actions bot pushed a commit that referenced this pull request


          [fix](orc) ignore null values when the literals of in_predicate conta…

fe2f617

…ins (#45104)

### What problem does this PR solve?

Related PR: #43255 

Problem Summary:
Should ignore null values when the literals of in_predicate contains
null value, like `in (1, null)`
For example, init table in hive:
```sql
CREATE TABLE sample_orc_table (
    id INT,
    name STRING,
    age INT
)
STORED AS ORC;
INSERT INTO TABLE sample_orc_table VALUES
    (1, 'Alice', 25),
    (2, NULL, NULL); 
```
select result in Doris should be:
```sql
mysql> select * from sample_orc_table where age in (null,25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.30 sec)

mysql> select * from sample_orc_table where age in (25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.27 sec)

mysql> select * from sample_orc_table where age in (null);
Empty set (0.01 sec)

mysql> select * from sample_orc_table where age is null;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    2 | NULL | NULL |
+------+------+------+
1 row in set (0.11 sec)
```

github-actions bot mentioned this pull request

branch-3.0: [fix](orc) ignore null values when the literals of in_predicate contains #45104 #45586

Merged

yiguolei pushed a commit that referenced this pull request


          branch-3.0: [fix](orc) ignore null values when the literals of in_pre…

55a51dc

…dicate contains #45104 (#45586)

Cherry-picked from #45104

Co-authored-by: Socrates <[email protected]>

yiguolei added dev/3.0.4-merged and removed dev/3.0.x labels

dongjoon-hyun pushed a commit to apache/orc that referenced this pull request


          ORC-1813: [C++] Fix has_null forward compatibility

84a32a9

close: #2079
relate pr: #2055
Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation. The Java implementation includes the following logic:
```java
if (stats.hasHasNull()) {
    hasNull = stats.getHasNull();
} else {
    hasNull = true;
}
```
In contrast, the C++ implementation directly uses the has_null value without any fallback logic:
```c++
ColumnStatisticsImpl::ColumnStatisticsImpl(const proto::ColumnStatistics& pb) {
    stats_.setNumberOfValues(pb.number_of_values());
    stats_.setHasNull(pb.has_null());
}
```
We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly. **This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(** I have tested this using [Doris](https://github.com/apache/doris) external pipeline: apache/doris#45104
apache/doris-thirdparty#259 No

Closes #2082 from suxiaogang223/fix_has_null.

Authored-by: Socrates <suxiaogang223icloud.com>

### What changes were proposed in this pull request?

### Why are the changes needed?

### How was this patch tested?

### Was this patch authored or co-authored using generative AI tooling?

Closes #2086 from suxiaogang223/cherry_pick_fix_has_null.

Authored-by: Socrates <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

morningman added the dev/2.1.x-experimental label

morningman added a commit to morningman/doris that referenced this pull request


          Revert "branch-3.0: [fix](orc) ignore null values when the literals o…

4598fd5

…f in_predicate contains apache#45104 (apache#45586)"

This reverts commit 55a51dc.

morningman mentioned this pull request

[revert](orc) revert orc predicate pushdown related commits #47662

Merged

morningman added a commit that referenced this pull request


          [revert](orc) revert orc predicate pushdown related commits (#47662)

7ed947e

revert:
branch-3.0: [fix](orc) ignore null values when the literals of
in_predicate contains #45104 (#45586)
[fix](orc) check all the cases before build_search_argument (#44615)
(#44802)
branch-3.0: [enhance](orc) Optimize ORC Predicate Pushdown for
OR-connected Predicate #43255 (#44436)

re-pick:
branch-3.0: [Fix](ORC) Not push down fixed char type in orc reader
#45484 (#45525)

---------

Co-authored-by: Socrates <[email protected]>

morningman added dev/3.0.x-experimental and removed dev/3.0.4-merged labels

morningman added the dev/2.1.x-lh label

morningman pushed a commit that referenced this pull request


          [fix](orc) ignore null values when the literals of in_predicate conta…

88df781

…ins (#45104)

### What problem does this PR solve?

Related PR: #43255 

Problem Summary:
Should ignore null values when the literals of in_predicate contains
null value, like `in (1, null)`
For example, init table in hive:
```sql
CREATE TABLE sample_orc_table (
    id INT,
    name STRING,
    age INT
)
STORED AS ORC;
INSERT INTO TABLE sample_orc_table VALUES
    (1, 'Alice', 25),
    (2, NULL, NULL); 
```
select result in Doris should be:
```sql
mysql> select * from sample_orc_table where age in (null,25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.30 sec)

mysql> select * from sample_orc_table where age in (25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.27 sec)

mysql> select * from sample_orc_table where age in (null);
Empty set (0.01 sec)

mysql> select * from sample_orc_table where age is null;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    2 | NULL | NULL |
+------+------+------+
1 row in set (0.11 sec)
```

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request


          [fix](orc) ignore null values when the literals of in_predicate conta…

083df72

…ins (apache#45104)

Related PR: apache#43255

Problem Summary:
Should ignore null values when the literals of in_predicate contains
null value, like `in (1, null)`
For example, init table in hive:
```sql
CREATE TABLE sample_orc_table (
    id INT,
    name STRING,
    age INT
)
STORED AS ORC;
INSERT INTO TABLE sample_orc_table VALUES
    (1, 'Alice', 25),
    (2, NULL, NULL);
```
select result in Doris should be:
```sql
mysql> select * from sample_orc_table where age in (null,25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.30 sec)

mysql> select * from sample_orc_table where age in (25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.27 sec)

mysql> select * from sample_orc_table where age in (null);
Empty set (0.01 sec)

mysql> select * from sample_orc_table where age is null;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    2 | NULL | NULL |
+------+------+------+
1 row in set (0.11 sec)
```

suxiaogang223 mentioned this pull request

branch-3.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate (#43255 #44615 #45104 #47506 #47625 #49088 #49835 #49927) #52192

Merged

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request


          [fix](orc) ignore null values when the literals of in_predicate conta…

f83e252

…ins (apache#45104)

Related PR: apache#43255

Problem Summary:
Should ignore null values when the literals of in_predicate contains
null value, like `in (1, null)`
For example, init table in hive:
```sql
CREATE TABLE sample_orc_table (
    id INT,
    name STRING,
    age INT
)
STORED AS ORC;
INSERT INTO TABLE sample_orc_table VALUES
    (1, 'Alice', 25),
    (2, NULL, NULL);
```
select result in Doris should be:
```sql
mysql> select * from sample_orc_table where age in (null,25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.30 sec)

mysql> select * from sample_orc_table where age in (25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.27 sec)

mysql> select * from sample_orc_table where age in (null);
Empty set (0.01 sec)

mysql> select * from sample_orc_table where age is null;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    2 | NULL | NULL |
+------+------+------+
1 row in set (0.11 sec)
```

morningman pushed a commit to suxiaogang223/doris that referenced this pull request


          [fix](orc) ignore null values when the literals of in_predicate conta…

424ffce

…ins (apache#45104)

Related PR: apache#43255

Problem Summary:
Should ignore null values when the literals of in_predicate contains
null value, like `in (1, null)`
For example, init table in hive:
```sql
CREATE TABLE sample_orc_table (
    id INT,
    name STRING,
    age INT
)
STORED AS ORC;
INSERT INTO TABLE sample_orc_table VALUES
    (1, 'Alice', 25),
    (2, NULL, NULL);
```
select result in Doris should be:
```sql
mysql> select * from sample_orc_table where age in (null,25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.30 sec)

mysql> select * from sample_orc_table where age in (25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.27 sec)

mysql> select * from sample_orc_table where age in (null);
Empty set (0.01 sec)

mysql> select * from sample_orc_table where age is null;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    2 | NULL | NULL |
+------+------+------+
1 row in set (0.11 sec)
```

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request


          [fix](orc) ignore null values when the literals of in_predicate conta…

eefeea1

…ins (apache#45104)

Related PR: apache#43255

Problem Summary:
Should ignore null values when the literals of in_predicate contains
null value, like `in (1, null)`
For example, init table in hive:
```sql
CREATE TABLE sample_orc_table (
    id INT,
    name STRING,
    age INT
)
STORED AS ORC;
INSERT INTO TABLE sample_orc_table VALUES
    (1, 'Alice', 25),
    (2, NULL, NULL);
```
select result in Doris should be:
```sql
mysql> select * from sample_orc_table where age in (null,25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.30 sec)

mysql> select * from sample_orc_table where age in (25);
+------+-------+------+
| id   | name  | age  |
+------+-------+------+
|    1 | Alice |   25 |
+------+-------+------+
1 row in set (0.27 sec)

mysql> select * from sample_orc_table where age in (null);
Empty set (0.01 sec)

mysql> select * from sample_orc_table where age is null;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    2 | NULL | NULL |
+------+------+------+
1 row in set (0.11 sec)
```

morrySnow pushed a commit that referenced this pull request


          branch-3.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-con…

0306f2b

…nected Predicate #43255 #44615 #45104 #47506 #47625 #49088 #49835 #49927 (#52192)

picks:
#43255
#44615
#45104
#47506
#47625
#49088
#49835
#49927

---------

Co-authored-by: Qi Chen <[email protected]>

morrySnow added the dev/3.1.0-merged label

morningman removed the dev/3.0.x-experimental label

GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request


          fix(orc): ignore null values when the literals of in_predicate contai…

bea97de

…ns (apache#45104)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dev/2.1.x-experimental dev/2.1.x-lh dev/3.1.0-merged reviewed