Fix materialize bug with RedisCluster #2311

kevjumba · 2022-02-15T17:53:22Z

Signed-off-by: Kevin Zhang [email protected]

What this PR does / why we need it:
Broken materialization script for feast 0.18.0. Refer to issue #2297.
Issue was that there are 2 versions of the python redis cluster package redis-py-cluster and redis-py. redis-py only recently added a new version of RedisCluster and our api was using the redis-py-cluster api that passes dictionary of host:port pairs while redis-py implementation requires ClusterNodes instead.

Which issue(s) this PR fixes:

Fixes #2297

Does this PR introduce a user-facing change?:

NONE

Makes changes to integration github workflow that cannot be tested accurately since github runs the current PR against the current github workflow. Attached is proof that the new integration tests with the new workflow:

Signed-off-by: Kevin Zhang <[email protected]>

codecov-commenter · 2022-02-15T17:55:24Z

Codecov Report

Merging #2311 (1a4a155) into master (a7d4cb7) will decrease coverage by 27.61%.
The diff coverage is 12.50%.

@@             Coverage Diff             @@
##           master    #2311       +/-   ##
===========================================
- Coverage   86.11%   58.50%   -27.62%     
===========================================
  Files         116      116               
  Lines        9834     9933       +99     
===========================================
- Hits         8469     5811     -2658     
- Misses       1365     4122     +2757

Flag	Coverage Δ
integrationtests	`?`
unittests	`58.50% <12.50%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../integration/online_store/test_universal_online.py	`14.28% <5.26%> (-82.91%)`	⬇️
sdk/python/tests/conftest.py	`56.48% <28.57%> (-35.60%)`	⬇️
sdk/python/feast/infra/online_stores/redis.py	`26.45% <50.00%> (-68.39%)`	⬇️
...ts/integration/feature_repos/repo_configuration.py	`44.95% <100.00%> (-49.50%)`	⬇️
.../integration/online_store/test_online_retrieval.py	`17.39% <0.00%> (-82.61%)`	⬇️
sdk/python/tests/utils/online_read_write_test.py	`20.68% <0.00%> (-79.32%)`	⬇️
...fline_store/test_universal_historical_retrieval.py	`23.50% <0.00%> (-76.50%)`	⬇️
...gration/registration/test_feature_service_apply.py	`31.25% <0.00%> (-68.75%)`	⬇️
... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7d4cb7...1a4a155. Read the comment docs.

sdk/python/feast/infra/online_stores/redis.py

Signed-off-by: Kevin Zhang <[email protected]>

…positives Signed-off-by: Kevin Zhang <[email protected]>

Signed-off-by: Kevin Zhang <[email protected]>

felixwang9817

/lgtm

kevjumba · 2022-02-17T01:25:01Z

@adchia PTAL!

adchia

/lgtm

feast-ci-bot · 2022-02-17T17:57:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, felixwang9817, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [adchia,felixwang9817]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…ents * upstream/master: (103 commits) refactor: Move create cluster script ci: Add validation for upper case PR titles Switch order of views to keep/delete to fix apply logging ci: Fix broken Prow plugin which ignores config.yaml ci: Add PR linting to ensure conventional commits are followed for PR titles Add owner field to FeatureService (feast-dev#2321) Disable Redis cluster tests (feast-dev#2327) Use PR title for commit message Add redis cluster initialziation to master github workflow (feast-dev#2317) Fix materialize bug with RedisCluster (feast-dev#2311) Add support for Dask during historical retrieval (feast-dev#1954) downgrade datatypes to 8 Update pom.xml to 0.18.2 SNAPSHOT Update changelog for 0.18.1 and update pom.xml Update helm chart version Don't require `snowflake` to always be installed (feast-dev#2309) Fixing the Java helm charts and adding a demo tutorial on how to use them (feast-dev#2298) Include infra objects in registry dump and fix Infra's from_proto (feast-dev#2295) Remove old flag warning with the python feature server (feast-dev#2300) podAnnotations Values in the feature-server chart (feast-dev#2304) ...

* Fix materialization bug Signed-off-by: Kevin Zhang <[email protected]> * oops removed something when i was copy pasting Signed-off-by: Kevin Zhang <[email protected]> * Test github workflow Signed-off-by: Kevin Zhang <[email protected]> * lint Signed-off-by: Kevin Zhang <[email protected]> * transfer to unit_tests Signed-off-by: Kevin Zhang <[email protected]> * Brute force implementation Signed-off-by: Kevin Zhang <[email protected]> * Brute force implementation add gcc Signed-off-by: Kevin Zhang <[email protected]> * Brute force implementation add gcc Signed-off-by: Kevin Zhang <[email protected]> * Brute force implementation add gcc Signed-off-by: Kevin Zhang <[email protected]> * Brute force implementation add gcc Signed-off-by: Kevin Zhang <[email protected]> * Continue fixing... Signed-off-by: Kevin Zhang <[email protected]> * Remove gcc setup Signed-off-by: Kevin Zhang <[email protected]> * Add integration test Signed-off-by: Kevin Zhang <[email protected]> * fix error Signed-off-by: Kevin Zhang <[email protected]> * fix error Signed-off-by: Kevin Zhang <[email protected]> * Add setup Signed-off-by: Kevin Zhang <[email protected]> * temp fix to get integration tests to work Signed-off-by: Kevin Zhang <[email protected]> * temp fix to get integration tests to work Signed-off-by: Kevin Zhang <[email protected]> * temp fix to get integration tests to work Signed-off-by: Kevin Zhang <[email protected]> * temp fix to get integration tests to work Signed-off-by: Kevin Zhang <[email protected]> * Fix integration even more Signed-off-by: Kevin Zhang <[email protected]> * Fix integration even more Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * only run one test Signed-off-by: Kevin Zhang <[email protected]> * Do some more integration testing Signed-off-by: Kevin Zhang <[email protected]> * Do some more integration testing by adding bug to make sure no false positives Signed-off-by: Kevin Zhang <[email protected]> * Integration testing works Signed-off-by: Kevin Zhang <[email protected]> * Clean upu code Signed-off-by: Kevin Zhang <[email protected]> * Add redis cluster script for starting a redis cluster Signed-off-by: Kevin Zhang <[email protected]> * Reset integration yml file Signed-off-by: Kevin Zhang <[email protected]> * lint Signed-off-by: Kevin Zhang <[email protected]> * Clean up Signed-off-by: Kevin Zhang <[email protected]> * Fix how to guide lint Signed-off-by: Kevin Zhang <[email protected]> * add fixtures and remove excess code Signed-off-by: Kevin Zhang <[email protected]> * lint Signed-off-by: Kevin Zhang <[email protected]>

Fix materialization bug

4134113

Signed-off-by: Kevin Zhang <[email protected]>

kevjumba requested a review from a team as a code owner February 15, 2022 17:53

kevjumba requested review from pyalex and removed request for a team February 15, 2022 17:53

feast-ci-bot added release-note-none needs-kind size/XS labels Feb 15, 2022

kevjumba added kind/bug and removed size/XS labels Feb 15, 2022

feast-ci-bot removed the needs-kind label Feb 15, 2022

github-actions bot assigned felixwang9817 Feb 15, 2022

kevjumba requested review from adchia and felixwang9817 February 15, 2022 17:56

felixwang9817 added ok-to-test size/S size/XS and removed size/S labels Feb 15, 2022

felixwang9817 requested changes Feb 15, 2022

View reviewed changes

sdk/python/feast/infra/online_stores/redis.py Show resolved Hide resolved

kevjumba added 3 commits February 15, 2022 10:04

oops removed something when i was copy pasting

926260b

Signed-off-by: Kevin Zhang <[email protected]>

Test github workflow

b4e9ac0

Signed-off-by: Kevin Zhang <[email protected]>

lint

1281b2d

Signed-off-by: Kevin Zhang <[email protected]>

feast-ci-bot added size/L and removed size/XS labels Feb 16, 2022

kevjumba added 6 commits February 15, 2022 16:23

transfer to unit_tests

b404f4f

Signed-off-by: Kevin Zhang <[email protected]>

Brute force implementation

f8dab19

Signed-off-by: Kevin Zhang <[email protected]>

Brute force implementation add gcc

c05d2b2

Signed-off-by: Kevin Zhang <[email protected]>

Brute force implementation add gcc

e53c75b

Signed-off-by: Kevin Zhang <[email protected]>

Brute force implementation add gcc

eafc27d

Signed-off-by: Kevin Zhang <[email protected]>

Brute force implementation add gcc

bbbb325

Signed-off-by: Kevin Zhang <[email protected]>

kevjumba added 15 commits February 16, 2022 11:53

Fix integration even more

c32ed21

Signed-off-by: Kevin Zhang <[email protected]>

Fix integration even more

68c9b61

Signed-off-by: Kevin Zhang <[email protected]>

Fix lint

caeb757

Signed-off-by: Kevin Zhang <[email protected]>

only run one test

8b9f2f1

Signed-off-by: Kevin Zhang <[email protected]>

Do some more integration testing

1f060e0

Signed-off-by: Kevin Zhang <[email protected]>

Do some more integration testing by adding bug to make sure no false …

931dc23

…positives Signed-off-by: Kevin Zhang <[email protected]>

Integration testing works

41f7c75

Signed-off-by: Kevin Zhang <[email protected]>

Clean upu code

174af8e

Signed-off-by: Kevin Zhang <[email protected]>

Add redis cluster script for starting a redis cluster

d818c0a

Signed-off-by: Kevin Zhang <[email protected]>

Reset integration yml file

b2903f8

Signed-off-by: Kevin Zhang <[email protected]>

lint

0b5dabd

Signed-off-by: Kevin Zhang <[email protected]>

Clean up

1a4a155

Signed-off-by: Kevin Zhang <[email protected]>

Fix how to guide lint

52f239d

Signed-off-by: Kevin Zhang <[email protected]>

add fixtures and remove excess code

a21e13f

Signed-off-by: Kevin Zhang <[email protected]>

lint

9c59fcc

Signed-off-by: Kevin Zhang <[email protected]>

felixwang9817 changed the title ~~[WIP] Fix #2297 Materialize bug with RedisCluster~~ Fix materialize bug with RedisCluster Feb 17, 2022

feast-ci-bot removed the do-not-merge/work-in-progress label Feb 17, 2022

felixwang9817 approved these changes Feb 17, 2022

View reviewed changes

feast-ci-bot added lgtm approved labels Feb 17, 2022

kevjumba assigned adchia and unassigned felixwang9817 Feb 17, 2022

adchia approved these changes Feb 17, 2022

View reviewed changes

adchia merged commit 7fa9fe3 into feast-dev:master Feb 17, 2022

This was referenced Feb 19, 2022

Disable Redis cluster tests #2327

Merged

Redis Cluster implementation is broken #2328

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix materialize bug with RedisCluster #2311

Fix materialize bug with RedisCluster #2311

Uh oh!

kevjumba commented Feb 15, 2022 •

edited

Loading

Uh oh!

codecov-commenter commented Feb 15, 2022 •

edited

Loading

Uh oh!

Uh oh!

felixwang9817 left a comment

Uh oh!

kevjumba commented Feb 17, 2022

Uh oh!

adchia left a comment

Uh oh!

feast-ci-bot commented Feb 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Fix materialize bug with RedisCluster #2311

Fix materialize bug with RedisCluster #2311

Uh oh!

Conversation

kevjumba commented Feb 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Feb 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

felixwang9817 left a comment

Choose a reason for hiding this comment

Uh oh!

kevjumba commented Feb 17, 2022

Uh oh!

adchia left a comment

Choose a reason for hiding this comment

Uh oh!

feast-ci-bot commented Feb 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kevjumba commented Feb 15, 2022 •

edited

Loading

codecov-commenter commented Feb 15, 2022 •

edited

Loading