Skip to content

fix: warehouse transformations #5875

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 21, 2025
Merged

fix: warehouse transformations #5875

merged 1 commit into from
May 21, 2025

Conversation

achettyiitr
Copy link
Member

@achettyiitr achettyiitr commented May 20, 2025 •
edited
Loading

Description

  • In rudder-transformer, we are using Number.isInteger to determine for int datatype, in comparison with Golang, which has a lower range 9223372036854775807. 654645465456456400000 is being calculated as a float in Golang and an int in JS. Using a big package to determine now.
  • utf16 Rune count for being in conjunction with JavaScript.
  • GeoLocation enrichment struct getting embedded directly to a singular event. These need to be handled separately, like we are doing for the Tracking Plan.
  • Exclude the extract event in case the event name is empty.
  • Exclude the identity event in case the userID is empty.

Linear Ticket

  • Resolves WAR-672.

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@achettyiitr achettyiitr force-pushed the fix.warehouse-transformations branch from ba4bbb0 to d028db8 Compare May 20, 2025 09:18
@codecov Codecov
Copy link

codecov bot commented May 20, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 62.12121% with 25 lines in your changes missing coverage. Please review.

Project coverage is 76.94%. Comparing base (081cf7b) to head (9026f80).
Report is 1 commits behind head on release/1.49.x.

Files with missing lines Patch % Lines
...ination_transformer/embedded/warehouse/datatype.go 60.37% 14 Missing and 7 partials ⚠️
.../destination_transformer/embedded/warehouse/set.go 25.00% 2 Missing and 1 partial ⚠️
...ination_transformer/embedded/warehouse/uploader.go 0.00% 1 Missing ⚠️
Additional details and impacted files 
@@                Coverage Diff                 @@
##           release/1.49.x    #5875      +/-   ##
==================================================
- Coverage           77.00%   76.94%   -0.07%     
==================================================
  Files                 496      496              
  Lines               67788    67847      +59     
==================================================
- Hits                52203    52202       -1     
- Misses              12761    12808      +47     
- Partials             2824     2837      +13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@achettyiitr achettyiitr force-pushed the fix.warehouse-transformations branch from d028db8 to 9026f80 Compare May 21, 2025 02:53
@achettyiitr achettyiitr merged commit a22f901 into release/1.49.x May 21, 2025
59 checks passed
@achettyiitr achettyiitr deleted the fix.warehouse-transformations branch May 21, 2025 03:16
@devops-github-rudderstack devops-github-rudderstack mentioned this pull request May 21, 2025
Merged
achettyiitr pushed a commit that referenced this pull request May 21, 2025
@devops-github-rudderstack
chore: release 1.49.4 (#5877)
225c42b 
🤖 I have created a release *beep* *boop*
---


##
[1.49.4](v1.49.3...v1.49.4)
(2025-05-21)


### Bug Fixes

* warehouse transformations
([#5875](#5875))
([a22f901](a22f901))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
@devops-github-rudderstack devops-github-rudderstack mentioned this pull request May 21, 2025
Merged
achettyiitr added a commit that referenced this pull request May 21, 2025
@devops-github-rudderstack @satishrudderstack
@ItsSudip @atzoum @achettyiitr
chore: sync release v1.49.4 to main branch (#5878)
715ccfa 
# Description

Syncing patch release v1.49.4 to main branch

**↓↓ Please review and edit commit overrides before merging ↓↓**

BEGIN_COMMIT_OVERRIDE
fix: warehouse transformations (#5875)
END_COMMIT_OVERRIDE

---------

Co-authored-by: Satish Kumar <[email protected]>
Co-authored-by: Sudip Paul <[email protected]>
Co-authored-by: Aris Tzoumas <[email protected]>
Co-authored-by: Akash Chetty <[email protected]>
ItsSudip added a commit that referenced this pull request May 21, 2025
@devops-github-rudderstack @satishrudderstack
@ItsSudip @atzoum @achettyiitr
chore: sync release v1.49.4 to main branch (#5878)
89c39da 
# Description

Syncing patch release v1.49.4 to main branch

**↓↓ Please review and edit commit overrides before merging ↓↓**

BEGIN_COMMIT_OVERRIDE
fix: warehouse transformations (#5875)
END_COMMIT_OVERRIDE

---------

Co-authored-by: Satish Kumar <[email protected]>
Co-authored-by: Sudip Paul <[email protected]>
Co-authored-by: Aris Tzoumas <[email protected]>
Co-authored-by: Akash Chetty <[email protected]>
This was referenced May 21, 2025
Merged
Merged
mihir20 pushed a commit that referenced this pull request May 26, 2025
@devops-github-rudderstack
chore: prerelease 1.50.0-rc.1 (#5894)
bd82f88 
🤖 I have created a release *beep* *boop*
---


##
[1.50.0-rc.1](v1.49.0...v1.50.0-rc.1)
(2025-05-26)


### Features

* implement aws v2 implementation
([#5852](#5852))
([18fd4fb](18fd4fb))


### Bug Fixes

* do not panic for invalid router payload
([#5880](#5880))
([f4a0e7c](f4a0e7c))
* server panicking with invalid byte sequence for encoding utf8 0x00
([#5882](#5882))
([4c1a0de](4c1a0de))
* skip source_id when fetching warehouse schema from database
([#5886](#5886))
([092ee2a](092ee2a))
* warehouse pending tables to skip
([#5867](#5867))
([864fd49](864fd49))
* warehouse transformations
([#5875](#5875))
([715ccfa](715ccfa))


### Miscellaneous

* **deps:** bump the go-deps group across 1 directory with 25 updates
([#5890](#5890))
([3c4ddd8](3c4ddd8))
* **jobsdb:** migrate incomplete datasets in pairs excluding completed
ones ([#5876](#5876))
([9d8b573](9d8b573))
* **jobsdb:** remove support for jobDoneMigrateThres
([#5873](#5873))
([4745d2b](4745d2b))
* make reporting client params configurable
([#5891](#5891))
([39f7e75](39f7e75))
* move house keeping action to commit sha
([#5885](#5885))
([d379e82](d379e82))
* refactor config change dtype for config to json.RawMessage
([#5874](#5874))
([940adb0](940adb0))
* refactor webhook pkg to make it reusable in other repos
([#5594](#5594))
([1ee57bc](1ee57bc))
* remove redundant structure for account with definition
([#5872](#5872))
([89bc62b](89bc62b))
* remove support for transformer-based kafka batching
([#5870](#5870))
([60a571c](60a571c))
* revert support new consent resolution strategy values
([#5855](#5855))
([c7e3b37](c7e3b37))
* revert use account to decide oauth type of a destination
([#5861](#5861))
([6232755](6232755))
* **router:** additional observability metrics
([#5887](#5887))
([da3ba3c](da3ba3c))
* sync release v1.49.0 to main branch
([#5851](#5851))
([78cc5a9](78cc5a9))
* use jsonrs from rudder-go-kit
([#5869](#5869))
([39064af](39064af))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
This was referenced May 26, 2025
Closed
Closed
Merged
Merged
atzoum pushed a commit that referenced this pull request May 26, 2025
@devops-github-rudderstack
chore: prerelease 1.50.0-rc.2 (#5902)
5cffd3c 
🤖 I have created a release *beep* *boop*
---


##
[1.50.0-rc.2](v1.49.0...v1.50.0-rc.2)
(2025-05-26)


### Features

* implement aws v2 implementation
([#5852](#5852))
([18fd4fb](18fd4fb))


### Bug Fixes

* do not panic for invalid router payload
([#5880](#5880))
([f4a0e7c](f4a0e7c))
* error gathering metrics, metric was collected before with the same
name and label values
([#5900](#5900))
([422f45b](422f45b))
* handle missing eventUpload config gracefully
([#5898](#5898))
([05a926d](05a926d))
* server panicking with invalid byte sequence for encoding utf8 0x00
([#5882](#5882))
([4c1a0de](4c1a0de))
* skip source_id when fetching warehouse schema from database
([#5886](#5886))
([092ee2a](092ee2a))
* warehouse pending tables to skip
([#5867](#5867))
([864fd49](864fd49))
* warehouse transformations
([#5875](#5875))
([715ccfa](715ccfa))


### Miscellaneous

* **deps:** bump the go-deps group across 1 directory with 25 updates
([#5890](#5890))
([3c4ddd8](3c4ddd8))
* **jobsdb:** migrate incomplete datasets in pairs excluding completed
ones ([#5876](#5876))
([9d8b573](9d8b573))
* **jobsdb:** remove support for jobDoneMigrateThres
([#5873](#5873))
([4745d2b](4745d2b))
* make reporting client params configurable
([#5891](#5891))
([39f7e75](39f7e75))
* move house keeping action to commit sha
([#5885](#5885))
([d379e82](d379e82))
* refactor config change dtype for config to json.RawMessage
([#5874](#5874))
([940adb0](940adb0))
* refactor webhook pkg to make it reusable in other repos
([#5594](#5594))
([1ee57bc](1ee57bc))
* remove redundant structure for account with definition
([#5872](#5872))
([89bc62b](89bc62b))
* remove support for transformer-based kafka batching
([#5870](#5870))
([60a571c](60a571c))
* revert support new consent resolution strategy values
([#5855](#5855))
([c7e3b37](c7e3b37))
* revert use account to decide oauth type of a destination
([#5861](#5861))
([6232755](6232755))
* **router:** additional observability metrics
([#5887](#5887))
([da3ba3c](da3ba3c))
* sync release v1.49.0 to main branch
([#5851](#5851))
([78cc5a9](78cc5a9))
* use jsonrs from rudder-go-kit
([#5869](#5869))
([39064af](39064af))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
mihir20 pushed a commit that referenced this pull request May 27, 2025
@devops-github-rudderstack
chore: release 1.50.0 (#5903)
2209fd6 
🤖 I have created a release *beep* *boop*
---


##
[1.50.0](v1.49.0...v1.50.0)
(2025-05-26)


### Features

* implement aws v2 implementation
([#5852](#5852))
([18fd4fb](18fd4fb))


### Bug Fixes

* do not panic for invalid router payload
([#5880](#5880))
([f4a0e7c](f4a0e7c))
* error gathering metrics, metric was collected before with the same
name and label values
([#5900](#5900))
([422f45b](422f45b))
* handle missing eventUpload config gracefully
([#5898](#5898))
([05a926d](05a926d))
* server panicking with invalid byte sequence for encoding utf8 0x00
([#5882](#5882))
([4c1a0de](4c1a0de))
* skip source_id when fetching warehouse schema from database
([#5886](#5886))
([092ee2a](092ee2a))
* warehouse pending tables to skip
([#5867](#5867))
([864fd49](864fd49))
* warehouse transformations
([#5875](#5875))
([715ccfa](715ccfa))


### Miscellaneous

* **deps:** bump the go-deps group across 1 directory with 25 updates
([#5890](#5890))
([3c4ddd8](3c4ddd8))
* **jobsdb:** migrate incomplete datasets in pairs excluding completed
ones ([#5876](#5876))
([9d8b573](9d8b573))
* **jobsdb:** remove support for jobDoneMigrateThres
([#5873](#5873))
([4745d2b](4745d2b))
* make reporting client params configurable
([#5891](#5891))
([39f7e75](39f7e75))
* move house keeping action to commit sha
([#5885](#5885))
([d379e82](d379e82))
* refactor config change dtype for config to json.RawMessage
([#5874](#5874))
([940adb0](940adb0))
* refactor webhook pkg to make it reusable in other repos
([#5594](#5594))
([1ee57bc](1ee57bc))
* remove redundant structure for account with definition
([#5872](#5872))
([89bc62b](89bc62b))
* remove support for transformer-based kafka batching
([#5870](#5870))
([60a571c](60a571c))
* revert support new consent resolution strategy values
([#5855](#5855))
([c7e3b37](c7e3b37))
* revert use account to decide oauth type of a destination
([#5861](#5861))
([6232755](6232755))
* **router:** additional observability metrics
([#5887](#5887))
([da3ba3c](da3ba3c))
* sync release v1.49.0 to main branch
([#5851](#5851))
([78cc5a9](78cc5a9))
* use jsonrs from rudder-go-kit
([#5869](#5869))
([39064af](39064af))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vyeshwanth vyeshwanth vyeshwanth approved these changes

@shekhar-rudder shekhar-rudder shekhar-rudder approved these changes

@RanjeetMishra RanjeetMishra Awaiting requested review from RanjeetMishra

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@achettyiitr @vyeshwanth @shekhar-rudder