Skip to content

fix: warehouse transformations responses ordering #5954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 9, 2025
Merged

fix: warehouse transformations responses ordering #5954

merged 1 commit into from
Jun 9, 2025

Conversation

achettyiitr
Copy link
Member

@achettyiitr achettyiitr commented Jun 6, 2025 •
edited
Loading

Description

  • Avoid making modifications, such as ordering a legacy response, as it will be consumed later.
  • Avoid concurrent transformations like what's happening in the rudder-transformer JS code. Once we completely migrated to warehouse transformations, we can then later have concurrency set.

Linear Ticket

  • Resolves WAR-742, WAR-697

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov Codecov
Copy link

codecov bot commented Jun 6, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (release/1.50.x@a931c60). Learn more about missing BASE report.

Additional details and impacted files 
@@                Coverage Diff                @@
##             release/1.50.x    #5954   +/-   ##
=================================================
  Coverage                  ?   76.81%           
=================================================
  Files                     ?      494           
  Lines                     ?    68256           
  Branches                  ?        0           
=================================================
  Hits                      ?    52428           
  Misses                    ?    12952           
  Partials                  ?     2876

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@achettyiitr achettyiitr merged commit 0fed576 into release/1.50.x Jun 9, 2025
69 checks passed
@achettyiitr achettyiitr deleted the fix.warehouse-tranformations-responses-ordering branch June 9, 2025 04:19
@devops-github-rudderstack devops-github-rudderstack mentioned this pull request Jun 9, 2025
Merged
achettyiitr pushed a commit that referenced this pull request Jun 9, 2025
@devops-github-rudderstack
chore: release 1.50.6 (#5958)
47cf141 
🤖 I have created a release *beep* *boop*
---


##
[1.50.6](v1.50.5...v1.50.6)
(2025-06-09)


### Bug Fixes

* warehouse transformations responses ordering
([#5954](#5954))
([0fed576](0fed576))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
This was referenced Jun 9, 2025
Merged
Merged
Merged
Merged
Merged
atzoum pushed a commit that referenced this pull request Jun 9, 2025
@devops-github-rudderstack
chore: prerelease 1.51.0-rc.2 (#5967)
1f56168 
🤖 I have created a release *beep* *boop*
---


##
[1.51.0-rc.2](v1.50.0...v1.51.0-rc.2)
(2025-06-09)


### Features

* integrate the transformer client with cslb for source transformations
([#5881](#5881))
([53b8156](53b8156))
* triggers deployment for staging and hosted releases
([#5950](#5950))
([c8d849e](c8d849e))


### Bug Fixes

* block internal IP access as destination
([#5888](#5888))
([a0a4b91](a0a4b91))
* collected metric named router_process_jobs_count collides with
previously collected histogram named router_process_jobs
([#5966](#5966))
([d60d224](d60d224))
* **processor:** throughput stalling due to mutex deadlock on full
connection pool
([#5919](#5919))
([b18df16](b18df16))
* under reporting of load files when using upload id
([#5889](#5889))
([4a3a11c](4a3a11c))
* under reporting of load files when using upload id
([#5889](#5889))
([#5906](#5906))
([4f300cf](4f300cf))
* use jsonrs stdlib for encoding warehouse transformations
([#5945](#5945))
([0963a17](0963a17))
* warehouse transformations geo enrichment as string
([#5907](#5907))
([4f300cf](4f300cf))
* warehouse transformations responses ordering
([#5954](#5954))
([ea1a527](ea1a527))


### Miscellaneous

* add config to disable load table stats for clickhouse
([#5932](#5932))
([55c26f2](55c26f2))
* add event delivery metrics for async destinations
([#5930](#5930))
([7e9f1b5](7e9f1b5))
* add new metric regulation_worker_deletion_status_count
([#5939](#5939))
([1aa00f9](1aa00f9))
* add test case for dynamic config pattern with numeric default value
([#5904](#5904))
([9cd7180](9cd7180))
* clean up destinationInfo from refreshTokenParams
([#5897](#5897))
([4eb40cd](4eb40cd))
* **deps:** bump github.com/apache/pulsar-client-go from
0.15.1-candidate-2 to 0.15.1
([#5895](#5895))
([4a3a11c](4a3a11c))
* fix aws sdk v2
([#5928](#5928))
([e5b0ab4](e5b0ab4))
* flakiness in upload jobs count
([#5913](#5913))
([b167104](b167104))
* proc sample store uploader
([#5917](#5917))
([2e3b796](2e3b796))
* remove unused schemas repo in warehouse router
([#5879](#5879))
([4e96935](4e96935))
* **router:** adaptive throttler decreasing throttling rate dynamically
based on current throttled rate
([#5931](#5931))
([3d3b5b3](3d3b5b3))
* **router:** use different stats for capturing process stage jobs and
requests
([#5944](#5944))
([6728b2a](6728b2a))
* use zero transport on new common reporting client
([#5959](#5959))
([d35e2cc](d35e2cc))
* warehouse transformations json diff with encoding
([#5937](#5937))
([2e3b796](2e3b796))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
@devops-github-rudderstack devops-github-rudderstack mentioned this pull request Jun 9, 2025
Merged
atzoum pushed a commit that referenced this pull request Jun 9, 2025
@devops-github-rudderstack
chore: prerelease 1.51.0-rc.3 (#5969)
1dff520 
🤖 I have created a release *beep* *boop*
---


##
[1.51.0-rc.3](v1.50.0...v1.51.0-rc.3)
(2025-06-09)


### Features

* integrate the transformer client with cslb for source transformations
([#5881](#5881))
([53b8156](53b8156))
* triggers deployment for staging and hosted releases
([#5950](#5950))
([c8d849e](c8d849e))


### Bug Fixes

* block internal IP access as destination
([#5888](#5888))
([a0a4b91](a0a4b91))
* collected metric named router_process_jobs_count collides with
previously collected histogram named router_process_jobs
([#5966](#5966))
([d60d224](d60d224))
* **processor:** throughput stalling due to mutex deadlock on full
connection pool
([#5919](#5919))
([b18df16](b18df16))
* router request failing with unsupported media type error
([#5968](#5968))
([edaa10c](edaa10c))
* under reporting of load files when using upload id
([#5889](#5889))
([4a3a11c](4a3a11c))
* under reporting of load files when using upload id
([#5889](#5889))
([#5906](#5906))
([4f300cf](4f300cf))
* use jsonrs stdlib for encoding warehouse transformations
([#5945](#5945))
([0963a17](0963a17))
* warehouse transformations geo enrichment as string
([#5907](#5907))
([4f300cf](4f300cf))
* warehouse transformations responses ordering
([#5954](#5954))
([ea1a527](ea1a527))


### Miscellaneous

* add config to disable load table stats for clickhouse
([#5932](#5932))
([55c26f2](55c26f2))
* add event delivery metrics for async destinations
([#5930](#5930))
([7e9f1b5](7e9f1b5))
* add new metric regulation_worker_deletion_status_count
([#5939](#5939))
([1aa00f9](1aa00f9))
* add test case for dynamic config pattern with numeric default value
([#5904](#5904))
([9cd7180](9cd7180))
* clean up destinationInfo from refreshTokenParams
([#5897](#5897))
([4eb40cd](4eb40cd))
* **deps:** bump github.com/apache/pulsar-client-go from
0.15.1-candidate-2 to 0.15.1
([#5895](#5895))
([4a3a11c](4a3a11c))
* fix aws sdk v2
([#5928](#5928))
([e5b0ab4](e5b0ab4))
* flakiness in upload jobs count
([#5913](#5913))
([b167104](b167104))
* proc sample store uploader
([#5917](#5917))
([2e3b796](2e3b796))
* remove unused schemas repo in warehouse router
([#5879](#5879))
([4e96935](4e96935))
* **router:** adaptive throttler decreasing throttling rate dynamically
based on current throttled rate
([#5931](#5931))
([3d3b5b3](3d3b5b3))
* **router:** use different stats for capturing process stage jobs and
requests
([#5944](#5944))
([6728b2a](6728b2a))
* use zero transport on new common reporting client
([#5959](#5959))
([d35e2cc](d35e2cc))
* warehouse transformations json diff with encoding
([#5937](#5937))
([2e3b796](2e3b796))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RanjeetMishra RanjeetMishra RanjeetMishra approved these changes

@shekhar-rudder shekhar-rudder shekhar-rudder approved these changes

@vyeshwanth vyeshwanth Awaiting requested review from vyeshwanth

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@achettyiitr @RanjeetMishra @shekhar-rudder