Skip to content

fix: embedded transformations upload #5848

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 13, 2025
Merged

fix: embedded transformations upload #5848

merged 1 commit into from
May 13, 2025

Conversation

achettyiitr
Copy link
Member

@achettyiitr achettyiitr commented May 13, 2025 •
edited
Loading

Description

  • Use a go-routine when you only need to compare and upload. In case of early return, no need for a go-routine. In most cases, when the limit is reached, we will be unnecessarily spinning a go-routine which is not required.
  • Use warehouse transformations for snowpipe streaming as well.
  • Resolves WAR-654

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@achettyiitr achettyiitr force-pushed the fix.warehouse-transformer-uploader branch 2 times, most recently from e9b684c to fff9e3d Compare May 13, 2025 06:24
@achettyiitr achettyiitr requested review from ktgowtham, itsmihir and fracasula and removed request for ktgowtham, itsmihir and fracasula May 13, 2025 06:24
@achettyiitr achettyiitr force-pushed the fix.warehouse-transformer-uploader branch 2 times, most recently from 3ded262 to 18cc0c4 Compare May 13, 2025 06:28
@achettyiitr achettyiitr changed the title fix: warehouse transformer uploader fix: embedded transformations upload May 13, 2025
@achettyiitr achettyiitr force-pushed the fix.warehouse-transformer-uploader branch from 18cc0c4 to 3b7bbf2 Compare May 13, 2025 06:42
@achettyiitr achettyiitr force-pushed the fix.warehouse-transformer-uploader branch from 3b7bbf2 to 0c7be0b Compare May 13, 2025 06:44
@itsmihir itsmihir requested review from fracasula, cisse21 and atzoum May 13, 2025 06:55
@achettyiitr achettyiitr merged commit 0b19968 into master May 13, 2025
59 checks passed
@achettyiitr achettyiitr deleted the fix.warehouse-transformer-uploader branch May 13, 2025 07:14
This was referenced May 13, 2025
Merged
Merged
satishrudderstack pushed a commit that referenced this pull request May 13, 2025
@devops-github-rudderstack
chore: release 1.49.0 (#5849)
3cda0e6 
🤖 I have created a release *beep* *boop*
---


##
[1.49.0](v1.48.0...v1.49.0)
(2025-05-13)


### Features

* add dynamic config existence flag for destinations
([#5821](#5821))
([1b4ea06](1b4ea06))
* add support for processing of upload_v2 job type by slave
([#5796](#5796))
([67988d1](67988d1))
* batch staging files for creating upload_v2 notifier jobs
([#5765](#5765))
([7dc3a45](7dc3a45))
* enable worker-based kafka client batching
([#5788](#5788))
([66dc19f](66dc19f))
* support claim renewal in notifier jobs
([#5818](#5818))
([a499d9c](a499d9c))
* support new consent resolution strategy values
([#5798](#5798))
([181b95b](181b95b))
* update config to add account details with destination in
workspaceConfig
([#5753](#5753))
([dbd46bf](dbd46bf))
* update contract for account and accountDefinition
([#5830](#5830))
([c704b07](c704b07))
* use account to decide oauth type of a destination
([#5810](#5810))
([9165e8c](9165e8c))


### Bug Fixes

* add check for nil secret on oauthv2
([#5807](#5807))
([fbc4abe](fbc4abe))
* convert bad request errors to 500 errors in oauth interceptor to
prevent panics
([#5813](#5813))
([3a08ec4](3a08ec4))
* embedded transformations upload
([#5848](#5848))
([0b19968](0b19968))
* **jobsdb:** completed datasets don't get deleted without a pair
([#5793](#5793))
([54aee71](54aee71))
* reporting common client path with query
([#5842](#5842))
([c068920](c068920))
* transformer client recycle ttl bound to connection idle timeout
([#5800](#5800))
([b13f92c](b13f92c))
* update account type to remove id which we are getting as map key
([#5835](#5835))
([bc52242](bc52242))
* ut mirroring race condition
([#5824](#5824))
([a4d579f](a4d579f))
* warehouse cached schema mismatch
([#5805](#5805))
([4656247](4656247))
* warehouse transformations mismatches
([#5779](#5779))
([01a7b83](01a7b83))


### Miscellaneous

* **deps:** bump github.com/snowflakedb/gosnowflake from 1.13.2 to
1.13.3 in the go_modules group
([#5787](#5787))
([41db33e](41db33e))
* **deps:** bump golangci/golangci-lint-action from 7 to 8
([#5815](#5815))
([bf3e808](bf3e808))
* enrich event with bot details
([#5836](#5836))
([6035658](6035658))
* **jobsdb:** cache distinct parameters query result for all datasets
except last
([#5752](#5752))
([e85811e](e85811e))
* migrate from denisenkom/go-mssqldb to microsoft/go-mssqldb
([#5776](#5776))
([dbd46bf](dbd46bf))
* revert synapse staging table with max varchar length
([#5817](#5817))
([2418329](2418329))
* **router:** support destination-specific configuration overrides for
all options
([#5841](#5841))
([695cf53](695cf53))
* synapse staging table with max varchar length
([#5775](#5775))
([2fc5384](2fc5384))
* update accountDefination type with authenticationType
([#5791](#5791))
([2a24e59](2a24e59))
* update rudder-go-kit to 0.49.2
([#5832](#5832))
([01a7b83](01a7b83))
* upload embedded dt response difference samples to s3
([#5792](#5792))
([c81001f](c81001f))
* warehouse transformer migration to embedded destination transformer
package
([#5827](#5827))
([e124bb0](e124bb0))
* warehouse transformer sample diff
([#5837](#5837))
([8af55de](8af55de))
* warehouse transformer uploader
([#5828](#5828))
([fc640bf](fc640bf))
* webhook integration test module upgrade
([#5665](#5665))
([f4130d1](f4130d1))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
@codecov Codecov
Copy link

codecov bot commented May 13, 2025

Codecov Report

Attention: Patch coverage is 45.00000% with 11 lines in your changes missing coverage. Please review.

Project coverage is 77.03%. Comparing base (1b4ea06) to head (0c7be0b).
Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
...ination_transformer/embedded/warehouse/uploader.go 20.00% 4 Missing ⚠️
...rnal/transformer/destination_transformer/logger.go 0.00% 4 Missing ⚠️
...destination_transformer/destination_transformer.go 72.72% 2 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #5848      +/-   ##
==========================================
- Coverage   77.14%   77.03%   -0.12%     
==========================================
  Files         496      496              
  Lines       67838    67850      +12     
==========================================
- Hits        52334    52265      -69     
- Misses      12690    12761      +71     
- Partials     2814     2824      +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ktgowtham ktgowtham ktgowtham approved these changes

@itsmihir itsmihir itsmihir approved these changes

@fracasula fracasula Awaiting requested review from fracasula

@cisse21 cisse21 Awaiting requested review from cisse21

@atzoum atzoum Awaiting requested review from atzoum

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@achettyiitr @ktgowtham @itsmihir