Prevent render failure from leaking secrets #983
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary --- > We handle Kubernetes secrets, so it is critical that changes do not cause the contents of any secrets in the template set to be logged.[^1] When a `krane render` task fails it can potentially leak base64 encoded secrets in the clear. Context ---- This was discovered when a developer used Ruby's `Base64.encode64` method to produce a secret value. The value was long enough to produced an encoded string longer than 60 characters. > The returned string ends with a newline character, and if sufficiently long will have one or more embedded newline characters...[^2] > An encoded string returned by Base64.encode64 or Base64.urlsafe_encode64 has an embedded newline character after each 60-character sequence, and, if non-empty, at the end: [^3] This caused krane to dump the base64 encoded secrets to the deployment logs. Deployment tasks take action to prevent secrets leaking in this way, and we can reuse that logic for rendering failures. 🎩 Example before: ``` [INFO][2025-07-09 10:14:03 -0400] [INFO][2025-07-09 10:14:03 -0400] ---------------------------------Phase 1: Initializing render task---------------------------------- [INFO][2025-07-09 10:14:03 -0400] Validating configuration [INFO][2025-07-09 10:14:03 -0400] [INFO][2025-07-09 10:14:03 -0400] -----------------------------------Phase 2: Rendering template(s)----------------------------------- [ERROR][2025-07-09 10:14:03 -0400] Failed to render secret.yaml.erb [INFO][2025-07-09 10:14:03 -0400] [INFO][2025-07-09 10:14:03 -0400] ------------------------------------------Result: FAILURE------------------------------------------- [FATAL][2025-07-09 10:14:03 -0400] Invalid template: secret.yaml.erb [FATAL][2025-07-09 10:14:03 -0400] > Error message: [FATAL][2025-07-09 10:14:03 -0400] (<rendered> secret.yaml.erb): could not find expected ':' while scanning a simple key at line 10 column 1 [FATAL][2025-07-09 10:14:03 -0400] > Template content: [FATAL][2025-07-09 10:14:03 -0400] [FATAL][2025-07-09 10:14:03 -0400] apiVersion: v1 [FATAL][2025-07-09 10:14:03 -0400] kind: Secret [FATAL][2025-07-09 10:14:03 -0400] metadata: [FATAL][2025-07-09 10:14:03 -0400] name: invalid-secret [FATAL][2025-07-09 10:14:03 -0400] type: Opaque [FATAL][2025-07-09 10:14:03 -0400] data: [FATAL][2025-07-09 10:14:03 -0400] username: YWRtaW4= [FATAL][2025-07-09 10:14:03 -0400] password: KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq [FATAL][2025-07-09 10:14:03 -0400] KioqKioqKioqKioqKioqKioqKioqKioqKg== [FATAL][2025-07-09 10:14:03 -0400] [FATAL][2025-07-09 10:14:03 -0400] ``` Example After: ``` [INFO][2025-07-09 10:27:04 -0400] [INFO][2025-07-09 10:27:04 -0400] ---------------------------------Phase 1: Initializing render task---------------------------------- [INFO][2025-07-09 10:27:04 -0400] Validating configuration [INFO][2025-07-09 10:27:04 -0400] [INFO][2025-07-09 10:27:04 -0400] -----------------------------------Phase 2: Rendering template(s)----------------------------------- [INFO][2025-07-09 10:27:04 -0400] [INFO][2025-07-09 10:27:04 -0400] ------------------------------------------Result: FAILURE------------------------------------------- [FATAL][2025-07-09 10:27:04 -0400] Invalid template: secret.yaml.erb [FATAL][2025-07-09 10:27:04 -0400] > Error message: [FATAL][2025-07-09 10:27:04 -0400] (<rendered> secret.yaml.erb): could not find expected ':' while scanning a simple key at line 10 column 1 [FATAL][2025-07-09 10:27:04 -0400] > Template content: Suppressed because it may contain a Secret ``` [^1]: https://github.com/Shopify/krane/blob/72ee07a311442751535b87a23ea843b71190e618/CONTRIBUTING.md?plain=1#L44 [^2]: https://ruby-doc.org/3.4.1/gems/base64/Base64.html#method-i-encode64 [^3]: https://ruby-doc.org/3.4.1/gems/base64/Base64.html#module-Base64-label-Newlines
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What are you trying to accomplish with this PR?
When a
krane rendertask fails it can potentially leak base64 encoded secrets in the clear.How is this accomplished?
This was discovered when a developer used Ruby's
Base64.encode64method to produce a secret value. The value was long enough to produced an encoded string longer than 60 characters.This caused krane to dump the base64 encoded secrets to the deployment logs.
Deployment tasks take action to prevent secrets leaking in this way, and we can reuse that logic for rendering failures.
🎩
Example before:
Example After:
What could go wrong?
From the outside, this doesn't seem to open krane up to any additional runtime risk.
Footnotes
https://github.com/Shopify/krane/blob/72ee07a311442751535b87a23ea843b71190e618/CONTRIBUTING.md?plain=1#L44 ↩
https://ruby-doc.org/3.4.1/gems/base64/Base64.html#method-i-encode64 ↩
https://ruby-doc.org/3.4.1/gems/base64/Base64.html#module-Base64-label-Newlines ↩