Skip to content

Conversation

@Code42Cate
Copy link
Contributor

@Code42Cate Code42Cate commented Oct 13, 2025

Problem
When a process binds to localhost inside a sandbox, envd scans every 1s to detect it and starts socat to forward the port to the host IP (169.254.0.21). This creates a race condition where HTTP requests can arrive before forwarding is active, causing immediate connection failures.

Solution
Added retry logic in the proxy's DialContext with linear backoff:

  • 5 retry attempts with backoff: 100ms, 200ms, 300ms, 400ms
  • Total retry window: ~1000ms (covers envd's 1s scan interval)
  • The amount of retries is configurable. Only the orchestrator proxy has 5 retries, the client proxy has 1

Behavior
Best case: Port already forwarded → immediate success (0ms)
Average: Request arrives mid-scan → 1-2 retries, ~300ms delay
Worst case (success): Request arrives just before scan → 4-5 retries, ~1000ms delay
Worst case (failure): Port doesn't exist → ~1000ms then error

Testing
Added TestProxyRetriesOnDelayedBackendStartup that simulates a 300ms backend startup delay. Test passes with retries, fails without them. You can verify that behavior by reducing the attempts to 1. The tests will then fail


Note

Adds linear-backoff connection retries to proxy DialContext (5 for orchestrator, 1 for client) and updates pool/proxy APIs and tests, including a delayed-backend startup test.

  • Proxy Core:
    • Add MaxConnectionAttempts and constants ClientProxyRetries (1) and SandboxProxyRetries (5).
    • Update reverseproxy.New(port, maxConnectionAttempts, idleTimeout, getDestination) and pass attempts to pool.
  • Pool/Client:
    • Implement retry loop in pool/client.go DialContext with linear backoff (100–400ms) and context-aware exits.
    • Extend pool.New(maxClientConns, maxConnectionAttempts, idleTimeout) and propagate attempts in ProxyPool.Get to newProxyClient.
  • Orchestrator/Client Proxies:
    • Wire retries: orchestrator uses SandboxProxyRetries; client uses ClientProxyRetries.
  • Tests:
    • Add TestProxyRetriesOnDelayedBackendStartup to validate retry behavior.
    • Update test helpers to pass retry constant; replace several error checks with require.NoError.

Written by Cursor Bugbot for commit 95393d6. This will update automatically on new commits. Configure here.

cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

@Code42Cate Code42Cate changed the title Add client proxy retries to handle port forwarding delay Add orchestrator proxy retries to handle port forwarding delay Oct 13, 2025
cursor[bot]

This comment was marked as outdated.

@ValentaTomas ValentaTomas self-assigned this Oct 14, 2025
Copy link
Member

@ValentaTomas ValentaTomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Code42Cate Code42Cate merged commit f829cb6 into main Oct 14, 2025
27 checks passed
@Code42Cate Code42Cate deleted the add-client-proxy-retries branch October 14, 2025 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants