OCPBUGS-64937: vsphere - Cache REST API sessions to prevent excessive vCenter logouts #1432
Conversation
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/test ? |
|
@jcpowermac: The following commands are available to trigger required jobs: The following commands are available to trigger optional jobs: Use DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test e2e-vsphere-ovn-serial |
|
/test e2e-vsphere-ovn |
|
Testing with #1446 showed this is a problem, moving forward with testing this. /test e2e-vsphere-ovn |
|
/test golint |
jcpowermac
force-pushed
the
fix-vsphere-rest-session-caching
branch
from
2caad48 to
3b31029
Compare
jcpowermac
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@jcpowermac: This pull request references Jira Issue OCPBUGS-64937, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/test all |
|
/jira refresh |
openshift-ci-robot
added
jira/severity-moderate
|
@jcpowermac: This pull request references Jira Issue OCPBUGS-64937, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
This commit fixes a critical issue where the machine-api-operator was creating and destroying vCenter REST API sessions on every machine reconciliation, causing excessive login/logout cycles that pollute vCenter audit logs and create unnecessary session churn. Root Cause: The WithRestClient() and WithCachingTagsManager() wrapper functions were creating new REST sessions, performing operations, and immediately logging out on every invocation. With hundreds of machines reconciling periodically, this created a constant stream of login/logout events. Solution (inspired by cluster-api-provider-vsphere): - Add TagManager field to Session struct to cache REST client - Initialize and cache REST client during session creation (GetOrCreate) - Validate both SOAP and REST session health before reusing cached sessions - Add GetCachingTagsManager() helper for direct access to cached tag manager - Update reconcileRegionAndZoneLabels() to use cached tag manager - Update reconcileTags() to use cached tag manager - Deprecate WithRestClient() and WithCachingTagsManager() for backward compatibility Key Changes: 1. pkg/controller/vsphere/session/session.go: - Added TagManager *tags.Manager field to Session struct - Modified GetOrCreate() to create and cache REST client once - Added dual session validation (SOAP + REST) before reusing sessions - Added GetCachingTagsManager() method for direct access - Deprecated old wrapper functions with migration guidance 2. pkg/controller/vsphere/reconciler.go: - Updated reconcileRegionAndZoneLabels() to use GetCachingTagsManager() - Updated reconcileTags() to use GetCachingTagsManager() - Eliminated callback pattern in favor of direct access Impact: - Eliminates excessive vCenter login/logout cycles - Reduces vCenter session churn from O(reconciliations) to O(1) per MAPI instance - Improves performance by removing authentication overhead on every tag operation - REST session now lives as long as SOAP session (until invalidation) Reference Implementation: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/pkg/session/session.go Backward Compatibility: The deprecated wrapper functions are maintained with warning logs to support existing test code. All production code paths now use the new pattern. Fixes: Excessive vCenter logout events reported by customers Signed-off-by: Claude Code Assistant <[email protected]> Co-Authored-By: Claude <[email protected]>
jcpowermac
force-pushed
the
fix-vsphere-rest-session-caching
branch
from
3b31029 to
114d062
Compare
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
|
/test e2e-vsphere-ovn-techpreview-serial |
|
@jcpowermac: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/lgtm |
|
/assign @damdo |
|
/verified by @jcpowermac via local install and ci |
openshift-ci-robot
added
the
verified
|
@jcpowermac: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
4 participants
This commit fixes a critical issue where the machine-api-operator was creating and destroying vCenter REST API sessions on every machine reconciliation, causing excessive login/logout cycles that pollute vCenter audit logs and create unnecessary session churn.
Root Cause:
The WithRestClient() and WithCachingTagsManager() wrapper functions were creating new REST sessions, performing operations, and immediately logging out on every invocation. With hundreds of machines reconciling periodically, this created a constant stream of login/logout events.
Solution (inspired by cluster-api-provider-vsphere):
Key Changes:
pkg/controller/vsphere/session/session.go:
pkg/controller/vsphere/reconciler.go:
Impact:
Reference Implementation:
https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/pkg/session/session.go
Backward Compatibility:
The deprecated wrapper functions are maintained with warning logs to support existing test code. All production code paths now use the new pattern.
Fixes: Excessive vCenter logout events reported by customers