[fix][operator] RayJob.Status.RayJobStatusInfo.EndTime nil deref error (#3742)

If users run the latest ray-operator code from master branch without updating
their RayJob CRD, the operator panics from a nil pointer dereference error in
`rayjob_controller.go`. `rayJob.Status.RayJobStatusInfo.EndTime` is nil when a
RayJob's Job fails with RayJob CRD from v1.3.2.

We check if `EndTime` is nil and return false from
`checkTransitionGracePeriodAndUpdateStatusIfNeeded()` if so.

Signed-off-by: David Xia <[email protected]>

Author: davidxia

ray-operator/controllers/ray/rayjob_controller.go

@@ -935,6 +935,13 @@ func checkTransitionGracePeriodAndUpdateStatusIfNeeded(ctx context.Context, rayJ
 			rayJobDeploymentGracePeriodTime = utils.DEFAULT_RAYJOB_DEPLOYMENT_STATUS_TRANSITION_GRACE_PERIOD_SECONDS
 		}
 
+		// EndTime isn't nil when JobStatus is in a terminal state under normal conditions.
+		// This check is a nil pointer dereference safeguard when older RayJob CRDs without the
+		// RayJobStatusInfo field are used with newer versions of KubeRay operator.
+		if rayJob.Status.RayJobStatusInfo.EndTime == nil {
+			return false
+		}
+
 		if time.Now().Before(rayJob.Status.RayJobStatusInfo.EndTime.Add(time.Duration(rayJobDeploymentGracePeriodTime) * time.Second)) {
 			return false
 		}