harden CircuitBreakerSpecs.cs
#7418
Conversation
harden `CircuitBreakerSpecs.Must_increment_failure_count_on_callTimeout_before_call_finishes`
| // meant to run as detached task | ||
| Task.Run(() => breaker.Instance.WithSyncCircuitBreaker(() => Thread.Sleep(Dilated(TimeSpan.FromSeconds(1))))); | ||
| var t = Task.Run(() => breaker.Instance.WithSyncCircuitBreaker(() => Thread.Sleep(Dilated(TimeSpan.FromSeconds(1))))); | ||
| await AwaitConditionAsync(() => t.Status >= TaskStatus.Running); // need to kick off the task before we can check the latch |
There was a problem hiding this comment.
This spec is pretty racy due to non-determinism of the TaskScheduler - now we force our assertions to wait until the Task.Run actually gets scheduled for execution, because otherwise the breaker never gets a chance to be scheduled and this will throw the timing off.
| _ = breaker.Instance.WithCircuitBreaker(() => Task.Run(SayHi)); | ||
| Enumerable.Range(1, 4).ForEach(_ => breaker.Instance.WithCircuitBreaker(() => Task.Run(ThrowException))); | ||
| await WaitForTaskToBeScheduled(breaker.Instance.WithCircuitBreaker(() => Task.Run(SayHi))); | ||
| await WaitForTaskToBeScheduled(Enumerable.Range(1, 4).Select(_ => breaker.Instance.WithCircuitBreaker(() => Task.Run(ThrowException))).ToList()); |
There was a problem hiding this comment.
Issues we've been running into this spec are due to scheduler lag again - when the CPU count is low these tasks may not get scheduled to run immediately.
|
These circuit breaker specs are kind of sketchy in the first place IMHO, as many of them rely on imprecise system clock behavior and the non-deterministic scheduling of the |
|
This has been impacted by #7419 a couple of times; we can debug that separately |
1 participant
Changes
harden
CircuitBreakerSpecs.Must_increment_failure_count_on_callTimeout_before_call_finishesChecklist
For significant changes, please ensure that the following have been completed (delete if not relevant):