Configurable debounce (meta-pytorch#854)

Summary:

When we increase the number of actors in our simulation it takes longer for all the events at a certain time to complete so we need to wait for longer. If we wait to long then the simulation just runs slower than it needs to so its nice to make this configurable. 

In the long term we will come up with a more robust solution to this but in the meantime that is not a priority. See EX528476 to understand the underlying problem the debounce is remedying

Reviewed By: pablorfb-meta

Differential Revision: D80137965

Author: thomasywang

hyperactor/src/simnet.rs

@@ -569,14 +569,23 @@ impl SimNet {
         let mut training_script_waiting_time = tokio::time::Duration::from_millis(0);
         // Duration elapsed while only non_advanceable_events has events
         let mut debounce_timer: Option<tokio::time::Instant> = None;
+
+        let debounce_duration = std::env::var("SIM_DEBOUNCE")
+            .ok()
+            .and_then(|val| val.parse::<u64>().ok())
+            .unwrap_or(1);
+
         'outer: loop {
             // Check if we should stop
             if stop_signal.load(Ordering::SeqCst) {
                 break 'outer self.records.clone();
             }
 
             while let Ok(Some((event, advanceable, time))) = RealClock
-                .timeout(tokio::time::Duration::from_millis(1), event_rx.recv())
+                .timeout(
+                    tokio::time::Duration::from_millis(debounce_duration),
+                    event_rx.recv(),
+                )
                 .await
             {
                 let scheduled_event = match time {