Clock drifts constantly on macOS because fedora's pool is unavailable

[nicdoye]

Describe the bug
The clock in the underlying VM on macOS drifts. Even over this weekend, it drifted nearly 2 mins.

Steps to reproduce
Start finch on macOS, run a long running container. Wait a couple of days. Run date and see it is no longer accurate

Expected behavior
I expect sub-second accuracy of the VM's clock.

Screenshots or logs
Sample logs on the VM:

Oct 21 10:07:18 lima-finch chronyd[665]: Source 176.58.127.165 replaced with 178.62.18.76 (2.fedora.pool.ntp.org)
Oct 21 10:08:50 lima-finch chronyd[665]: Detected falseticker 129.250.35.250 (2.fedora.pool.ntp.org)
Oct 21 10:08:50 lima-finch chronyd[665]: Selected source 213.5.132.231 (2.fedora.pool.ntp.org)
Oct 21 10:08:54 lima-finch chronyd[665]: Can't synchronise: no majority
Oct 21 10:09:54 lima-finch chronyd[665]: Detected falseticker 129.250.35.250 (2.fedora.pool.ntp.org)
Oct 21 10:09:54 lima-finch chronyd[665]: Selected source 213.5.132.231 (2.fedora.pool.ntp.org)
Oct 21 10:10:00 lima-finch chronyd[665]: Can't synchronise: no majority
Oct 21 10:10:59 lima-finch chronyd[665]: Detected falseticker 129.250.35.250 (2.fedora.pool.ntp.org)
Oct 21 10:10:59 lima-finch chronyd[665]: Selected source 213.5.132.231 (2.fedora.pool.ntp.org)
Oct 21 10:11:04 lima-finch chronyd[665]: Can't synchronise: no majority

Additional context
Replacing the fedora pool in /etc/chrony.conf with: pool time.aws.com iburst and running sudo service chronyd force-reload might fix the problem (it certainly synced the clock this morning, but only time will tell).

[austinvazquez]

(it certainly synced the clock this morning, but only time will tell)

😄

Interesting, it looks like there is some history with this issue reported before. re: #248

@nicdoye, can you confirm the virtual machine type the issue was observed?

[nicdoye]

Yep, I have vmType: vz

[austinvazquez]

Thanks, looking to create a repro on my side to see if can get any new information on this issue.

[austinvazquez]

Running container for 72 hours with no drift between host, VM, and container clock.

Details:

• finch v1.2.3
• VM type: vz
• macOS 14.6.1
• 2.6 GHz 6-core Intel Core i7

[austinvazquez]

@nicdoye, just to verify when updating the VM type from qemu to vz, it would require a VM re-initialization. i.e. finch vm stop && finch vm remove && finch vm init

Otherwise, perhaps can you share your Finch, macOS, and architecture you are experiencing the issue?

[nicdoye]

I don't think I've ever bothered with the qemu type. I did remove and init the vm a couple of weeks ago. I first reported this back in February (#837) on 1.1.2 and now I've seen it on 1.4.0 and 1.4.1

• finch v1.4.1
• VM type: vz
• macOS 15.0.1
• M1 Pro 16" 2021

A second M1 Mac runs Podman, which is also based on fedora (39 not 40) on Apple Virtualisation, and while it's not used as much as my work Mac, I've not seen the same issue there. It seems to have the same/similar chronyd setup as the finch one, here.

I'll fire finch up there, and see if I can get it to happen. Maybe our MDM, or using cap SYS_ADMIN or something is causing the issue. 🤷🏻‍♂️

[nicdoye]

This morning, both Macs have drifted. The finch vm on my work laptop is over 15 minutes behind. My personal laptop is about 75 seconds behind.

I've attached a support bundle and the output from the vm's sudo systemctl status chronyd --lines 1000 --full in chronyd.log from my work laptop.

On my work laptop I see in the vm, 20 occurrences of systemd-journald[476]: Time jumped backwards, rotating. starting at 07:07 this morning (a couple of hours before I started work).

[lima@lima-finch tmp]$ sudo dmesg -T | grep -c jumped
20

finch-support-20241111100418.zip
chronyd.log