Someone sets the system clock by hand — from a control panel, or with a date command — and from that moment the box stays wrong far longer than it has any need to. timedatectl reports System clock synchronized: no, every source is unreachable, and the obvious suspicion is that something is blocking NTP. It is not. chrony owns the clock, and a manual jump strands it in a state it cannot leave on its own. This is on RHEL 9, AlmaLinux 9 and Rocky Linux 9, where chrony is the default.
Worth separating first: changing the timezone is harmless. A timezone is a display setting — UTC and every timestamp inside the system are unaffected, so timedatectl set-timezone does not cause any of this. It is setting the clock that does the damage. (If your logs went wrong after a timezone change, that is a different problem — see rsyslog showing the wrong time in /var/log.)
"Backward time jump detected!" in the journal
In the journal, the two lines that tell the whole story sit hours apart:
Sep 07 18:48:27 alma-a01 chronyd[1032]: Selected source 192.0.2.185
Sep 07 20:30:45 alma-a01 chronyd[1032]: Backward time jump detected!
Sep 07 20:30:45 alma-a01 chronyd[1032]: Can't synchronise: no selectable sources (0 unreachable sources)
Backward time jump detected! means a human or a panel set the clock. It is not a fault, a drift or a network problem — something outside chrony moved the clock underneath it. That single line saves you an hour of looking in the wrong place.
And the state it leaves behind:
[root@alma-a01 ~]# chronyc tracking
Reference ID : 00000000 ()
Stratum : 0
Ref time (UTC) : Thu Jan 01 00:00:00 1970
Reference ID 00000000 and Stratum 0 mean chrony is synchronised to nothing at all.
The diagnostic that matters: read the Reach column
Before touching the firewall, run this:
[root@alma-a01 ~]# chronyc -n sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.0.2.185 2 6 377 44 +43000ms[+43000ms] +/- 21ms
^? 198.51.100.82 2 6 377 45 +43000ms[+43000ms] +/- 18ms
^? 203.0.113.19 2 6 377 43 +43000ms[+43000ms] +/- 25ms
Reach 377 is the answer. It is an octal bit field of the last eight polls, and 377 octal is 11111111 binary — all eight answered. The network is fine. NTP is passing the firewall. Every source is replying.
So a "not synchronized" verdict with good reach is a chrony state problem, and never a firewall problem. Do not go re-opening UDP 123 over this. NTP here is outbound-initiated and only needs 123 in your firewall's outbound list; if the packets were being dropped, Reach would be 0 and LastRx would climb without bound.
The ^? in the first column is the other half: the sources answer, but chrony has marked every one of them unusable.
This post is about the case where Reach is full and chrony still will not sync. If Reach never climbs past 0, or flaps between values, that is a different problem with a different cause — usually a stateful firewall whose UDP connection-tracking timeout is shorter than chrony's poll interval, so the reply arrives after the state entry has already been dropped.
Why chrony resyncs so slowly on its own
Look at /etc/chrony.conf:
makestep 1.0 3
That means: step the clock rather than slew it, but only for the first 3 updates after startup. Stepping is the abrupt correction; slewing is speeding up or slowing down the clock until it catches up, which for an error this size is far too slow to be worth waiting for.
Those three steps were spent at boot, when the daemon first disciplined the clock. After a later manual jump chrony is only allowed to slew. On top of that the jump invalidates its sample history, so it first has to rebuild enough measurements to select a source at all — which is why the window above shows every source as ^?.
It does get there eventually, on its own, in minutes rather than never. But it then arrives at the slow path: a correction it may only slew. That is the trap. The one mechanism that would fix a large error quickly is the one that is no longer available, and nothing about the situation tells you that you are now waiting on the slow one.
The fix: restart chronyd
systemctl restart chronyd
A restart grants a fresh makestep 1.0 3 budget, and because the config uses iburst the sources are re-polled immediately rather than at the normal interval:
20:37:54 alma-a01 chronyd[2140]: Selected source 198.51.100.82
20:37:54 alma-a01 chronyd[2140]: System clock wrong by 42.998410 seconds
20:38:37 alma-a01 chronyd[2140]: System clock was stepped by 42.998410 seconds
About a minute, start to finish.
chronyc makestep on its own is the answer most search results give, and on its own it is the wrong tool here: it only re-applies a correction that is already in flight, and while nothing is selected there is no such correction. Paired with burst it does work, because burst forces the measurements it needs first:
chronyc burst 2/2
chronyc makestep 1.0 1
That is the surgical version if you would rather not restart the daemon. The restart is simpler and gets you the same fresh budget.
Verify
[root@alma-a01 ~]# timedatectl
System clock synchronized: yes
NTP service: active
[root@alma-a01 ~]# chronyc tracking | head -4
Reference ID : C6336452 (198.51.100.82)
Stratum : 3
System time : 0.000001975 seconds slow of NTP time
[root@alma-a01 ~]# chronyc -n sources | head -3
^* 198.51.100.82 2 6 377 21 -12us[ -18us] +/- 18ms
Three things to check, in this order: System clock synchronized: yes, a real Reference ID instead of 00000000, and at least one source marked ^* — the currently selected one — rather than ^?.
If rtcsync is in your chrony.conf, the hardware clock is rewritten to match UTC automatically, so you do not need hwclock -w afterwards. If it is not in your config, run hwclock -w once the clock is correct to write the new time to the RTC. Either way, check with timedatectl that RTC time now agrees with Universal time.
Why it is worth fixing rather than ignoring
A box tens of seconds out of step is not a cosmetic problem:
- Certificate issuance breaks. Let's Encrypt and any ACME client validate time-sensitive nonces and signatures. A renewal that fails for no visible reason, on a host that was fine last month, is worth a
timedatectlbefore anything else. - Log correlation stops working. Comparing this host's
/var/logagainst another one's silently misleads you, and the offset is invisible in the file. - Mail
Date:headers go wrong, which some receivers score against you.
How to approach it next time
When a host reports that the clock is not synchronised, check three things before changing anything:
timedatectl
chronyc -n sources # read the Reach column FIRST
journalctl -u chronyd -b
Reach 377 plus ^? on every line means chrony, not the network. A Backward time jump detected! in the journal means a person or a panel, not a fault. Then restart the daemon — and go find out who set the clock, because nothing stops it happening again next week.
