A Dell EMC ECS rack installed on the vendor's default zone stamps its logs an hour or three away from every other system you correlate against. To modify timezone across all the nodes you do not go node by node — viprexec runs the same command on every one of them.
Before you modify timezone on ECS: Dell wants UTC
Worth knowing before you run any of this. Dell's position is that ECS is a UTC-only system, and xDoctor has a dedicated check for it — RAP012, "Time zone is not set to UTC", raised as an ERROR. The KB is explicit: "ECS must be set to the UTC time zone. If ECS is set to a non-UTC time zone, then it may cause further issues", and it names data replication, reporting, logging and VDC health checks as the functions that assume it. The documented remediation is to contact Dell support, not to set the zone yourself.
So treat what follows as something you do knowingly: the nodes will flag RAP012 on the next xDoctor run, and if you ever open a case it will be the first thing you are asked about. If all you actually want is readable local timestamps when you correlate logs, converting at the reader is the cheaper answer than moving the nodes off UTC.
Point /etc/localtime at the right zone
viprexec "mv /etc/localtime /etc/localtime-bkp"
viprexec "ln /usr/share/zoneinfo/Europe/Bucharest /etc/localtime"
The first line keeps the original rather than deleting it, which is the difference between a change and a one-way door. The rollback is then:
viprexec "mv -f /etc/localtime-bkp /etc/localtime"
Note that this creates a hard link, not a symlink. It works on these nodes, but it has three consequences worth knowing:
- A hard link cannot cross filesystems, so on any host where
/usris separate the barelnfails outright withln: failed to create hard link: Invalid cross-device link. timedatectland most tooling report the zone by reading the target of the symlink. With a hard link there is nothing to read, so they cannot name the zone at all.- A hard link keeps pointing at the same inode. When
tzdatais updated the package writes a new file over/usr/share/zoneinfo/..., so your/etc/localtimequietly keeps serving the old rules while the system's zone database has the new ones.
If you are doing this fresh, prefer the symlink — it follows the update:
viprexec "ln -sf /usr/share/zoneinfo/Europe/Bucharest /etc/localtime"
Then check it landed on every node, which is the step people skip:
viprexec "date"
viprexec "ls -l /etc/localtime"
date should show the same offset everywhere, and ls -l shows what /etc/localtime actually is — one line per node, so a node where the ln failed stands out immediately.
Force an NTP resync with ntpd -gnq
viprexec "systemctl stop ntpd"
viprexec "/usr/sbin/ntpd -gnq"
viprexec "systemctl start ntpd"
The flags: -g corrects the clock however large the offset, ignoring the 1000-second panic threshold — but only once; -n keeps the process in the foreground, which under viprexec is what gets you the output and exit status per node instead of a fork that reports nothing; -q exits as soon as the clock is set.
ntpd has to be stopped first because the running daemon owns UDP/123 and will not share it. Skip that step and the second instance exits with the NTP socket is in use, exiting.
Check the offset with ntpq -p
viprexec -i "ntpq -p"
-i prints the node identity alongside each block of output, which is what makes a per-node comparison readable.
The offset column is in milliseconds and signed, and should be small in absolute value and stable on every node. If one node is an outlier, do not read the offset alone — look at its reach column in the same output: 0 means it has never had an answer from that server, which is a routing or firewall problem rather than a clock problem. The * in front of a peer marks the one the daemon is actually synchronised to and + marks the others it considers usable; no * anywhere means nothing has been selected yet, which is normal for the first minute and a problem after five.
Daemons keep the old timezone until they are restarted
At the Linux level, changing the zone is display only — the system clock stays UTC and nothing about how time is stored changes. But a process that was already running keeps the old zone until it is restarted, because the timezone is cached per process the first time it converts a time. The one that bites hardest is the logging daemon, which then stamps every line with the old offset while date shows the new one, and nothing warns you.
That failure is worth knowing in its own right — the symptom, two ways to confirm it and the one-line fix are in wrong time in /var/log after a timezone change. That post is written against RHEL 9 while these nodes run SUSE Linux Enterprise, but the mechanism is glibc's, so it behaves identically here. Applied to ECS: after you modify timezone on the nodes, restart the services that write timestamps, or accept that everything logged until the next reboot is an hour out.
