Network Time & Synchronization

The Clock Is Infrastructure#

Every log line, database transaction, certificate check, token expiry, and distributed consensus decision depends on the machines involved agreeing on what time it is. When clocks drift apart, systems fail in confusing ways: logs from different hosts appear in the wrong order, valid certificates are rejected, one-time codes stop working, and distributed databases cannot tell which write came first. Time synchronisation is quiet infrastructure that everything else assumes.

Every computer keeps time with a quartz oscillator, and no two oscillators run at exactly the same rate. Temperature and manufacturing variation make an unsynchronised clock drift by seconds per week, and virtual machines can do far worse after being paused or migrated. Synchronisation continuously measures and corrects that error.

NTP and the Stratum Hierarchy#

The Network Time Protocol (NTP), on UDP port 123, distributes time through a hierarchy of strata:

• Stratum 0: reference clocks such as GPS receivers and atomic clocks. They are not on the network themselves.
• Stratum 1: servers directly attached to a stratum 0 source.
• Stratum 2: servers synchronised to stratum 1, and so on down.

Stratum counts hops from a reference, not quality. A nearby, stable stratum 3 server can give better results than a distant, congested stratum 1.

How NTP Measures Offset#

A client can't simply ask "what time is it?", because the answer ages while it crosses the network. NTP records four timestamps:

• t1: client sends the request (client clock)
• t2: server receives it (server clock)
• t3: server sends the reply (server clock)
• t4: client receives the reply (client clock)

From these, the round-trip delay is (t4 - t1) - (t3 - t2), which excludes the server's processing time, and the client's offset from the server is ((t2 - t1) + (t3 - t4)) / 2. The formula assumes the outbound and return trips take equal time. Any asymmetry in the path becomes error in the offset, up to half the difference, which is why nearby, well-connected servers produce better time.

A minimal SNTP client makes the arithmetic concrete:

import socket
import struct
import time

NTP_EPOCH_OFFSET = 2_208_988_800        # seconds from 1900-01-01 to 1970-01-01

def to_ntp(ts):
    secs = int(ts)
    return struct.pack("!II", secs + NTP_EPOCH_OFFSET, int((ts - secs) * 2**32))

def from_ntp(data):
    secs, frac = struct.unpack("!II", data)
    return secs - NTP_EPOCH_OFFSET + frac / 2**32

def query(server="pool.ntp.org"):
    packet = bytearray(48)
    packet[0] = 0x23                     # LI=0, version 4, mode 3 (client)
    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
        sock.settimeout(3)
        t1 = time.time()
        packet[40:48] = to_ntp(t1)       # our transmit time; the server echoes it back
        sock.sendto(packet, (server, 123))
        reply, _ = sock.recvfrom(512)
        t4 = time.time()

    if reply[24:32] != packet[40:48]:
        raise ValueError("reply does not match our request")
    stratum = reply[1]
    t2 = from_ntp(reply[32:40])          # server receive time
    t3 = from_ntp(reply[40:48])          # server transmit time
    offset = ((t2 - t1) + (t3 - t4)) / 2
    delay = (t4 - t1) - (t3 - t2)
    return stratum, offset, delay

stratum, offset, delay = query()
print(f"stratum {stratum}  offset {offset*1000:+.2f} ms  round-trip delay {delay*1000:.2f} ms")

stratum 2  offset +8.97 ms  round-trip delay 19.76 ms

The non-obvious details: NTP timestamps count from 1900 rather than 1970, so the constant converts between the two epochs, and the fractional part is a 32-bit binary fraction of a second. The server copies the client's transmit timestamp into the "originate" field of its reply (bytes 24–31), and checking that it matches ours rejects stray or spoofed replies that were not answers to this request. A positive offset means the local clock is behind the server.

A real NTP daemon does much more than this one-shot query. It polls several servers repeatedly, filters out samples with unusually high delay, discards "falsetickers" whose time disagrees with the majority, and steers the clock gradually.

Slewing vs. Stepping#

A daemon corrects small errors by slewing: running the clock slightly fast or slow until the offset disappears, so time never jumps and never runs backwards. Large errors, typically at boot, are fixed by stepping, a one-off jump. Steps are the dangerous kind: a backwards step can make a log show a reply before its request, reorder events, or trigger timeouts. The usual policy is to allow steps only during startup.

Running chrony#

On modern Linux, chrony is the common NTP implementation. A solid /etc/chrony.conf:

# Several independent sources so a single bad one can be outvoted
pool pool.ntp.org iburst maxsources 4
# An authenticated source using Network Time Security
server time.cloudflare.com iburst nts

# Step the clock only if it is off by more than 1 s, and only in the first 3 updates
makestep 1.0 3
# Keep the hardware clock in sync for the next boot
rtcsync

Check the state with:

chronyc tracking          # current offset, frequency error, stratum, leap status
chronyc sources -v        # each source; '*' = selected, '+' = combined, 'x' = falseticker, '?' = unreachable
timedatectl status        # "System clock synchronized: yes"

In chronyc tracking, System time shows how far the clock currently is from NTP time, Frequency shows how fast the local oscillator drifts (in parts per million), and Leap status should read Normal. On Windows, use w32tm /query /status, and w32tm /stripchart /computer:time.windows.com /samples:5 to measure the offset against a specific server.

Best practices:

• Use at least four sources when you rely on public servers. With two, the client cannot tell which one is wrong; with three or more, it can outvote a bad one.
• In a data centre, run two or more internal servers (ideally with their own GPS references or synchronised to diverse upstreams) and point every host at them, so thousands of machines do not each query the internet.
• Alert on synchronisation loss and on offsets above your tolerance, not just on the daemon process being up.

Why Drift Breaks Things#

The authentication failures are covered from the other side in authentication concepts. For correlating logs across hosts during an investigation, see detection and SIEM: a few seconds of skew is enough to put a login after the activity it enabled.

Time in Your Own Code#

Two rules prevent most time bugs in applications:

• Measure durations with a monotonic clock. time.time() follows the wall clock, which NTP can step. A timeout computed as the difference of two time.time() readings can come out negative or huge. Use time.monotonic() (or time.perf_counter()), which only moves forward.
• Store and compare timestamps in UTC, and convert to local time only for display. Time zones and daylight-saving changes are a display concern; including them in stored data causes duplicated or missing hours.

import time
from datetime import datetime, timezone

start = time.monotonic()
total = sum(range(1_000_000))                     # stand-in for real work
elapsed = time.monotonic() - start                # safe even if NTP steps the clock
recorded_at = datetime.now(timezone.utc).isoformat()   # e.g. 2026-09-30T14:03:11.402913+00:00

Leap Seconds#

UTC occasionally inserted a leap second to stay aligned with the Earth's rotation, most recently at the end of 2016. Software that assumed every minute has 60 seconds has crashed or spun at 100% CPU during these events. Some large providers smear the extra second, slowing their clocks slightly over many hours instead of inserting it all at once. Never mix smeared and non-smeared sources in one client's configuration, because they disagree by up to half a second during the smear. International timekeeping bodies decided in 2022 to stop adding leap seconds by 2035, but existing systems must still handle them until then.

When NTP Is Not Enough: PTP#

The Precision Time Protocol (PTP, IEEE 1588) reaches sub-microsecond accuracy, where NTP typically manages milliseconds over the internet and tens of microseconds on a good LAN. The difference is hardware timestamping: the network card records the exact moment a packet leaves or arrives, removing the operating system's scheduling jitter. PTP networks use a grandmaster clock (often GPS-disciplined), and boundary or transparent clock switches that account for their own queuing delay. On Linux, ethtool -T eth0 shows whether a card supports hardware timestamps, and the linuxptp tools ptp4l and phc2sys run the protocol and sync the system clock to the card. PTP is used in financial trading, telecom networks, broadcast, and industrial control. It needs capable hardware end to end, so general-purpose fleets stay on NTP.

Time Is a Security Boundary#

An attacker who can shift a host's clock can make expired certificates look valid, revive expired tokens, break authentication, and scramble forensic timelines. Plain NTP is unauthenticated, so an on-path attacker can forge replies. Defences:

• Use NTS (Network Time Security) where your servers and clients support it, or keep time traffic on trusted internal paths.
• Use multiple independent sources so one lying server is outvoted.
• Allow steps only at boot, and alert on any large step afterwards. An unexpected clock jump deserves investigation.
• Do not expose NTP servers you run to the internet unless that is their purpose, and keep them patched: old NTP server features have been abused for traffic amplification in DDoS attacks.

The network security lesson places these controls in the broader defensive picture.

Practice#

Run the SNTP script against three different servers and compare offsets and delays. Then check chronyc sources -v (or w32tm /query /status) on your own machine: how many sources does it use, which one is selected, and would it survive one of them lying?