Upgrading Kubernetes

The upgrade is refused before it starts

What it means: The version you typed is checked before anything runs at all: Kubernetes moves one minor version at a time (from v1.31.x you can go to v1.32.x, not straight to v1.33.x), a downgrade isn’t allowed, and the version has to be well-formed (vX.Y.Z+k3sN, e.g. v1.32.4+k3s1).

Fix: Pick a target within the “next eligible” version the screen shows you: that’s the furthest you can go in one upgrade from where the cluster is now.

If that didn’t work: If the version you typed looks correct and it’s still refused, check for a typo against the exact form the screen expects, including the +k3sN suffix.

“An agent is ahead of the server”

What it means: The upgrade refuses to start if any worker is already on a newer Kubernetes version than the master, an unsupported gap Kubernetes itself can’t resolve on its own. This can also happen as a side effect of a halted upgrade that got the server behind where a worker already landed.

Fix: Bring the master up to at least the workers’ version first — either by upgrading the server to that version, or by rejoining the ahead worker at the server’s current version — then run the cluster upgrade again. The upgrade is built to always keep the master at least as new as every worker, never the other way round.

If that didn’t work: If you’re not sure which node is ahead, Cluster → Kubernetes shows every node’s current version side by side: compare them directly rather than guessing.

The panel blipped when the server upgraded

What it means: This is expected. The master is upgraded first, which restarts Kubernetes on it and briefly takes the Kubernetes API and the panel down together; workloads on your workers keep running the whole time.

Fix: Nothing to do: the roll continues without the panel, and the screen reconnects on its own and picks up the current phase, even across a full restart of the panel’s own service mid-roll. Don’t re-run the upgrade while waiting; just let the panel come back.

If that didn’t work: If the panel doesn’t reconnect after a reasonable wait, confirm the master itself is actually back up before assuming the upgrade is stuck.

A node is stuck at the old version / the roll halted

What it means: If a node doesn’t come back Ready at the target version within the roll’s timeout, the whole roll halts on purpose: that node is left cordoned, every node after it in the roll is left untouched, and the action is marked failed with a hint pointing at the cause.

Fix: Check the stuck node directly: its own log (journalctl -u k3s-agent on a worker, -u k3s on the master), its disk space, and whether it could actually reach the network to fetch the target version (the installer needs it). A failed fetch doesn’t damage the node’s current, running install. Fix the cause, then re-run the upgrade; it skips nodes already at the target and resumes the roll from where it stopped.

If that didn’t work: If the node still won’t reach the target version after a clean retry, treat it as A node’s Kubernetes health shows down, with a reason first, then re-run the upgrade.

The drain is stuck mid-roll

What it means: As with node maintenance, a policy limiting how many copies of an app can be down at once, or a pod pinned to node-local storage on that specific worker, can block its drain during the roll.

Fix: The drain has its own timeout, so it fails cleanly — that worker cordoned, the roll halted — rather than hanging. Reschedule or scale down the blocking workload, then re-run the upgrade.

If that didn’t work: See The drain is stuck for the same guidance in more depth, since the cause and the fix are identical, just mid-roll instead of a single node’s own maintenance.

The server upgrade itself failed (recovering with the recovery snapshot)

What it means: Before touching the master at all, Buddy takes a recovery snapshot and shows its name on screen: your anchor if the master’s own upgrade genuinely fails.

Fix: Restoring that snapshot is destructive and rarely actually needed, so it’s deliberately not a guided, one-click action; you do it by hand, directly on the master:

sudo systemctl stop k3s
sudo k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<snapshot-name>

Then start Kubernetes again (sudo systemctl start k3s). Snapshots live under /var/lib/rancher/k3s/server/db/snapshots on the master.

This restores the cluster’s control-plane database to the point of the snapshot. It is a destructive recovery step, not a routine one. Only use it if the master’s upgrade has genuinely failed and it won’t come back on its own.

If that didn’t work: Before reaching for the snapshot, first rule out a simpler cause — network reachability, disk space — the same way as “A node is stuck at the old version,” above; the snapshot restore is a last resort, not the first thing to try.

“A cluster upgrade is already in progress”

What it means: Only one cluster-wide or per-node operation runs at a time. This upgrade won’t start while a per-node maintenance action is already in flight, and vice versa: the same message appears from either direction.

Fix: Finish or cancel whichever operation is already running, then retry the one you actually wanted.

If that didn’t work: If nothing looks like it’s actually running anywhere in the panel but you still see this message, check Cluster → Kubernetes and each node’s Maintenance screen directly for anything still mid-action before assuming the message itself is wrong.