Updating buddy-agent

A worker is stuck on the old agent version

What it means: Two situations look similar on screen but mean very different things, and it’s worth telling them apart before doing anything:

Badged deferred, with a muted (not red) reason like “agent offline since … — not attempted.” Nothing was ever pushed to this worker; its agent simply wasn’t reachable when the update was planned. It’s still on its old, working version, and there is genuinely nothing to fix. Power the node on; its agent reconnecting triggers a fresh roll for it automatically, with nothing to click. The overall roll still reports done: a summary line like “1 worker updated, 1 deferred (bc-n2)” means the roll wasn’t complete for every node, not that anything failed.
The roll itself shows failed, halted on that node, with a red warning on its row, not the muted deferred note. The update was sent to it and didn’t re-register at the target version within the timeout, so Buddy stopped and left that node on its old, working agent; every other node is untouched. Check the node itself: journalctl -u buddy-agent on the worker, whether it could reach the master, and disk space under /usr/local/bin. Fix the cause, then re-run the update from the screen: nodes already at the target are skipped automatically.
Badged needs-rebootstrap, with a reason saying its update task completed and reported no change. This happens on an agent old enough that it can’t tell “rebuilt with the same version string” apart from “already up to date”: a genuine skew this old, but no longer a live one, since current agents compare the actual binary, not just the version text. Buddy still sends the update to it (this case can’t be told apart ahead of time), and the moment it reports back unchanged, the node is marked needs-rebootstrap immediately rather than waiting out the whole timeout, and the roll carries on to the rest of the fleet. Three ways to move it forward, in order of convenience: re-add it over SSH (the same one-time re-bootstrap as an agent older than v0.0.8, below); on a development cluster, redeploy directly to that node’s address; or simply wait for the next real version release, which this node’s short-circuit does correctly recognise and act on.

Fix: See each case above. They need different responses, not a single generic retry.

If that didn’t work: Export a diagnostics bundle from the panel and check the specific node’s own log for the actual error, rather than re-running the whole roll repeatedly against the same failure.

“SHA-256 mismatch” in the logs

What it means: The bytes a worker downloaded didn’t match the checksum the master published for them. The agent refuses the swap and keeps its currently running binary. Nothing changes on that node when this happens.

Fix: This usually means the file on the master changed mid-roll, or the read itself was corrupted. Re-run the update: the master re-reads and re-publishes the binary and its checksum fresh each time.

If that didn’t work: If the mismatch keeps happening on retry, check available disk space on the master under wherever it serves the binary from, and on the worker under /usr/local/bin.

A node needs an SSH re-bootstrap

What it means: The node’s agent predates the version that can update itself over its normal connection to the master; it doesn’t understand the self-update process at all. This is expected for any node added before that point.

Fix: Click Re-add and supply its SSH credentials. This one-time re-bootstrap brings it onto the normal self-update path; every update after this one works over the ordinary connection like any other node.

If that didn’t work: If SSH itself can’t reach the node for the re-bootstrap, that’s a network or credentials problem, not an agent-version one. Confirm you can SSH to it manually first.

The roll is halted or “pinned” and won’t proceed

What it means: Two different reasons the roll isn’t moving:

Pinned — someone paused it on purpose, using Pin / pause, usually to hold the fleet still during an incident.
Failed — it halted on a node that didn’t come back at the target version within the timeout.

Fix: For a pinned roll, choose Resume (unpin) to continue. For a failed roll, fix whatever’s wrong with the node it halted on, then re-run. Either way the fleet is never left with some nodes part-updated; each worker is always fully on the old agent or fully on the new one, never in between.

If that didn’t work: If resuming a pinned roll doesn’t actually continue it, check whether it’s also failed on a node underneath the pin: unpinning only removes the pause, it doesn’t fix an underlying failure.

“The master’s agent is not at the target version”

What it means: The roll always verifies the master first and never pushes an update to it directly. The master’s own copy of the management software is upgraded by re-running the installer on it, not by this roll. If the master reports behind the target, that installer run didn’t fully complete.

Fix: Re-run the installer on the master (see Install the master) so both its management software and the rest of what it installs are at the same version, then start the fleet update again.

If that didn’t work: If the installer reports success but the master still shows behind afterwards, confirm you actually downloaded the version you meant to install; a stale download is a common cause.

A row is badged “same version, different build.”

What it means: The version text matches the target, but the node’s actual binary digest doesn’t. Buddy checks both, not just the version string, so a node that was rebuilt or manually patched without bumping its version number is no longer invisible to the roll.

Fix: Nothing urgent: this isn’t a warning about data loss or a broken node, just a note that two builds differ in bytes. Run the update as normal; it brings this node to the exact same bytes as everything else.

If that didn’t work: If this badge won’t clear after a normal update run, check whether something outside the panel (a manual binary swap on that node) is putting a different build back after each update.

“An agent self-update is already in progress”

What it means: Only one cluster-wide or per-node operation runs at a time. This roll won’t start while a Kubernetes upgrade or a per-node maintenance action is already in flight, and either of those refuses to start while this roll is running, in the same way.

Fix: Finish or cancel whichever operation is already running, then retry the one you actually wanted.

If that didn’t work: Check Cluster → Agent, Cluster → Kubernetes, and each node’s Maintenance screen directly for anything still mid-action if nothing looks like it’s running but you still see this message.

An Activity row for an agent update looks stuck or frozen

What it means: Two related, now-fixed defects (both 2026-07-22) could make this screen misleading:

An individual update entry could sit running for hours, or read failed / “orphaned: buddy-server restarted before this task completed” on a node that had actually updated fine, because its own process is replaced by the update before it can report a result, so the entry used to stay open forever and then get mislabeled by a later restart’s cleanup.
The whole Activity list could stop updating at once, not just one entry. A plain-sentence result written for a silently-closed entry broke the response the panel uses to refresh the list, so it kept showing its last good snapshot even though the update behind it had actually finished. Reloading the page didn’t help, since the same broken request just failed again.

Fix: Both are fixed. The roll now closes an update’s entry itself: succeeded, naming the target version and digest, once the node re-registers and stays healthy, or failed with the real reason if it genuinely never comes back.

If that didn’t work: If you’re seeing either symptom on a build from before that date, trust the node’s own reported version and digest in the Nodes list over the stale entry. If the whole list ever looks frozen again — every row aging while nodes are plainly live — that mismatch between what the list shows and what the fleet is actually doing is the tell, not any single row.