A node offline or NotReady

A node shows as offline / NotReady

What it means: Two different signals can make a node look unhealthy, and it helps to tell them apart:

• Offline — the node’s own management agent stopped, or lost its connection. Buddy marks a node offline after it misses a run of heartbeats, and flips it back to online automatically once it recovers.
• NotReady — the node is known to the cluster, but Kubernetes itself reports it not ready. This is often just a kubelet still starting up, or a transient network-overlay issue. The Nodes screen flags this kind of mismatch when it sees one.

Fix:

• For offline, check the node itself:

  systemctl status buddy-agent           # run on the worker
  sudo journalctl -u buddy-agent -n 50

  The agent reconnects on its own, with an increasing wait between attempts, once the master is reachable again. There’s nothing to trigger by hand.
• For NotReady, if the underlying cause is the node’s private network connection to the rest of the cluster, the node’s Overview screen and Cluster → Network now name that directly and offer a one-click fix. See A node’s overlay shows down. If it isn’t the network, check A node’s Kubernetes health shows down, with a reason next.

If that didn’t work: Export a diagnostics bundle from the panel (the Diagnostics button): it captures every node’s status and recent task logs, with no secrets in it, which is the fastest way to see the whole picture at once.

The Activity panel no longer shows routine health checks

What it means: This is expected behaviour, not a sign anything stopped working. The periodic checks behind a node’s network, Kubernetes and storage health, and its software inventory, run continuously in the background but no longer create an entry in Activity for every single check; only when something actually changes (a disk appears, a health verdict gets worse, a reboot becomes pending) does a line show up there. A node’s Overview screen shows an Observed state card instead, with how long ago each of those areas was last checked.

Fix: Nothing to fix: a quiet Activity feed on a healthy cluster is the intended result. If you want to confirm checks are still actually running, look at the Observed state timestamps on a node’s Overview: they should be recent. On-demand actions you trigger yourself (like Verify buttons) are unaffected by any of this; they always produce a normal, followable entry with live logs.

If that didn’t work: If an Observed state timestamp for a node is genuinely old and not advancing, that — not a quiet Activity feed — is the real sign something stopped: check that node’s agent is online (see “A node shows as offline / NotReady” above).

Tasks keep failing, then work when you retry them

What it means: Actions the panel runs on a node — an install, a health check, anything dispatched to the agent — retry automatically with an increasing wait between attempts. A flaky package mirror or a Kubernetes API that isn’t ready yet are common, self-resolving causes.

Fix: If an action does eventually fail outright, its logs (open its entry in Activity) show the actual error. Re-running it is always safe: every action is built to pick up cleanly from where it left off, not to duplicate what already happened.

If that didn’t work: If the same action keeps failing for the same reason on retry, the error text in its log is the specific thing to act on: the underlying cause (network, disk space, an unreachable service) rather than the retry itself.

Removal seems stuck

What it means: Removing a worker is a multi-step process — draining its workloads, having it leave the cluster, tearing down its network connection, and removing its entry — and it’s built to be safe to interrupt and resume: if a step fails partway through, it re-runs from where it left off rather than leaving the node half-removed.

Fix: Watch the live progress in the panel. If the worker’s own agent is offline, its on-node steps are skipped automatically and the rest of the cleanup — drain, leave the cluster, free its network address, revoke its certificate, remove its entry — still completes from the master’s side.

The master cannot be removed through this flow.

If that didn’t work: Export a diagnostics bundle and check the removal’s own log entry in Activity for the specific step it’s stuck on.