Skip to content
Buddy Cluster
buddy-cluster.dev

Kubernetes health & recovery

One layer above the network connection, a node can be fully connected to the cluster and still never become a genuinely working Kubernetes member, or it was one and silently dropped out. Buddy checks every node’s Kubernetes membership from the inside on a slow, continuous tick, and fuses that with Kubernetes’ own Ready/NotReady view of the node. The result shows on the node’s Overview screen, in the k3s health card next to its overlay health.

A node's Overview screen, showing the k3s health card with its verdict and Repair k3s action
A node's Overview screen, showing the k3s health card with its verdict and Repair k3s action
A node's Overview screen, showing the k3s health card with its verdict and Repair k3s action
  • healthy — the Kubernetes service (k3s on the master, k3s-agent on a worker) is active, its binary is present, and — on a worker — no rogue standalone server is running alongside it.
  • degraded — the check itself couldn’t fully run (for example, it couldn’t read the service’s own status). The last known-good verdict stays shown until the next check succeeds.
  • down, with the specific reason named rather than a bare “NotReady.”

A node reachable on port 6443 is never, by itself, treated as a passed health check: a node can answer there while still being completely broken at the Kubernetes layer. That reachability is only used to tell a network problem apart from a Kubernetes one, never as a pass on its own.

A node’s Kubernetes health shows down, with a reason

What it means: The k3s health card names one of a small set of specific causes:

  • Stale cluster CA — the worker’s secure tunnel to the master fails with a certificate error, typically after the master’s own identity changed (a fresh master install, say) while the worker kept its old one.
  • Rogue standalone server — a worker is running a full, independent Kubernetes server process alongside its normal worker agent (this happens by default if k3s was ever installed by hand on that machine, outside of Buddy). It holds the port the worker’s own agent needs and carries its own cluster identity, which is exactly what causes the stale-CA case above.
  • Agent enabled but inactive — the service is configured to run but isn’t actually running; this can happen silently when an update to it detects nothing changed and skips actually starting it.
  • Node password rejected — usually a duplicate machine name, or a leftover registration on the master from a previous attempt.

Fix: Open the node’s Overview screen and choose Repair k3s. What it does depends on the node’s role:

  • Worker — a clean rejoin: tear down any rogue standalone server, clean up any leftover processes, wipe stale local state, and rejoin using a fresh identity read live from the master, never a cached one, since a stale identity is exactly what caused the problem in the stale-CA case. The confirmation offers an extra checkbox — also clean up a stale node registration on the master — for the duplicate-name case; safe to leave unchecked otherwise.
  • Master — non-destructive only: make sure the service is enabled and restart it. The control plane and its database are never reinstalled by this action: rebuilding a genuinely broken master is out of scope for a one-click repair.

Either way, the action finishes with a trailing check that confirms the node is actually Ready before it reports success. This is the same guarded shape as an overlay repair: you confirm, watch the steps, and a failed recovery is surfaced rather than a false “done.”

If that didn’t work: Repair needs the node online; the button is disabled with a reason if it isn’t. See A node shows as offline / NotReady. If the repair runs but the node still isn’t confirmed Ready, its own service logs are the next thing to check on the node itself: sudo journalctl -u k3s-agent on a worker, or sudo journalctl -u k3s on the master.

A node’s Kubernetes health shows degraded

What it means: This specifically means the health check itself couldn’t fully read the node’s status, not that the node is necessarily unhealthy. The card keeps showing the last verdict it actually managed to confirm until the next check succeeds.

Fix: Usually nothing: the next check, a short while later, resolves it on its own. If it stays degraded for an extended period rather than clearing, treat it as you would offline: check the node’s own agent.

If that didn’t work: See A node shows as offline / NotReady for checking whether the node itself is reachable at all.