IPVS - no destination available


A kURL installation may report many errors like the following in dmesg:

[  500.00000 ] IPVS: rr: TCP - no destination available

These errors occur when a Kubernetes service does not have any pods to forward traffic to.
In this example is the ClusterIP of a Kubernetes service. Kube-proxy watches the Kubernetes API server for endpoints associated with this virtual service IP and forwards traffic to one of the Pod endpoints, which will be a different IP and may also be a different port.

Because kube-proxy uses IPVS, an in-kernel load-balancer, the errors are reported in the kernel ring buffer.

Although these errors are expected after deploying a new application or after reboot, they may be cause for concern if they persist for several minutes. To troubleshoot, determine the service associated with the IP:

kubectl get service -A | grep <ip>

This will show a service such as kostadm. Then you can run kubectl get endpoints kotsadm to see which pod IPs or destinations are available for the service. Usually there are no Pods ready for the service and so no endpoints. If the error messages continue after endpoints are available this may indicate a problem with kube-proxy.