Replies: 2 comments 4 replies
-
|
I have noticed log entries like these seem to happen a lot of the time networking stops: |
Beta Was this translation helpful? Give feedback.
-
|
those are from kine and have nothing to do with networking |
Beta Was this translation helpful? Give feedback.
-
|
Agreed. But hear me out :) What if the fact it seems to happen at the same time suggests some kind of kadence which the compaction has as the thing causing the problem? What if the process handling the networking is on the same core as the compaction and the compaction is set to take 100% uninterrupted CPU and kernel.sched_rt_runtime_us = -1? (The latter is true on this unit). Just coming up with wild ideas because this has me really stumped. |
Beta Was this translation helpful? Give feedback.
-
|
Datastore compact happens every 5 minutes. It is absolutely possible that there is something else on your node that also happens every 5 minutes, but that doesn't mean they're related. |
Beta Was this translation helpful? Give feedback.
-
|
Are you leaving docker and other docker containers running on this node, after switching to containerd? We generally recommend using only a single container runtime on a node, as both docker and containerd will try to configure iptables to route traffic to their containers. If you haven't yet, make sure that docker is disabled, then reboot the node, and see if things stabilize. |
Beta Was this translation helpful? Give feedback.
-
|
Great question. I have tried both making sure that docker and containerd are not even installed on the unit as well as leaving them there. running. Same problem in both configurations. I have tried several different physical units with different expansion cards - same again. |
Beta Was this translation helpful? Give feedback.
-
I need to move to containerd instead of docker. Currently k3s is started with:
ExecStart=/usr/local/bin/k3s server \ --write-kubeconfig-mode 640 \ --disable traefik --disable metrics-server --disable-cloud-controller \ --docker \ --cluster-cidr 10.242.0.0/16 --service-cidr 10.243.0.0/16 --cluster-dns 10.243.0.10 \ --kube-controller-arg=terminated-pod-gc-threshold=100 \ --kubelet-arg=image-gc-high-threshold=100 \ --kubelet-arg=eviction-hard= \ --kubelet-arg=cpu-cfs-quota=true \ --kubelet-arg=volume-stats-agg-period=15m \ --kubelet-arg=housekeeping-interval=30s \ --node-name singlenodecluster \ --flannel-iface tap0 \ --kube-proxy-arg "proxy-mode=ipvs"I am using 1.29.3. I have tried 1.29.5 and 1.30.x.
Since removing the
--docker \line I intermittently get networking issues - I have only seen this between pods. I have had connection refused and even dns failures. Looking at the logs it's like networking stops working for 1 to 15 seconds. I get pod log message like:Connection refused: AH00957: HTTP: attempt to connect to 10.243.88.156:8888 (*) failedThen it recovers after say 15 seconds of retries.In general everything else is fine. No obvious changes in CPU, memory. Pods are healthy.
Things of note to explain why I have the above options:
--*-cidr: to avoid overlap with internal networks
--flannel-iface tap0: The idea of using a tap interface is to allow flannel, and thus k3s, to work if eth0/net0 is down for whatever reason and when there is no default gateway.
--kube-proxy-arg "proxy-mode=ipvs": an IP alias is added for each service to the kube-ipvs0 interface which means routing to services works even when there's no default gateway - as needed for true airgapped installations.
Any ideas?
Beta Was this translation helpful? Give feedback.
All reactions