-
Notifications
You must be signed in to change notification settings - Fork 581
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
/var/lib/dockershim
and /var/lib/cri-dockerd
should not be ephemeral
#2993
Comments
Reproduced the issue on Rancher 2.6head commit id: On an upgrade of RKE1 node driver Linode cluster (1 etcd/cp, 2 worker nodes) from Other useases tested:
|
On 2.6.7-rc5 On a Linode node driver Node driver cluster - 1 etcd/cp and 3 worker nodes
|
/var/lib/dockershim
should not be ephemeral/var/lib/dockershim
and /var/lib/cri-dockerd
should not be ephemeral
@markusewalker I have added the testing template to the PR 3001 -- testing this should be very simple. This issue also ties into #2999 and testing that one is a little more nuanced. |
Verified that this is addressed on ENVIRONMENT DETAILS
TEST RESULT REPRODUCTION STEPS
VERIFICATION STEPS
@Oats87 as noted in step 6 of the verification steps, when I upgraded, UPDATE |
Reopening for more validations |
Can we also validate for these k8s versions ? on 2.6-head latest
|
In response to additional validation tests found herehttps://github.com//issues/2993#issuecomment-1208666694, please find results below. ENVIRONMENT DETAILS
TEST SCENARIO 1
TEST SCENARIO 2
|
RKE version:
RKE <= 1.3.12?
Docker version: (
docker version
,docker info
preferred)Doesn't matter
Operating system and kernel: (
cat /etc/os-release
,uname -r
preferred)Doesn't matter
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Doesn't matter
cluster.yml file:
Doesn't matter
Steps to Reproduce:
The
/var/lib/dockershim
(/var/lib/cri-dockerd
for >= 1.24.x) directory for thekubelet
container should not be ephemeral. The kubelet/dockershim relies on this directory to store runtime metadata such asportMappings
for management of pods.We ran into this issue when debugging a case where hostport pods were not being properly cleaned up on tear down (with duplicate iptables rules being found) after Kubernetes version upgrades.
This issue was specifically a combination of multiple problems, namely one where we updated the CNI plugin versions from
v0.8.6
tov1.0.0
which brought along a change (see: containernetworking/plugins#509) that changed to performing aNOOP
ifportMappings
were not specified. The issue we saw was that the kubelet/dockershim were sending an emptyportMappings
after a Kubernetes version upgrade, as the Kubelet would be replaced and the/var/lib/dockershim
folder was empty, and could not properly construct aportMappings
struct for teardown.A workaround for this issue is to specify an
extra_bind
for thekubelet
service, i.e.and rebooting the host after an
rke up
(if already dealing with duplicate iptables rules).This will not be done automatically for
v1.23
and older versions of Kubernetes, but will be done forv1.24
and newer versions of K8s.SURE-4702
The text was updated successfully, but these errors were encountered: