bind rke kubelet docker config path to default docker config path #3002
Conversation
rke stores auth info in /var/lib/kubelet/config.json but cri-dockerd relies on k8.io credential provider which uses only default config provider, this allows cri-dockerd to pull sandbox pause image using private registry
| if parsedRangeAtLeast124(parsedVersion) { | ||
| CommandArgs["container-runtime-endpoint"] = "unix:///var/run/cri-dockerd.sock" | ||
| Binds = []string{fmt.Sprintf("%s:/var/lib/cri-dockerd:z", path.Join(host.PrefixPath, "/var/lib/cri-dockerd"))} | ||
| Binds = []string{fmt.Sprintf("%s:/var/lib/cri-dockerd:z", path.Join(host.PrefixPath, "/var/lib/cri-dockerd")), | ||
| fmt.Sprintf("%s:%s", path.Join(host.PrefixPath, KubeletDockerConfigPath), "/.docker/config.json")} |
There was a problem hiding this comment.
hi @kinarashah , Although I approved the PR, I just had this question and want to get confirmation from you:
The binds are added when the cluster version is at least 1.24, so they are not applied when cri-dockerd is enabled on cluster <= 1.23, right? Why do we do it this way, is it because the problem does not exist for the older version of cri-dockerd used on cluster <= 1.23?
There was a problem hiding this comment.
nope you're right, this should be outside the v1.24 block but under IsCRIDockerdEnabled check..
There was a problem hiding this comment.
@jiaqiluo On second thought, I’m going to hold off on not adding change for 1.23 or older clusters…we’ll have to hardcode checks for latest patch versions here in order to avoid cluster auto upgrades due to change in kubelet process.
We do have a workaround if users want to enable cri-dockerd on <=1.23 clusters and use private registry, so I think we should be good there.
The workaround is to add the same bind mount in cluster.yml under services/kubelet/extra_binds.
There was a problem hiding this comment.
So we can merge this PR if the change looks good for 1.24 and if needed, will put up a new PR for other k8s versions.
Add bind
/var/lib/kubelet/config.json:/.docker/config.jsonIssue:
Pod creation failing with following error when using private registry:
E0801 21:49:55.450436 6271 kuberuntime_manager.go:815] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed pulling image \"ec2-3-144-97-102.us-east-2.compute.amazonaws.com/rancher/mirrored-pause:3.6\": Error response from daemon: Get https://ec2-3-144-97-102.us-east-2.compute.amazonaws.com/v2/rancher/mirrored-pause/manifests/3.6: no basic auth credentials" pod="kube-system/rke-network-plugin-deploy-job-92w6t"rancher/rancher#38473
Problem
RKE stores docker config at
/var/lib/kubelet/config.json(https://github.com/rancher/rke-tools/blob/master/entrypoint.sh#L75) , but the external cri-dockerd relies on k8s credential plugin which reads only from default docker config locations (https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/config.go#L136). No auth information is currently found so cri-dockerd defaults to pulling sandbox image (pause image) without credentials and provisioning breaks when using private registry (https://github.com/Mirantis/cri-dockerd/blob/master/core/sandbox_helpers.go#L422)Solution
The solution is to mount rke's default config.json to /.docker/config.json path where cri-dockerd expects it.
Testing
kinarashah/rke-tools:testwhich is updated to pull hardcoded private imagekinarashah/pause:3.6https://github.com/kinarashah/rke-tools/blob/pro/entrypoint.sh#L85