Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rook-ceph-osd pod fails after reboot: Block already exists #14469

Open
electricstrm-redgate opened this issue Jul 17, 2024 · 7 comments
Open

rook-ceph-osd pod fails after reboot: Block already exists #14469

electricstrm-redgate opened this issue Jul 17, 2024 · 7 comments
Labels
bug

Comments

@electricstrm-redgate
Copy link

electricstrm-redgate commented Jul 17, 2024 •
edited
Loading

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

  1. OSD pod fails after a reboot.
  2. If I delete this folder before rebooting, it works again: /var/lib/rook

Expected behavior:
OSD pod is running.

How to reproduce it (minimal and precise):
I'm testing on a cluster with 1 node. Reboot the node. This seems to happen 100% on RHEL versions and sometimes on Ubuntu.

File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit:

  • Crashing pod(s) logs : osd pod -c chown-container-datda-dir
ownership of '/var/log/ceph/set1-data-029j9p/ceph-volume.log' retained as ceph:ceph
ownership of '/var/log/ceph/set1-data-029j9p' retained as ceph:ceph
ownership of '/var/log/ceph' retained as ceph:ceph
ownership of '/var/lib/ceph/crash' retained as ceph:ceph
ownership of '/run/ceph/ceph-client.ceph-exporter.asok' retained as ceph:ceph
ownership of '/run/ceph/ceph-mgr.a.asok' retained as ceph:ceph
ownership of '/run/ceph/ceph-osd.0.asok' retained as ceph:ceph
ownership of '/run/ceph/ceph-mon.b.asok' retained as ceph:ceph
ownership of '/run/ceph' retained as ceph:ceph
ownership of '/var/lib/ceph/osd/ceph-0/whoami' retained as ceph:ceph
ownership of '/var/lib/ceph/osd/ceph-0/keyring' retained as ceph:ceph
ownership of '/var/lib/ceph/osd/ceph-0/ceph_fsid' retained as ceph:ceph
ownership of '/var/lib/ceph/osd/ceph-0/fsid' retained as ceph:ceph
ownership of '/var/lib/ceph/osd/ceph-0/type' retained as ceph:ceph
ownership of '/var/lib/ceph/osd/ceph-0/ready' retained as ceph:ceph
ownership of '/var/lib/ceph/osd/ceph-0/require_osd_release' retained as ceph:ceph
changed ownership of '/var/lib/ceph/osd/ceph-0/block' from root:disk to ceph:ceph
ownership of '/var/lib/ceph/osd/ceph-0' retained as ceph:ceph
  • Crashing pod(s) logs : from osd pod -c expand-bluefs
inferring bluefs devices from bluestore path
1 : device size 0x1900000000 : using 0x1f30000(31 MiB)
Expanding DB/WAL...
  • Crashing pod(s) logs : from osd pod -c blkdevmapper
+ PVC_SOURCE=/set1-data-0dxftc
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat --format %t%T /set1-data-0dxftc
+ PVC_SOURCE_MAJ_MIN=1030
++ stat --format %t%T /var/lib/ceph/osd/ceph-0/block
+ PVC_DEST_MAJ_MIN=1030
+ [[ 1030 == \1\0\3\0 ]]
+ echo 'PVC /var/lib/ceph/osd/ceph-0/block already exists and has the same major and minor as /set1-data-0dxftc: 1030'
+ exit 0
PVC /var/lib/ceph/osd/ceph-0/block already exists and has the same major and minor as /set1-data-0dxftc: 1030
  • log from rook-ceph-mon
debug 2024-07-17T16:44:01.681+0000 7fde16786700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
debug 2024-07-17T16:44:01.681+0000 7fde16786700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
debug 2024-07-17T16:44:01.934+0000 7fde12d7c700  0 -- [v2:10.96.1.224:3300/0,v1:10.96.1.224:6789/0] send_to message mon_probe(probe 1631918c-3cce-43e8-ae85-e2bf5ae76c1a name b leader -1 new mon_release reef) v8 with empty dest 
debug 2024-07-17T16:44:03.938+0000 7fde12d7c700  0 -- [v2:10.96.1.224:3300/0,v1:10.96.1.224:6789/0] send_to message mon_probe(probe 1631918c-3cce-43e8-ae85-e2bf5ae76c1a name b leader -1 new mon_release reef) v8 with empty dest 
debug 2024-07-17T16:44:05.713+0000 7fde12d7c700 -1 mon.b@-1(probing) e0 get_health_metrics reporting 942 slow ops, oldest is log(1 entries from seq 1 at 2024-07-17T15:25:20.228439+0000)
debug 2024-07-17T16:44:05.942+0000 7fde12d7c700  0 -- [v2:10.96.1.224:3300/0,v1:10.96.1.224:6789/0] send_to message mon_probe(probe 1631918c-3cce-43e8-ae85-e2bf5ae76c1a name b leader -1 new mon_release reef) v8 with empty dest 
debug 2024-07-17T16:44:07.946+0000 7fde12d7c700  0 -- [v2:10.96.1.224:3300/0,v1:10.96.1.224:6789/0] send_to message mon_probe(probe 1631918c-3cce-43e8-ae85-e2bf5ae76c1a name b leader -1 new mon_release reef) v8 with empty dest 
debug 2024-07-17T16:44:09.950+0000 7fde12d7c700  0 -- [v2:10.96.1.224:3300/0,v1:10.96.1.224:6789/0] send_to message mon_probe(probe 1631918c-3cce-43e8-ae85-e2bf5ae76c1a name b leader -1 new mon_release reef) v8 with empty dest 
debug 2024-07-17T16:44:10.713+0000 7fde12d7c700 -1 mon.b@-1(probing) e0 get_health_metrics reporting 942 slow ops, oldest is log(1 entries from seq 1 at 2024-07-17T15:25:20.228439+0000)
debug 2024-07-17T16:44:11.679+0000 7fde16786700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
debug 2024-07-17T16:44:11.679+0000 7fde16786700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
debug 2024-07-17T16:44:11.954+0000 7fde12d7c700  0 -- [v2:10.96.1.224:3300/0,v1:10.96.1.224:6789/0] send_to message mon_probe(probe 1631918c-3cce-43e8-ae85-e2bf5ae76c1a name b leader -1 new mon_release reef) v8 with empty dest

Cluster Status to submit:
unable to read

Environment:

  • OS This happen much more frequently on RHEL. I used RHEL 9.2, but saw it on other versions of RHEL
    NAME="Red Hat Enterprise Linux"
    VERSION="9.2 (Plow)"
    ID="rhel"
    ID_LIKE="fedora"
    VERSION_ID="9.2"
    PLATFORM_ID="platform:el9"
    PRETTY_NAME="Red Hat Enterprise Linux 9.2 (Plow)"

  • Kernel : Linux 5.14.0-284.48.1.el9_2.x86_64 Monitor bootstrapping with libcephd #1 SMP PREEMPT_DYNAMIC Thu Jan 4 03:49:47 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

  • Cloud provider or hardware configuration: I test on AWS EC2, but customers that aren't on the cloud have the problem too.

  • Rook version (use rook version inside of a Rook Pod): 14.8

  • Storage backend version: quay.io/ceph/ceph:v18.2.2

  • Kubernetes version (use kubectl version): 1.29.5

  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubernetes installed on a VM with Kurl.

  • Storage backend status (e.g. for Ceph use ceph health: auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
@travisn
Copy link
Member

travisn commented Jul 17, 2024

  • You are using the cluster-on-pvc approach, right? What storage provider are you using for the PVCs?
  • Which container is failing? The activate container? The formatting of the logs wasn't clear (which hopefully is accurate now)

@electricstrm-redgate
Copy link
Author

  1. Do you mean the provisioner redgate-clone-app.rbd.csi.ceph.com, csi-rbdplugin
  2. The cluster ends up with these 2 pods failed rook-ceph-mgr and rook-ceph-osd
    rook-ceph-mgr -> Defaulted container "mgr" out of: mgr
    rook-ceph-osd -> Defaulted container "osd" out of: osd

@electricstrm-redgate
Copy link
Author

It seems like the first reboot is ok, but then if you wait 15 mins and reboot the problems happen. Deleting one folder and rebooting again fixes it.

@travisn
Copy link
Member

travisn commented Jul 22, 2024

  1. Do you mean the provisioner redgate-clone-app.rbd.csi.ceph.com, csi-rbdplugin

Are you setting volumeClaimTemplate.spec.storageClassName as seen in this example? I wonder if there is some issue with the provider that is backing the OSDs. This is different from the csi driver you mentioned.

  1. The cluster ends up with these 2 pods failed rook-ceph-mgr and rook-ceph-osd
    rook-ceph-mgr -> Defaulted container "mgr" out of: mgr
    rook-ceph-osd -> Defaulted container "osd" out of: osd

Where are you seeing the failure exactly? Are the pods in CLBO? What does kubectl -n rook-ceph describe pod show for them?

@electricstrm-redgate
Copy link
Author

  1. local-storage

  2. It varies, sometimes they CLBO, but then it's like it gives up and it shows 0/1.

mgr describe pod
Normal Scheduled 3m36s default-scheduler Successfully assigned redgate-clone-app/rook-ceph-mgr-a-7b9b86d497-c7cd7 to ip-172-31-1-191.eu-west-1.compute.internal
Normal Pulled 3m30s kubelet Container image "quay.io/ceph/ceph:v17.2.1-20220623" already present on machine
Normal Created 3m29s kubelet Created container chown-container-data-dir
Normal Started 3m28s kubelet Started container chown-container-data-dir
Normal Started 105s (x2 over 3m22s) kubelet Started container mgr
Warning Unhealthy 45s (x12 over 3m5s) kubelet Startup probe failed: ceph daemon health check failed with the following output:

admin_socket: exception getting command descriptions: [Errno 111] Connection refused
Normal Killing 45s (x2 over 2m15s) kubelet Container mgr failed startup probe, will be restarted
Normal Pulled 15s (x3 over 3m26s) kubelet Container image "quay.io/ceph/ceph:v17.2.1-20220623" already present on machine
Normal Created 15s (x3 over 3m22s) kubelet Created container mgr

@electricstrm-redgate
Copy link
Author

osd describe pod:
Normal Scheduled 3m2s default-scheduler Successfully assigned redgate-clone-app/rook-ceph-osd-0-5bffbc8c7c-48mk2 to ip-172-31-1-191.eu-west-1.compute.internal
Normal SuccessfulMountVolume 3m2s kubelet MapVolume.MapPodDevice succeeded for volume "local-pv-77ad3755" globalMapPath "/var/lib/kubelet/plugins/kubernetes.iolocal-volume/volumeDevices/local-pv-77ad3755"
Normal SuccessfulMountVolume 3m2s kubelet MapVolume.MapPodDevice succeeded for volume "local-pv-77ad3755" volumeMapPath "/var/lib/kubelet/pods/ad2cb853-b3e5-4e31-981d-f0accd51bccc/volumeDevices/kubernetes.iolocal-volume"
Normal Pulled 2m56s kubelet Container image "quay.io/ceph/ceph:v17.2.1-20220623" already present on machine
Normal Created 2m55s kubelet Created container blkdevmapper
Normal Started 2m54s kubelet Started container blkdevmapper
Normal Pulled 2m52s kubelet Container image "quay.io/ceph/ceph:v17.2.1-20220623" already present on machine
Normal Created 2m48s kubelet Created container activate
Normal Started 2m47s kubelet Started container activate
Normal Pulled 2m46s kubelet Container image "quay.io/ceph/ceph:v17.2.1-20220623" already present on machine
Normal Created 2m46s kubelet Created container expand-bluefs
Normal Started 2m46s kubelet Started container expand-bluefs
Normal Pulled 2m45s kubelet Container image "quay.io/ceph/ceph:v17.2.1-20220623" already present on machine
Normal Created 2m45s kubelet Created container chown-container-data-dir
Normal Started 2m45s kubelet Started container chown-container-data-dir
Normal Pulled 2m44s kubelet Container image "quay.io/ceph/ceph:v17.2.1-20220623" already present on machine
Normal Created 2m44s kubelet Created container osd
Normal Started 2m44s kubelet Started container osd
Warning Unhealthy 81s (x8 over 2m31s) kubelet Startup probe failed: ceph daemon health check failed with the following output:

@github-actions GitHub Actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added wontfix and removed wontfix labels Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@travisn @electricstrm-redgate