Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1 mgr handle_mgr_map respawning because set of enabled modules changed! #12520

Open
yehaifeng opened this issue Jul 17, 2023 · 7 comments
Open
Labels
bug ceph-mgr Relating to the Ceph mgr or mgr modules

Comments

@yehaifeng
Copy link

yehaifeng commented Jul 17, 2023

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

Expected behavior:

How to reproduce it (minimal and precise):

File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit:
logs for mgr.b

debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr handle_mgr_map respawning because set of enabled modules changed!
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  e: 'ceph-mgr'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  0: 'ceph-mgr'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  1: '--fsid=8b101e47-b346-4bb8-9a42-2c051ec2005e'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  2: '--keyring=/etc/ceph/keyring-store/keyring'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  3: '--default-log-to-stderr=true'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  4: '--default-err-to-stderr=true'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  5: '--default-mon-cluster-log-to-stderr=true'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  6: '--default-log-stderr-prefix=debug '
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  7: '--default-log-to-file=false'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  8: '--default-mon-cluster-log-to-file=false'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  9: '--mon-host=[v2:10.146.189.167:3300,v1:10.146.189.167:6789],[v2:10.153.190.207:3300,v1:10.153.190.207:6789],[v2:10.147.234.226:3300,v1:10.147.234.226:6789]'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  10: '--mon-initial-members=c,a,b'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  11: '--id=b'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  12: '--setuser=ceph'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  13: '--setgroup=ceph'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  14: '--client-mount-uid=0'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  15: '--client-mount-gid=0'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  16: '--foreground'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  17: '--public-addr=10.244.8.48'
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn respawning with exe /usr/bin/ceph-mgr
debug 2023-07-17T09:50:32.844+0000 7faf22b88700  1 mgr respawn  exe_path /proc/self/exe
ignoring --setuser ceph since I am not root
ignoring --setgroup ceph since I am not root
debug 2023-07-17T09:50:32.949+0000 7fcf74dd41c0  0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable), process ceph-mgr, pid 13
debug 2023-07-17T09:50:32.949+0000 7fcf74dd41c0  0 pidfile_write: ignore empty --pid-file
debug 2023-07-17T09:50:32.993+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'alerts'
debug 2023-07-17T09:50:33.228+0000 7fcf74dd41c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:33.228+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'balancer'
debug 2023-07-17T09:50:33.426+0000 7fcf74dd41c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:33.426+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'cephadm'
debug 2023-07-17T09:50:34.896+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'crash'
debug 2023-07-17T09:50:35.102+0000 7fcf74dd41c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:35.103+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'dashboard'
debug 2023-07-17T09:50:36.347+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'devicehealth'
debug 2023-07-17T09:50:36.538+0000 7fcf74dd41c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:36.538+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'diskprediction_local'
debug 2023-07-17T09:50:37.135+0000 7fcf74dd41c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:37.135+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'influx'
debug 2023-07-17T09:50:37.332+0000 7fcf74dd41c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:37.332+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'insights'
debug 2023-07-17T09:50:37.521+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'iostat'
debug 2023-07-17T09:50:37.833+0000 7fcf74dd41c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:37.833+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'k8sevents'
debug 2023-07-17T09:50:38.758+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'localpool'
debug 2023-07-17T09:50:38.944+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'mds_autoscaler'
debug 2023-07-17T09:50:39.323+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'mirroring'
debug 2023-07-17T09:50:39.548+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'nfs'
debug 2023-07-17T09:50:40.170+0000 7fcf74dd41c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:40.170+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'orchestrator'
debug 2023-07-17T09:50:40.558+0000 7fcf74dd41c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:40.558+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'osd_perf_query'
debug 2023-07-17T09:50:40.766+0000 7fcf74dd41c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:40.766+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'osd_support'
debug 2023-07-17T09:50:40.960+0000 7fcf74dd41c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:40.960+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'pg_autoscaler'
debug 2023-07-17T09:50:41.177+0000 7fcf74dd41c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:41.177+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'progress'
debug 2023-07-17T09:50:41.374+0000 7fcf74dd41c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:41.374+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'prometheus'
debug 2023-07-17T09:50:42.361+0000 7fcf74dd41c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:42.361+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'rbd_support'
debug 2023-07-17T09:50:42.617+0000 7fcf74dd41c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:42.617+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'restful'
debug 2023-07-17T09:50:43.072+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'rook'
debug 2023-07-17T09:50:44.688+0000 7fcf74dd41c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:44.688+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'selftest'
debug 2023-07-17T09:50:44.975+0000 7fcf74dd41c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:44.975+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'snap_schedule'
debug 2023-07-17T09:50:45.192+0000 7fcf74dd41c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:45.192+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'stats'
debug 2023-07-17T09:50:45.396+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'status'
debug 2023-07-17T09:50:45.611+0000 7fcf74dd41c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:45.611+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'telegraf'
debug 2023-07-17T09:50:45.816+0000 7fcf74dd41c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:45.816+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'telemetry'
debug 2023-07-17T09:50:46.159+0000 7fcf74dd41c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:46.159+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'test_orchestrator'
debug 2023-07-17T09:50:46.977+0000 7fcf74dd41c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:46.977+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'volumes'
debug 2023-07-17T09:50:47.470+0000 7fcf74dd41c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:47.470+0000 7fcf74dd41c0  1 mgr[py] Loading python module 'zabbix'
debug 2023-07-17T09:50:47.677+0000 7fcf74dd41c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:47.697+0000 7fcf6a2e4700  0 ms_deliver_dispatch: unhandled message 0x55e613b04c60 mon_map magic: 0 v1 from mon.0 v2:10.153.190.207:3300/0
debug 2023-07-17T09:50:47.698+0000 7fcf6aae5700  0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2023-07-17T09:50:47.698+0000 7fcf6aae5700  1 mgr load Constructed class from module: prometheus
debug 2023-07-17T09:50:47.698+0000 7fcf2d377700  0 [prometheus INFO root] server_addr: :: server_port: 9283
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr handle_mgr_map respawning because set of enabled modules changed!
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  e: 'ceph-mgr'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  0: 'ceph-mgr'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  1: '--fsid=8b101e47-b346-4bb8-9a42-2c051ec2005e'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  2: '--keyring=/etc/ceph/keyring-store/keyring'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  3: '--default-log-to-stderr=true'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  4: '--default-err-to-stderr=true'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  5: '--default-mon-cluster-log-to-stderr=true'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  6: '--default-log-stderr-prefix=debug '
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  7: '--default-log-to-file=false'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  8: '--default-mon-cluster-log-to-file=false'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  9: '--mon-host=[v2:10.146.189.167:3300,v1:10.146.189.167:6789],[v2:10.153.190.207:3300,v1:10.153.190.207:6789],[v2:10.147.234.226:3300,v1:10.147.234.226:6789]'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  10: '--mon-initial-members=c,a,b'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  11: '--id=b'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  12: '--setuser=ceph'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  13: '--setgroup=ceph'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  14: '--client-mount-uid=0'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  15: '--client-mount-gid=0'
debug 2023-07-17T09:50:47.699+0000 7fcf6a2e4700  1 mgr respawn  16: '--foreground'ignoring --setuser ceph since I am not root
ignoring --setgroup ceph since I am not root
debug 2023-07-17T09:50:47.794+0000 7f472a4811c0  0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable), process ceph-mgr, pid 13
debug 2023-07-17T09:50:47.794+0000 7f472a4811c0  0 pidfile_write: ignore empty --pid-file
debug 2023-07-17T09:50:47.839+0000 7f472a4811c0  1 mgr[py] Loading python module 'alerts'
debug 2023-07-17T09:50:48.068+0000 7f472a4811c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:48.068+0000 7f472a4811c0  1 mgr[py] Loading python module 'balancer'
debug 2023-07-17T09:50:48.265+0000 7f472a4811c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:48.265+0000 7f472a4811c0  1 mgr[py] Loading python module 'cephadm'
debug 2023-07-17T09:50:49.504+0000 7f472a4811c0  1 mgr[py] Loading python module 'crash'
debug 2023-07-17T09:50:49.712+0000 7f472a4811c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:49.712+0000 7f472a4811c0  1 mgr[py] Loading python module 'dashboard'
debug 2023-07-17T09:50:50.908+0000 7f472a4811c0  1 mgr[py] Loading python module 'devicehealth'
debug 2023-07-17T09:50:51.098+0000 7f472a4811c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:51.098+0000 7f472a4811c0  1 mgr[py] Loading python module 'diskprediction_local'
debug 2023-07-17T09:50:51.722+0000 7f472a4811c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:51.722+0000 7f472a4811c0  1 mgr[py] Loading python module 'influx'
debug 2023-07-17T09:50:51.920+0000 7f472a4811c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:51.920+0000 7f472a4811c0  1 mgr[py] Loading python module 'insights'
debug 2023-07-17T09:50:52.109+0000 7f472a4811c0  1 mgr[py] Loading python module 'iostat'
debug 2023-07-17T09:50:52.424+0000 7f472a4811c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:52.424+0000 7f472a4811c0  1 mgr[py] Loading python module 'k8sevents'
debug 2023-07-17T09:50:53.358+0000 7f472a4811c0  1 mgr[py] Loading python module 'localpool'
debug 2023-07-17T09:50:53.551+0000 7f472a4811c0  1 mgr[py] Loading python module 'mds_autoscaler'
debug 2023-07-17T09:50:53.941+0000 7f472a4811c0  1 mgr[py] Loading python module 'mirroring'
debug 2023-07-17T09:50:54.165+0000 7f472a4811c0  1 mgr[py] Loading python module 'nfs'
debug 2023-07-17T09:50:55.024+0000 7f472a4811c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:55.024+0000 7f472a4811c0  1 mgr[py] Loading python module 'orchestrator'
debug 2023-07-17T09:50:55.405+0000 7f472a4811c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:55.405+0000 7f472a4811c0  1 mgr[py] Loading python module 'osd_perf_query'
debug 2023-07-17T09:50:55.607+0000 7f472a4811c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:55.607+0000 7f472a4811c0  1 mgr[py] Loading python module 'osd_support'
debug 2023-07-17T09:50:55.801+0000 7f472a4811c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:55.801+0000 7f472a4811c0  1 mgr[py] Loading python module 'pg_autoscaler'
debug 2023-07-17T09:50:56.017+0000 7f472a4811c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:56.017+0000 7f472a4811c0  1 mgr[py] Loading python module 'progress'
debug 2023-07-17T09:50:56.236+0000 7f472a4811c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:56.236+0000 7f472a4811c0  1 mgr[py] Loading python module 'prometheus'
debug 2023-07-17T09:50:57.156+0000 7f472a4811c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:57.156+0000 7f472a4811c0  1 mgr[py] Loading python module 'rbd_support'
debug 2023-07-17T09:50:57.414+0000 7f472a4811c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:57.414+0000 7f472a4811c0  1 mgr[py] Loading python module 'restful'
debug 2023-07-17T09:50:57.877+0000 7f472a4811c0  1 mgr[py] Loading python module 'rook'
debug 2023-07-17T09:50:59.405+0000 7f472a4811c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:59.405+0000 7f472a4811c0  1 mgr[py] Loading python module 'selftest'
debug 2023-07-17T09:50:59.631+0000 7f472a4811c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:59.631+0000 7f472a4811c0  1 mgr[py] Loading python module 'snap_schedule'
debug 2023-07-17T09:50:59.847+0000 7f472a4811c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member
debug 2023-07-17T09:50:59.847+0000 7f472a4811c0  1 mgr[py] Loading python module 'stats'
debug 2023-07-17T09:51:00.058+0000 7f472a4811c0  1 mgr[py] Loading python module 'status'
debug 2023-07-17T09:51:00.274+0000 7f472a4811c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member
debug 2023-07-17T09:51:00.274+0000 7f472a4811c0  1 mgr[py] Loading python module 'telegraf'
debug 2023-07-17T09:51:00.475+0000 7f472a4811c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member
debug 2023-07-17T09:51:00.475+0000 7f472a4811c0  1 mgr[py] Loading python module 'telemetry'
debug 2023-07-17T09:51:00.818+0000 7f472a4811c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member
debug 2023-07-17T09:51:00.818+0000 7f472a4811c0  1 mgr[py] Loading python module 'test_orchestrator'
debug 2023-07-17T09:51:01.778+0000 7f472a4811c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member
debug 2023-07-17T09:51:01.778+0000 7f472a4811c0  1 mgr[py] Loading python module 'volumes'
debug 2023-07-17T09:51:02.282+0000 7f472a4811c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member
debug 2023-07-17T09:51:02.282+0000 7f472a4811c0  1 mgr[py] Loading python module 'zabbix'
debug 2023-07-17T09:51:02.486+0000 7f472a4811c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member
debug 2023-07-17T09:51:02.503+0000 7f4720192700  0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2023-07-17T09:51:02.503+0000 7f4720192700  1 mgr load Constructed class from module: dashboard
debug 2023-07-17T09:51:02.505+0000 7f471f991700  0 ms_deliver_dispatch: unhandled message 0x557eb4b44c60 mon_map magic: 0 v1 from mon.0 v2:10.153.190.207:3300/0
debug 2023-07-17T09:51:02.506+0000 7f4720192700  0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2023-07-17T09:51:02.506+0000 7f4720192700  1 mgr load Constructed class from module: prometheus
debug 2023-07-17T09:51:02.507+0000 7f46baa45700  0 [dashboard INFO root] server: ssl=yes host=:: port=8443
debug 2023-07-17T09:51:02.508+0000 7f46c3246700  0 [prometheus INFO root] server_addr: :: server_port: 9283
debug 2023-07-17T09:51:02.510+0000 7f46c3246700  0 [prometheus INFO root] Starting engine...
[17/Jul/2023:09:51:02] ENGINE Bus STARTING
debug 2023-07-17T09:51:02.511+0000 7f46c3246700  0 [prometheus INFO cherrypy.error] [17/Jul/2023:09:51:02] ENGINE Bus STARTING
debug 2023-07-17T09:51:02.562+0000 7f46baa45700  0 [dashboard INFO root] Configured CherryPy, starting engine...
debug 2023-07-17T09:51:02.562+0000 7f46baa45700  0 [dashboard INFO root] Starting engine...
CherryPy Checker:
The Application mounted at '' has an empty config.

[17/Jul/2023:09:51:02] ENGINE Serving on http://:::9283
debug 2023-07-17T09:51:02.624+0000 7f46c3246700  0 [prometheus INFO cherrypy.error] [17/Jul/2023:09:51:02] ENGINE Serving on http://:::9283
[17/Jul/2023:09:51:02] ENGINE Bus STARTED
debug 2023-07-17T09:51:02.624+0000 7f46c3246700  0 [prometheus INFO cherrypy.error] [17/Jul/2023:09:51:02] ENGINE Bus STARTED
debug 2023-07-17T09:51:02.624+0000 7f46c3246700  0 [prometheus INFO root] Engine started.
debug 2023-07-17T09:51:02.685+0000 7f46baa45700  0 [dashboard INFO root] Engine started...
  • Operator's logs, if necessary

  • Crashing pod(s) logs, if necessary

INFO:ceph-crash:pinging cluster to exercise our key
2023-07-17T09:52:46.922+0000 7fe6c2ffd700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-07-17T09:52:46.923+0000 7fe6c27fc700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-07-17T09:52:46.924+0000 7fe6c37fe700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
[errno 13] RADOS permission denied (error connecting to the cluster)
INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s

To get logs, use kubectl -n <namespace> logs <pod name>
When pasting logs, always surround them with backticks or use the insert code button from the Github UI.
Read GitHub documentation if you need help.

Cluster Status to submit:

NAME        DATADIRHOSTPATH   MONCOUNT   AGE   PHASE   MESSAGE                        HEALTH        EXTERNAL   FSID
rook-ceph   /var/lib/rook     3          27d   Ready   Cluster created successfully   HEALTH_WARN              8b101e47-b346-4bb8-9a42-2c051ec2005e
  • Output of krew commands, if necessary
  cluster:
    id:     8b101e47-b346-4bb8-9a42-2c051ec2005e
    health: HEALTH_WARN
            1087 daemons have recently crashed

  services:
    mon: 3 daemons, quorum a,b,c (age 4h)
    mgr: a(active, since 12m), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 6 osds: 5 up (since 4h), 5 in (since 6h)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   12 pools, 185 pgs
    objects: 76.25k objects, 294 GiB
    usage:   883 GiB used, 4.7 TiB / 5.6 TiB avail
    pgs:     185 active+clean

  io:
    client:   938 B/s rd, 21 KiB/s wr, 1 op/s rd, 2 op/s wr

To get the health of the cluster, use kubectl rook-ceph health
To get the status of the cluster, use kubectl rook-ceph ceph status
For more details, see the Rook Krew Plugin

Environment:

  • OS (e.g. from /etc/os-release):
    NAME="CentOS Stream"
    VERSION="8"
    ID="centos"
    ID_LIKE="rhel fedora"
    VERSION_ID="8"
    PLATFORM_ID="platform:el8"
    PRETTY_NAME="CentOS Stream 8"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:centos:centos:8"
    HOME_URL="https://centos.org/"
    BUG_REPORT_URL="https://bugzilla.redhat.com/"
    REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
    REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

  • Kernel (e.g. uname -a):
    Linux hci001.yhf.com 4.18.0-338.el8.x86_64 Monitor bootstrapping with libcephd #1 SMP Fri Aug 27 17:32:14 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

  • Cloud provider or hardware configuration:

  • Rook version (use rook version inside of a Rook Pod): v1.11.9

  • Storage backend version (e.g. for ceph do ceph -v): ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)

  • Kubernetes version (use kubectl version): v1.27.2

  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubeadm v1.27.2

  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

I use helm deploy the rook-ceph and rook-ceph-cluster,but the mgr.b always restart due to enabled modules changedmgr.a is also have error.

@yehaifeng yehaifeng added the bug label Jul 17, 2023
@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

@rkachach
Copy link
Contributor

@yehaifeng can you please provide more details on how to reproduce the issue? ... for instance did you enable/disable any mgr modules?

@rkachach rkachach added the ceph-mgr Relating to the Ceph mgr or mgr modules label Oct 25, 2023
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

Copy link

github-actions bot commented Jan 4, 2024

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 4, 2024
@CrimsonFez
Copy link

I'm also having this issue. Included are operator and mgr logs from when the reload happened.
mgr-a-logs.txt
operator-logs.txt
Also included my ceph cluster cr (its a txt since gh wont let me upload a yaml)
cephcluster.txt

The mgrs will reload without any user action. They seam to do it every 5-10 minutes. My guess is that it's tied to operator reconciliation.

@rkachach rkachach reopened this Jan 22, 2024
@github-actions github-actions bot removed the wontfix label Jan 25, 2024
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

Copy link

github-actions bot commented Apr 6, 2024

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 6, 2024
@parth-gr parth-gr reopened this Sep 19, 2024
@github-actions github-actions bot removed the wontfix label Sep 19, 2024
@rkachach rkachach removed their assignment Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug ceph-mgr Relating to the Ceph mgr or mgr modules
Projects
None yet
Development

No branches or pull requests

4 participants