it has its dedicated PS now: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1063167

host is fully repooled with no issue.

Here is the cookbook update with the proper methods, from the released spicerack version.

thanks @VRiley-WMF for the disk swap, I'll keep the task open to check how the rebuild went later on

sanitarium instances have been restarted with the patch

Sure, let me know when you're up today

@Marostegui I've sent this patch:

@VRiley-WMF please let me know when you're ready, I'll depool the node then

In T370852#10028352, @Marostegui wrote:

@ABran-WMF please coordinate with @cmooney for this.

forgot to add:

All done, ready for the views creation.

Thank you for the explanation 🙏

@Marostegui and I will be absent on tuesday, hosts have been depooled and are ready.

data-persistence hosts handled, ready whenever you are @cmooney

Since https://github.com/wikimedia/operations-software-spicerack/blob/v8.8.0/CHANGELOG.rst has been tested, this can be considered quite done, only remains the improvement of exception carrying when a SQL command fails on a host, tracked in T370419

sure thing!

Could be considered as redundant of T367283 which would offer a more specific angle.

done!

I've updated

• dbstore1009
• db1164 (m1)
• db1176 (m5)

In T370029#9984960, @Volans wrote:

The issue is due to the fact that cumin2002 has python3-wmfmariadbpy at version 0.10 while cumin1002 and most of the host with that package has version 0.11.2.
[...] how do you manage the versioning and upgrade of this package?
Full list of hosts with the old version available in Debmonitor: https://debmonitor.wikimedia.org/packages/python3-wmfmariadbpy

This server has been down for a few days, @wiki_willy please let me know if I can help

thanks! will roll the change there as well

dbstore1009 has replication up to date on all 3 instances

We could probably try and rely on read_default_file like we do in other tools

debug is already in progress, this task is to track what's been done

In T362529#9980552, @BTullis wrote:

@ABran-WMF, @fnegri - I believe that we are ready to run the sre.wikireplicas.add-wiki cookbook for this wiki, which should make it available on both clouddb* and an-redacteddb1001 hosts.
Are you happy for us (myself and @Stevemunene ) to run that now, or is there anything else that either of you feels need be done first?

execution collided with T367856 on s7, stopped and repooling will resume monday.

I am unable to reach it via management interface either, it might need a bit of hands on

dbhost repooling
dbproxy reloaded
backuphost checked and looks green

@Zabe it seems we were missing the "storage layer" task we usually get. Anyway, this is done on our side.

db1190 repooling
dbproxy reloaded

This is done, we'll iterate and monitor elsewhere if needed.

Exporter new configuration has been merged, it's rolling out on production. Rules have been merged as well

we've not seen any regression since you released the update, I think you're good to go!

In T367278#9956653, @Marostegui wrote:

We could also go for Misc hosts in codfw as testing.

will do!

@fgiunchedi fyi I've started rolling out the backport version on clouddb hosts, I've left aside clouddb1019 as @fnegri told me it was kind of unstable recently. I'll let it sit for the week-end before merging anything and go for the remaining hosts that need that version installed. @jcrespo don't worry about this change, it's very safe and monitored on my side, it'll cover backup hosts that have mysqld-exporter installed as well.

I'm affraid thats an answer I don't have @BTullis maybe @Marostegui or @Ladsgroup knows.

Amazing 🎉 lets maye try the first deployment from a canary cumin host so we're 100% sure that there is no breaking change.

Ah @BTullis I see the issue you face, I had the same one, sorry for not spotting it sooner!