User Details
- User Since
- Apr 3 2017, 6:23 PM (384 w, 5 d)
- Availability
- Available
- IRC Nick
- xionox
- LDAP User
- Ayounsi
- MediaWiki User
- AYounsi (WMF) [ Global Accounts ]
Fri, Aug 16
All done.
Aug 16 09:24:25 netbox1003 python[1079619]: 09:24:25 default: extras.scripts.run_script(commit=True, data={}, job=<Job: 61e3e2e3-59b1-4982-81a7-5589221d7ed9>, request=<utilities.request.NetBoxFakeRequest ob>
Aug 16 09:25:47 netbox1003 python[1622900]: 09:25:47 default: Job OK (61e3e2e3-59b1-4982-81a7-5589221d7ed9)Wed, Aug 14
Sounds good to me !
>>> pprint.pprint(results) defaultdict(<class 'int'>, {'an-redacteddb': 1, 'clouddb': 9, 'db': 251, 'dbprov': 6, 'dbproxy': 18, 'dbstore': 3, 'dumpsdata': 1, 'es': 42, 'ganeti': 29, 'maps': 12, 'ms-be': 16, 'pc': 14, 'restbase': 30, 'snapshot': 1, 'thanos-fe': 6, 'wdqs': 5}) >>> >>> print(set(NO_V6_DEVICE_NAME_PREFIXES) - set(results.keys())) {'thumbor', 'mc-gp', 'restbase-dev', 'mc', 'wtp', 'mwlog', 'mw', 'parse', 'graphite', 'ores', 'sessionstore'}
Tue, Aug 13
With the patches above, the last urgent-ish thing needed is to package the new pynetbox for the cumin hosts (bullseye)
https://netbox.wikimedia.org/extras/scripts/results/78992/
cloudcephosd1039 (WMF11571) /dcim/devices/5296/ Primary IPv6 missing DNS name
I guess the skip IPv6 box got checked by mistake, could someone add the host's FQDN to https://netbox.wikimedia.org/ipam/ip-addresses/17171/ (similar to https://netbox.wikimedia.org/ipam/ip-addresses/17159/) then run the sre.dns.netbox cookbook ?
Mon, Aug 12
Emailed AS54994 and cleared the errors for the others.
Nevertheless, it should be possible to publish ASPA records in RPKI through the ARIN portal
I looked a bit around Arin's RPKI's portal but couldn't find it, is there doc about it ?
However, in reality, it should be possible to reject all IP packets where the source IP is not part of the IP prefixes that the Foundation has been assigned (i.e. prefix lists production{4,6}, which are a superset of the publicly routable LVS service IPs).
We would need to at least permit traffic from the transit interface IPs, as they do BGP to their peers, v6 link local for neighbor discovery, some land GRE tunnels, etc. Not sure what is the cleanest way for that, maybe using an apply-path like for bgp-sessions ? Ideally we wouldn't have to list them all :)
Thu, Aug 8
Cleaned up.
This went very well until it didn't. Changes fully rolled back.
Thanks, it's fixed for those 2 hosts.
Wed, Aug 7
Sent the patches for the last few ones left in the task description.
Notes from the Debrief meeting
All is done !
Tue, Aug 6
testing done!
Mon, Aug 5
@Dwisehaupt could you send a patch to add 10.195.1.0/25to subnet-administration-codfw in the pfw filters (alongside of the current 10.195.0.64/28) ?
Fri, Aug 2
I played a bit with the plugin and it's quite easy to duplicate what we currently do:
As a data point, GetDeviceStats runs ~5000 times per day, which clutters the DB and probably contributes to Netbox's overall slowness. https://netbox.wikimedia.org/core/jobs/?q=GetDeviceStats&created__before=2024-08-02+12%3A00%3A00&created__after=2024-08-01+12%3A00%3A00
Confirmed working:
>>> robj = api.extras.scripts.get('import_server_facts.ImportPuppetDB').url [...] TypeError: a bytes-like object is required, not 'str'
>>> robj = api.extras.scripts.get('import_server_facts.ImportPuppetDB').url >>> robj 'https://netbox-next.discovery.wmnet/api/extras/scripts/20/'
The re-image cookbook is hitting this bug too: https://github.com/netbox-community/pynetbox/pull/632
Thu, Aug 1
No pb! Done. See for example https://netbox.wikimedia.org/dcim/devices/1969/
python
Traceback (most recent call last):
File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/fields/related_descriptors.py\", line 236, in __get__
rel_obj = self.field.get_cached_value(instance)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/fields/mixins.py\", line 15, in get_cached_value
return instance._state.fields_cache[cache_name]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'role'Going to close that one as I can't reproduce it on Netbox 4. Please reopen if needed.
+1 to keep both for now.
Wed, Jul 31
Tue, Jul 30
We can potentially re-use the move_server.MoveServer script but make the server selection a MultiObjectVar as input and make the rack U and switch port optional.
Mon, Jul 29
Thu, Jul 25
Mon, Jul 22
Amazing progress !
This doesn't actually seem to be the case? We have no priority set on either router.
Indeed, not anymore since we moved to 100G between eqiad/codfw
There is an outstanding diff on the switch for cloudcephmon1006. It looks correct, but could DCops double check it and make sure the switch config is in sync with Netbox ?
Jul 18 2024
That's a great idea ! Happy to help if needed.
fixed-address sretest2001.codfw.wmnet; this needs to be fixed-address $some-ip-address;
I'd suggest to abstract the device creation by a custom script or cookbook. This could run additional safeguards, ask for addition information (and store them where necessary), and hide the unnecessary fields.
Or we could just use a IPv6 /64 and stop worrying about space :)
Thanks for the investigation ! Seems like the last step was :
asw1-b3-magru> restart analytics-agent gracefully Analytics agent started, pid 87102
Solved the issue.
Jul 17 2024
@cmooney @fgiunchedi I'm wondering if the probe could/should be changed to a TCP handshake only or totally removed. In order to reduce log spam.
Thanks to Rancid and the daily diff scripts we already get a notification within the hour if SSH gets unreachable.
While still keeping the ICMP check for the "normal" connectivity probe.
We will need to migrate the whole range to a new prefix :( Running 2 ranges is going to be a pain long term, and would need much more work on the automation side than a migration.
Yep it's all good ! I manually added the host to gNMIc and metrics are properly being collected/exposed. Thanks !
Jul 16 2024
Thanks.
I went that way, master on https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/netbox has been updated to match upstream's master.
https://gerrit.wikimedia.org/r/c/operations/software/netbox-deploy/+/1053243 for the netbox-deploy side.
There has been a spike of CPU usage on cr1-eqiad (with no impact), not sure if just a coincidence.
Closing this task in favor of T364092: Upgrade core routers to Junos 22.4R3.