This is a draft of a possible one-off script that can be run within homer's venv to gather the FQDNs to test, attempt a connection and grab the fingerprint. This would allow to pre-populate a list that we could hardcode in hiera for now given the low refresh-rate of network devices. Then we could just adapt a bit ssh::publish_fingerprints in puppet to prepend/append those lines to the generated fingerprint files.

cumin1001$ REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt SSH_AUTH_SOCK=/run/keyholder/proxy.sock PATH="/srv/deployment/homer/venv/bin:${PATH}" PYTHONWARNINGS="ignore::UserWarning:paramiko.transport" python

from homer.config import load_yaml_config
from homer import Homer
from paramiko.client import SSHClient
from paramiko.ssh_exception import SSHException

config = load_yaml_config('/etc/homer/config.yaml')
homer = Homer(config)
devices = homer._devices.query('status:active')

fingerprints = {'ecdsa': [], 'ed25519': []}

for device in devices:
    targets = [device.fqdn, device.metadata['ip4']]
    if device.metadata.get('ip6'):
        targets.append(device.metadata['ip6'])
    client = SSHClient()
    # TODO: make 2 connections forcing the key type to get both fingerprints for each device
    try:
        client.connect(device.fqdn)  # Expected to fail because not in known_hosts
    except SSHException:
        rsa_key = client.get_transport().get_remote_server_key()
        fingerprint = f'{",".join(targets)} {rsa_key.get_name()} {rsa_key.get_base64()}'
        fingerprints[rsa_key.__class__.__name__.replace('Key', '').lower()].append(fingerprint)
    finally:
        client.close()

for key_type, lines in fingerprints.items():
    print(key_type)
    for line in lines:
        print(line)

With T336485 almost completed, we could consider integrating the two things, getting this one off exported in some place and then have the sre.network.provision cookbook take care to add new device fingerprints.
As for the deletion of old ones it could be manual for now or the cookbook could also remove any device not anymore active in Netbox from the existing list.
As to where to push the list I'm open for suggestions.
One possibility could be a git repo on the cumin hosts managed via spicerack's reposync.
The other option is of course to add it in Netbox.
@jbond do you have alternative suggestions?

My initial guess was to add them to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common.yaml#2046 but as this is going to be generated from Netbox with T329272 maybe having them directly in Netbox would be better. In Netbox 3.3 (planned to be upgraded with T336275: Upgrade Netbox to 4.x it will also be possible to hide custom fields (or set them read only).
A dedicated git repo seems a bit convoluted to me.

The network.provision cookbook could call a dedicated network.update-devices-fingerprints cookbook. So the latter could also be used independently (like after new OS install). We could also have monitoring in the form of a systemd-timer that runs the cookbook periodically on all the devices and sends an email/alert if there is anything outstanding.

It's also not very high up in the list of priorities :)

Netbox would be better.

+1 this would also allow use to have them in the netbox-hiera pipeline which in turn makes it easier to add them all to https://config-master.wikimedia.org/known_hosts instead of having a separate file