//srv/git/operations/puppet is no more updated due to a conflict. That prevent recent changes from being deployed as noticed in T370465#10048432
Description
Related Objects
Event Timeline
Mentioned in SAL (#wikimedia-releng) [2024-08-07T14:46:30Z] <hashar> deployment-prep: removed git tags from 2021, 2022, 2023 in /srv/git/operations/puppet # T371982
The git repo was quite slow and I could not rebase, it turns out the system has ran out of inodes:
hashar@deployment-puppetserver-1:~$ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb 640K 640K 1 100% /srv
/srv being a 9.8G disk
Eventually I have deleted the Puppet modules directory to free up some inodes. I have cleaned the reflog git reflog expire --expire=now --all and I was then able to run a git gc:
gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$ git gc --prune=now Enumerating objects: 852596, done. Counting objects: 100% (852596/852596), done. Delta compression using up to 4 threads Compressing objects: 100% (190275/190275), done. Writing objects: 100% (852596/852596), done. Total 852596 (delta 658159), reused 832054 (delta 638147), pack-reused 0 Removing duplicate objects: 100% (256/256), done. Checking connectivity: 852596, done. Expanding reachable commits in commit graph: 124698, done.
$ git count-objects -vH count: 0 size: 0 bytes in-pack: 852596 packs: 1 size-pack: 190.11 MiB prune-packable: 0 garbage: 0 size-garbage: 0 bytes
For some reason, there was 580K files in .git/objects and that exhausted the inodes. My guess is the auto syncer should do a git gc from time to time.
df -h -i /srv Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb 640K 67K 574K 11% /srv
Mentioned in SAL (#wikimedia-releng) [2024-08-07T15:10:12Z] <hashar> deployment-prep: fix permissions on puppet server: sudo chown gitpuppet:gitpuppet /srv/git/operations/puppet/modules/karapace # T371982
I ran sudo systemctl start puppet-git-sync-upstream.service to ensure it worked fine and surely that worked.
I don't know why it broke in the first place, but it is fixed.