Maniphest T371982

puppet having a merge conflict
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Aug 7 2024, 2:44 PM

Description

//srv/git/operations/puppet is no more updated due to a conflict. That prevent recent changes from being deployed as noticed in T370465#10048432

Related Objects

Mentioned In: T370465: Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation)
Mentioned Here: T370465: Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation)

Event Timeline

hashar created this task.Aug 7 2024, 2:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2024, 2:44 PM

hashar mentioned this in T370465: Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation).Aug 7 2024, 2:45 PM

Mentioned in SAL (#wikimedia-releng) [2024-08-07T14:46:30Z] <hashar> deployment-prep: removed git tags from 2021, 2022, 2023 in /srv/git/operations/puppet # T371982

The git repo was quite slow and I could not rebase, it turns out the system has ran out of inodes:

hashar@deployment-puppetserver-1:~$ df -hi
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sdb         640K  640K     1  100% /srv

/srv being a 9.8G disk

Eventually I have deleted the Puppet modules directory to free up some inodes. I have cleaned the reflog git reflog expire --expire=now --all and I was then able to run a git gc:

gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$ git gc --prune=now
Enumerating objects: 852596, done.
Counting objects: 100% (852596/852596), done.
Delta compression using up to 4 threads
Compressing objects: 100% (190275/190275), done.
Writing objects: 100% (852596/852596), done.
Total 852596 (delta 658159), reused 832054 (delta 638147), pack-reused 0
Removing duplicate objects: 100% (256/256), done.
Checking connectivity: 852596, done.
Expanding reachable commits in commit graph: 124698, done.

$ git count-objects -vH
count: 0
size: 0 bytes
in-pack: 852596
packs: 1
size-pack: 190.11 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

For some reason, there was 580K files in .git/objects and that exhausted the inodes. My guess is the auto syncer should do a git gc from time to time.

df -h -i /srv
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sdb         640K   67K  574K   11% /srv

Mentioned in SAL (#wikimedia-releng) [2024-08-07T15:10:12Z] <hashar> deployment-prep: fix permissions on puppet server: sudo chown gitpuppet:gitpuppet /srv/git/operations/puppet/modules/karapace # T371982

I ran sudo systemctl start puppet-git-sync-upstream.service to ensure it worked fine and surely that worked.

I don't know why it broke in the first place, but it is fixed.

deployment-prep puppet server is stall from two weeks ago due to /srv/git/operations/puppet having a merge conflictClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

deployment-prep puppet server is stall from two weeks ago due to /srv/git/operations/puppet having a merge conflict
Closed, ResolvedPublic
Actions