Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore the 404 error when deleting the bucket to make it more resilie… #161

Merged
merged 1 commit into from
Aug 9, 2022

Conversation

rmt2021
Copy link
Contributor

@rmt2021 rmt2021 commented Aug 6, 2022

…nt to transient errors.

Bug Description

We encounter an unhandled exception which is caused by an error code 404 when callingDeleteBucketAsync to delete an existing S3 bucket. The exception is unexpected because the existing bucket should be successfully deleted and no 404 exception should be thrown.

We found that the retry mechanism of DeleteBucketAsync can actually lead to a 404 error even if the bucket exists when issuing the deletion. More concretely:

  1. DeleteBucketAsync sends a HTTP request to delete the bucket
  2. The bucket is deleted in the remote by the first try but a transient network errors happen (e.g., timeout), so the successful response never arrive the client, instead the transient error code 503 Server Busy may arrive Sleet.
  3. The timeout (or transient error code) triggers the SDK API to retry the deletion
  4. The second try fails and encounters a 404 error because the bucket has already been deleted by the first try
  5. The AWS S3 SDK throws this 404 exception to Sleet

Expected behavior & Possible Solution

As implemented in this PR:

  1. Ignore the 404 error when deleting the bucket.
  2. Add a logging of the transient error could help improve the diagnosability.

I hope this can help! Thanks.

Copy link
Owner

@emgarten emgarten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the great explanation and fix @rmt2021 🏆

@emgarten emgarten merged commit 03f8eb9 into emgarten:main Aug 9, 2022
@emgarten
Copy link
Owner

emgarten commented Aug 9, 2022

This has been released in https://www.nuget.org/packages/Sleet/5.0.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants