aws: add rate_limiter_capacity to configure client-side rate limits #3497
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vangent
reviewed
Oct 9, 2024
stanhu
force-pushed
the
sh-disable-aws-client-side-ratelimits
branch
from
a54e8c8
to
9593ec5
Compare
stanhu
changed the title
stanhu
force-pushed
the
sh-disable-aws-client-side-ratelimits
branch
from
9593ec5
to
60a130a
Compare
Issuing many S3 uploads in quick succession causes the following error: operation error S3: PutObject, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested By default, this rate limit is disabled to restore the same behavior as SDK v1. If `rate_limiter_capacity` is set, then client-side rate limits will take effect. Closes google#3496
stanhu
force-pushed
the
sh-disable-aws-client-side-ratelimits
branch
from
60a130a
to
652c9d3
Compare
vangent
approved these changes
Oct 10, 2024
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3497 +/- ##
==========================================
+ Coverage 73.24% 73.76% +0.52%
==========================================
Files 113 113
Lines 15019 15317 +298
==========================================
+ Hits 11000 11299 +299
- Misses 3257 3258 +1
+ Partials 762 760 -2 ☔ View full report in Codecov by Sentry. |
gitlab-runner-bot
pushed a commit
to gitlabhq/gitlab-runner
that referenced
this pull request
Oct 11, 2024
This commit bumps the version of gocloud.dev to disable AWS client-side rate limiting if `rate_limiter_capacity` is not specified (google/go-cloud#3497). Gitaly users ran into these limits when backing files up to S3 (https://gitlab.com/gitlab-org/gitaly/-/issues/6417). This likely won't be an issue for GitLab Runner since a new client is created each time a cache file is uploaded, but just in case let's update. Changelog: fixed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issuing many S3 uploads in quick succession causes the following error:
By default, this rate limit is disabled to restore the same behavior as SDK v1. If
rate_limiter_capacityis set, then client-side rate limits will take effect.Closes #3496