API rate limit exceeded for installation

[gr2m]

I was hoping that rate limiting is now properly taken care of since we moved to Ocotkit via #487, but the error popped up again

https://github.com/semantic-release/commit-analyzer/actions/runs/5159013164/jobs/9293389281#step:6:126

We can tell from the user agent that the correct @semantic-release/github version was used

      'user-agent': '@semantic-release/github v8.1.0 octokit-core.js/4.2.1 Node.js/18.16.0 (linux; x64)',

What's odd: the request option does not have a retries key, it looks like it didn't attempt to retry at all:
https://github.com/semantic-release/commit-analyzer/actions/runs/5159013164/jobs/9293389281#step:6:138

I will look into it. Sorry, I'm not sure why this is happening and rate limit errors are hard to reproduce reliably, it will take me a moment to get to the bottom of this. If anyone is inclined to help out, you are very welcome to

[gr2m]

One thing I want to look into is to avoid using search in the first place, we can look up associated pull requests based on commits using GraphQL now, there is no need to use search for that.

[Kampfmoehre]

If you need any example failed pipeline you can look into this one.

[dhensby]

I got hit by this issue earlier today and have done some digging to see what is going on.

So far there I've found 1 reason that is potentially causing retries to fail: https://github.com/semantic-release/github/blob/v9.0.4/lib/octokit.js#L25-L48

In the setup of the Octokit class, the retry config is passed in via RETRY_CONF, where RETRY_CONF is:

{
  // By default, Octokit does not retry on 404s.
  // But we want to retry on 404s to account for replication lag.
  doNotRetry: [400, 401, 403, 422],
}

But in the onRetry function, RETRY_CONFIG.retries is used and is undefined, this means the onRetry function will never return true and so retries are not attempted (I assume).

I tried to write a test to cover off the rate limit scenario, but I also noticed that the TestOctokit doesn't use SemanticReleaseOctokit, so the test suite doesn't test the logic as it is when run outside the test suite (ie: no retry, no throttle checks, etc). I suspect this has been done to improve speed of the test suite as when I did use the SemanticReleaseOctokit the test suite did take much longer to complete.

[Kampfmoehre]

Here is another failed pipeline, I saw in the logs that the search request was already limited. We use dependabot to keep deps up to date, resulting in a lot of small merge requests. Combined with not so regularly "manual" updates this results in a lot of issues having to be updated after release. But before at least some issues got updated, now it seems it doesn't even update one issue because searching for affecting issues already fails.

https://github.com/droidsolutions/semantic-release-update-file/actions/runs/8229131488/job/22499768584

Since logs vanish after some time, here is the request from the log

   status: 403,
  response: {
    url: 'https://api.github.com/search/issues?q=repo%3Adroidsolutions%2Fsemantic-release-update-file+type%3Apr+is%3Amerged+203258a9db965ddd4c498d1256ba4609809d1c61+c0b8a0c51a5efab6453fdd7175ed3250301c8196+0e56bf1bb4622b6cb28b58afe8426d2c591eb194+bd8e51d5ebaae09faff3310464a865586daa70a8',
    status: 403,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
      'content-encoding': 'gzip',
      'content-security-policy': "default-src 'none'",
      'content-type': 'application/json; charset=utf-8',
      date: 'Mon, 11 Mar 2024 07:30:29 GMT',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      server: 'GitHub.com',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'transfer-encoding': 'chunked',
      vary: 'Accept-Encoding, Accept, X-Requested-With',
      'x-accepted-oauth-scopes': 'repo',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-media-type': 'github.v3; format=json',
      'x-github-request-id': 'E20E:33A877:280331C:48F2596:65EEB313',
      'x-oauth-scopes': 'repo, workflow',
      'x-ratelimit-limit': '30',
      'x-ratelimit-remaining': '0',
      'x-ratelimit-reset': '1710142229',
      'x-ratelimit-resource': 'search',
      'x-ratelimit-used': '30',
      'x-xss-protection': '0'
    },
    data: {
      message: 'API rate limit exceeded for user ID 14077931. If you reach out to GitHub Support for help, please include the request ID E20E:33A877:280331C:48F2596:65EEB313.',
      documentation_url: 'https://docs.github.com/rest/overview/rate-limits-for-the-rest-api'
    }
  },
  request: {
    method: 'GET',
    url: 'https://api.github.com/search/issues?q=repo%3Adroidsolutions%2Fsemantic-release-update-file+type%3Apr+is%3Amerged+203258a9db965ddd4c498d1256ba4609809d1c61+c0b8a0c51a5efab6453fdd7175ed3250301c8196+0e56bf1bb4622b6cb28b58afe8426d2c591eb194+bd8e51d5ebaae09faff3310464a865586daa70a8',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': '@semantic-release/github v9.0.2 octokit-core.js/4.2.1 Node.js/20.11.1 (linux; x64)',
      authorization: 'token [REDACTED]'
    },
    request: { agent: undefined, hook: [Function: bound bound register] }
  },
  pluginName: '@semantic-release/github'
}

[dhensby]

@Kampfmoehre your version of @semantic-release/github is v9.0.2, but fixes have come in since then (eg: my fix was released in v9.0.5.

Upgrade your version of the package and I suspect it won't happen anymore (I've not experienced any issues since v9.0.5)

[Kampfmoehre]

@dhensby thx, I'll give it a try. We don't have the plugin as extra dep though, was previously using the one that comes with semantic-release package so you might want to update the min version there.

[gr2m]

using the one that comes with semantic-release package

The semantic-release package defines a range, you should get all updates automatically. If you have it as a dev dependency, make sure to update your lock file. We recommend to run it with just npx semantic-release without adding it as dependency

[ab-arao]

I think I'm running into the same problem as @Kampfmoehre and would appreciate some help getting the dependencies upgraded properly.

First, here's the error we are seeing:

[7:44:55 PM] [semantic-release] [@semantic-release/changelog] › ℹ  Create /__w/repo/repo/repo/CHANGELOG.md
[7:44:55 PM] [semantic-release] › ✔  Completed step "prepare" of plugin "@semantic-release/changelog"
[7:44:58 PM] [semantic-release] › ✔  Created tag v4.1.1
[7:44:58 PM] [semantic-release] › ℹ  Start step "publish" of plugin "@semantic-release/github"
[7:44:59 PM] [semantic-release] [@semantic-release/github] › ℹ  Published GitHub release: https://github.com/ourorg/repo/releases/tag/v4.1.1
[7:44:59 PM] [semantic-release] › ✔  Completed step "publish" of plugin "@semantic-release/github"
[7:44:59 PM] [semantic-release] › ℹ  Start step "success" of plugin "@semantic-release/github"
[7:45:00 PM] [semantic-release] [@semantic-release/github] › ℹ  Skip commenting on issues and pull requests.
Request quota exhausted for request GET /search/issues
Request quota exhausted for request GET /search/issues
Request quota exhausted for request GET /search/issues
Request quota exhausted for request GET /search/issues
Request quota exhausted for request GET /search/issues
[7:49:00 PM] [semantic-release] › ✘  Failed step "success" of plugin "@semantic-release/github"
[7:49:00 PM] [semantic-release] › ✘  An error occurred while running semantic-release: RequestError [HttpError]: You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 2840:295B44:3868737:64A2183:6232352C.
    at /usr/lib/node_modules/semantic-release/node_modules/@octokit/request/dist-node/index.js:124:21
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async requestWithGraphqlErrorHandling (/usr/lib/node_modules/semantic-release/node_modules/@octokit/plugin-retry/dist-node/index.js:71:20)
    at async Job.doExecute (/usr/lib/node_modules/semantic-release/node_modules/bottleneck/light.js:405:18) {
  status: 403,
  response: {
    url: 'https://api.github.com/search/issues?q=in%3Atitle+repo%3Aourorg%2Frepo+type%3Aissue+state%3Aopen+The%20automated%20release%20is%20failing%20%F0%9F%9A%A8',
    status: 403,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
      'content-encoding': 'gzip',
      'content-security-policy': "default-repo 'none'",
      'content-type': 'application/json; charset=utf-8',
      date: 'Tue, 28 May 2024 19:49:00 GMT',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      server: 'GitHub.com',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'transfer-encoding': 'chunked',
      vary: 'Accept-Encoding, Accept, X-Requested-With',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-api-version-selected': '2022-11-28',
      'x-github-media-type': 'github.v3; format=json',
      'x-github-request-id': '2840:295B44:3868737:64A2183:6232352C',
      'x-ratelimit-limit': '30',
      'x-ratelimit-remaining': '30',
      'x-ratelimit-reset': '1716925800',
      'x-ratelimit-resource': 'search',
      'x-ratelimit-used': '1',
      'x-xss-protection': '0'
    },
    data: {
      documentation_url: 'https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits',
      message: 'You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 2840:295B44:3868737:64A2183:6232352C.'
    }
  },
  request: {
    method: 'GET',
    url: 'https://api.github.com/search/issues?q=in%3Atitle+repo%3Aourorg%2Frepo+type%3Aissue+state%3Aopen+The%20automated%20release%20is%20failing%20%F0%9F%9A%A8',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': '@semantic-release/github v9.2.6 octokit-core.js/5.0.2 Node.js/21',
      authorization: 'token [REDACTED]'
    },
    request: {
      agent: undefined,
      hook: [Function: bound bound register],
      retryCount: 3
    }
  },
  pluginName: '@semantic-release/github'
}

As you can see, it says we are using @semantic-release/github v9.2.6.

However, in the build container we use the dependencies are the following:

$ docker run --rm -it build-container:latest npm view semantic-release dependencies
{
  '@semantic-release/commit-analyzer': '^12.0.0',
  '@semantic-release/error': '^4.0.0',
  '@semantic-release/github': '^10.0.0',
  '@semantic-release/npm': '^12.0.0',
  '@semantic-release/release-notes-generator': '^13.0.0',
  'aggregate-error': '^5.0.0',
  cosmiconfig: '^9.0.0',
  debug: '^4.0.0',
  'env-ci': '^11.0.0',
  execa: '^9.0.0',
  figures: '^6.0.0',
  'find-versions': '^6.0.0',
  'get-stream': '^6.0.0',
  'git-log-parser': '^1.2.0',
  'hook-std': '^3.0.0',
  'hosted-git-info': '^7.0.0',
  'import-from-esm': '^1.3.1',
  'lodash-es': '^4.17.21',
  marked: '^12.0.0',
  'marked-terminal': '^7.0.0',
  micromatch: '^4.0.2',
  'p-each-series': '^3.0.0',
  'p-reduce': '^3.0.0',
  'read-package-up': '^11.0.0',
  'resolve-from': '^5.0.0',
  semver: '^7.3.2',
  'semver-diff': '^4.0.0',
  signale: '^1.2.1',
  yargs: '^17.5.1'
}

We also execute semantic-release with npx as recommended in a previous comment.

Given that semantic-release should be using at least version 10.0.0 of @semantic-release/github, I am not sure how v9.2.6 is actually responsible for making the request.

I'd appreciate any guidance on what to upgrade in order to use a newer version that has the rate-limiting fix.

Thank you!

[dhensby]

@ab-arao you've not run into the same problem as @Kampfmoehre. They ran into the canonical issue of this thread (which is that if the API limit was hit then the job failed immediately).

What was meant to happen is that a secondary back-off was meant to take place to try to "wait out" the API limit resetting. Your job has done that (hence the Request quota exhausted for request GET /search/issues which didn't happen for @Kampfmoehre), and so is working "as expected" even though it's not perfect.

I think it's time this issue was closed, as I believe the issue is fixed. It may be worth a new issue opened about the secondary rate-limit and a more reliable way to wait-out the API limit even more gracefully.

[ab-arao]

Got it, thanks for clarifying @dhensby! I agree it makes sense to treat this as a separate issue since now the secondary rate limit is the issue.