"You have exceeded a secondary rate limit."

[austinpray-mixpanel]

Hey friends!

I'm using this action to clean up 15,000+

I set operations-per-run to a somewhat conservative 100 so we can make slow but steady progress on that giant backlog. I'm getting hit with secondary rate limit errors though:

RequestError [HttpError]: You have exceeded a secondary rate limit. Please wait a few minutes before you try again.

Full error output

/home/runner/work/_actions/fpicalausa/remove-stale-branches/v2.1.0/dist/index.js:4181
      const error = new requestError.RequestError(toErrorMessage(data), status, {
                    ^

RequestError [HttpError]: You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID XXXXXXXX.
    at /home/runner/work/_actions/fpicalausa/remove-stale-branches/v2.1.0/dist/index.js:4181:21
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  status: 403,
  response: {
    url: 'https://api.github.com/graphql',
    status: 403,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
      'content-encoding': 'gzip',
      'content-security-policy': "default-src 'none'; base-uri 'self'; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com api.githubcopilot.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com/v1/engines/github-completion/completions proxy.enterprise.githubcopilot.com/v1/engines/github-completion/completions; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/",
      'content-type': 'application/json; charset=utf-8',
      date: 'Tue, 30 Jul 2024 17:17:23 GMT',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      'retry-after': '60',
      server: 'GitHub.com',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'transfer-encoding': 'chunked',
      vary: 'Accept-Encoding, Accept, X-Requested-With',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-media-type': 'github.v4; format=json',
      'x-github-request-id': 'XXXXXXXXXX',
      'x-xss-protection': '0'
    },
    data: {
      documentation_url: 'https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits',
      message: 'You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID XXXXXXXXXX.'
    }
  },
  request: {
    method: 'POST',
    url: 'https://api.github.com/graphql',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'octokit-core.js/3.6.0 Node.js/20.13.1 (linux; x64)',
      authorization: 'token [REDACTED]',
      'content-type': 'application/json; charset=utf-8'
    },
    body: '{"query":"query ($repo: String!, $owner: String!, $after: String) {\\n  repository(name: $repo, owner: $owner) {\\n    id\\n    refs(\\n      refPrefix: \\"refs/heads/\\",\\n      first: 10,\\n      after: $after,\\n    ) {\\n      edges {\\n        node {\\n          name\\n          associatedPullRequests(first: 1, states: OPEN) {\\n            nodes {\\n              state\\n            }\\n          }\\n          prefix\\n          ... on Ref {\\n            refUpdateRule {\\n              allowsDeletions\\n            }\\n          }\\n          target {\\n          ... on Commit {\\n              oid\\n              authoredDate\\n              author {\\n                email\\n                user {\\n                  login\\n                }\\n              }\\n            }\\n          }\\n        }\\n      }\\n      pageInfo {\\n        hasNextPage\\n        endCursor\\n      }\\n    }\\n  }\\n}","variables":{"owner":"mixpanel","repo":"analytics","after":"NDU1MA","organization":""}}',
    request: {
      agent: Agent {
        _events: [Object: null prototype] {
          free: [Function (anonymous)],
          newListener: [Function: maybeEnableKeylog]
        },
        _eventsCount: 2,
        _maxListeners: undefined,
        defaultPort: 443,
        protocol: 'https:',
        options: [Object: null prototype] {
          keepAlive: true,
          scheduling: 'lifo',
          timeout: 5000,
          noDelay: true,
          path: null
        },
        requests: [Object: null prototype] {},
        sockets: [Object: null prototype] {},
        freeSockets: [Object: null prototype] {
          'api.github.com:443:::::::::::::::::::::': [ [TLSSocket] ]
        },
        keepAliveMsecs: 1000,
        keepAlive: true,
        maxSockets: Infinity,
        maxFreeSockets: 256,
        scheduling: 'lifo',
        maxTotalSockets: Infinity,
        totalSocketCount: 1,
        maxCachedSessions: 100,
        _sessionCache: {
          map: {
            'api.github.com:443:::::::::::::::::::::': [Buffer [Uint8Array]]
          },
          list: [ 'api.github.com:443:::::::::::::::::::::' ]
        },
        [Symbol(shapeMode)]: false,
        [Symbol(kCapture)]: false
      },
      hook: [Function: bound bound register]
    }
  }
}

Node.js v20.13.1

https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#about-secondary-rate-limits

I think octokit has some retry middlewares and stuff that comply with these best practices that might make this issue less prevalent:

https://docs.github.com/en/rest/using-the-rest-api/best-practices-for-using-the-rest-api?apiVersion=2022-11-28

[austinpray-mixpanel]

I'm actually still getting this issue with operations-per-run set to 10

name: Remove Stale Branches

on:
  schedule:
    - cron: '0 16-21 * * 1-5' # couple times a day M-F
  workflow_dispatch: {}

jobs:
  remove_stale_branches:
    name: Remove Stale Branches
    runs-on: ubuntu-20.04
    timeout-minutes: 100
    steps:
      - uses: fpicalausa/[email protected]
        with:
          # dry-run: true
          days-before-branch-stale: 2190 # 6 years
          days-before-branch-delete: 14
          ignore-unknown-authors: true
          default-recipient: austinpray-mixpanel
          operations-per-run: 10

[tucker-pw]

I am also running into this, we have an old and still active repo with 1000+ stale branches we're trying to remove.

---

Workflow:

name: Stale Branch Cleanup

on:
  schedule:
    - cron: "0 0 * * 1" # At 00:00 on Monday.
  workflow_dispatch: # set this so workflow can be run on demand

permissions:
  contents: write # Required to delete branches
  pull-requests: read # Ensures open PRs are checked

jobs:
  remove-stale-branches:
    name: Remove Stale Branches
    runs-on: ubuntu-latest
    steps:
      - uses: fpicalausa/[email protected]
        with:
          dry-run: false
          ignore-unknown-authors: true
          days-before-branch-stale: 365
          days-before-branch-delete: 0
          default-recipient: tucker-pw
          operations-per-run: 1500
          ignore-branches-with-open-prs: true
          exempt-authors-regex: "^(dependabot|renovate)"

---

Error:

✏ branch JIRA-0000-seed-recs-native-translations-addtoplandrawer
  -> branch was last updated by andrewest on 2023-08-30T18:17:55Z
  -> branch will be removed on 2024-10-23T21:25:59Z
  -> marking branch as stale (notifying: andrewest)
/home/runner/work/_actions/fpicalausa/remove-stale-branches/v2.1.0/dist/index.js:4181
      const error = new requestError.RequestError(toErrorMessage(data), status, {
                    ^

RequestError [HttpError]: You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later. If you reach out to GitHub Support for help, please include the request ID C001:1348C8:50091BA:50B102D:67196A24 and timestamp 2024-10-23 21:27:00 UTC.
    at /home/runner/work/_actions/fpicalausa/remove-stale-branches/v2.1.0/dist/index.js:4181:21
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  status: 403,
  response: {
    url: 'https://api.github.com/repos/MyOrg/my-repo/commits/91855d641227dae84b7a2244f2fe132e76e296b2/comments',
    status: 403,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
      'content-encoding': 'gzip',
      'content-security-policy': "default-src 'none'",
      'content-type': 'application/json; charset=utf-8',
      date: 'Wed, 23 Oct 2024 21:27:00 GMT',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      server: 'github.com',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'transfer-encoding': 'chunked',
      vary: 'Accept-Encoding, Accept, X-Requested-With',
      'x-accepted-github-permissions': 'contents=read',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-api-version-selected': '2022-11-28',
      'x-github-media-type': 'github.v3; format=json',
      'x-github-request-id': 'C001:1348C8:50091BA:50B102D:67196A24',
      'x-ratelimit-limit': '15000',
      'x-ratelimit-remaining': '14754',
      'x-ratelimit-reset': '1729721251',
      'x-ratelimit-resource': 'core',
      'x-ratelimit-used': '246',
      'x-xss-protection': '0'
    },
    data: {
      message: 'You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later. If you reach out to GitHub Support for help, please include the request ID C001:1348C8:50091BA:50B102D:67196A24 and timestamp 2024-10-23 21:27:00 UTC.',
      documentation_url: 'https://docs.github.com/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits',
      status: '403'
    }
  },
  request: {
    method: 'POST',
    url: 'https://api.github.com/repos/MyOrg/my-repo/commits/91855d641227dae84b7a2244f2fe132e76e296b2/comments',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'octokit-core.js/3.6.0 Node.js/20.13.1 (linux; x64)',
      authorization: 'token [REDACTED]',
      'content-type': 'application/json; charset=utf-8'
    },
    body: `{"body":"[stale:JIRA-0000-seed-recs-native-translations-addtoplandrawer]\\r\\n\\r\\n@andrewest Your branch [JIRA-0000-seed-recs-native-translations-addtoplandrawer](https://github.com/MyOrg/my-repo/tree/JIRA-0000-seed-recs-native-translations-addtoplandrawer) hasn't been updated in the last 365 days and is marked as stale. It will be removed in 0 days.\\r\\nIf you want to keep this branch around, add new commits to this branch or protect it."}`,
    request: {
      agent: Agent {
        _events: [Object: null prototype] {
          free: [Function (anonymous)],
          newListener: [Function: maybeEnableKeylog]
        },
        _eventsCount: 2,
        _maxListeners: undefined,
        defaultPort: 443,
        protocol: 'https:',
        options: [Object: null prototype] {
          keepAlive: true,
          scheduling: 'lifo',
          timeout: 5000,
          noDelay: true,
          path: null
        },
        requests: [Object: null prototype] {},
        sockets: [Object: null prototype] {},
        freeSockets: [Object: null prototype] {
          'api.github.com:443:::::::::::::::::::::': [ [TLSSocket] ]
        },
        keepAliveMsecs: 1000,
        keepAlive: true,
        maxSockets: Infinity,
        maxFreeSockets: 256,
        scheduling: 'lifo',
        maxTotalSockets: Infinity,
        totalSocketCount: 1,
        maxCachedSessions: 100,
        _sessionCache: {
          map: {
            'api.github.com:443:::::::::::::::::::::': [Buffer [Uint8Array]]
          },
          list: [ 'api.github.com:443:::::::::::::::::::::' ]
        },
        [Symbol(shapeMode)]: false,
        [Symbol(kCapture)]: false
      },
      hook: [Function: bound bound register]
    }
  }
}

[fpicalausa]

@tucker-pw with 1500 operations per runs, github is likely complaining that we're trying to mark too many branches as stale (i.e. writing a comment) within a short period of time.

@austinpray-mixpanel if you are still meeting this issue even with 10 operations per runs, could it be that the same action with the same access token is reused across different repos at around the same time? I ran into a similar issue, and the solution was to stagger the runs throughout the day.

[austinpray-mixpanel]

could it be that the same action with the same access token is reused across different repos at around the same time?

@fpicalausa I'm using the default github actions token. The problem is definitely this:

Make too many requests to a single endpoint per minute. No more than 900 points per minute are allowed for REST API endpoints, and no more than 2,000 points per minute are allowed for the GraphQL API endpoint. For more information about points, see "Calculating points for the secondary rate limit."

The graphql queries fetching all the branches are definitely hitting that 2000 points per minute issue since we have 15k branches