Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reduce the number of parallel file reads on Windows to avoid EMFILE type errors #6009

Merged
merged 5 commits into from
Jun 12, 2024

Conversation

RamIdeas
Copy link
Contributor

@RamIdeas RamIdeas commented Jun 11, 2024

What this PR solves / how to test

Reapplies the batching reverted in #6002 but with a tested helper method to ensure batches are correctly compiled

Author has addressed the following

  • Tests
    • TODO (before merge)
    • Included
    • Not necessary because:
  • E2E Tests CI Job required? (Use "e2e" label or ask maintainer to run separately)
    • I don't know
    • Required / Maybe required
    • Not required because:
  • Changeset (Changeset guidelines)
    • TODO (before merge)
    • Included
    • Not necessary because:
  • Public documentation
    • TODO (before merge)
    • Cloudflare docs PR(s):
    • Not necessary because:

@RamIdeas RamIdeas requested review from a team as code owners June 11, 2024 15:22
Copy link

changeset-bot bot commented Jun 11, 2024

🦋 Changeset detected

Latest commit: f919908

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
wrangler Patch
@cloudflare/vitest-pool-workers Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link
Contributor

github-actions bot commented Jun 11, 2024

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9488330209/npm-package-wrangler-6009

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/6009/npm-package-wrangler-6009

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9488330209/npm-package-wrangler-6009 dev path/to/script.js
Additional artifacts:
npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9488330209/npm-package-create-cloudflare-6009 --no-auto-update
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9488330209/npm-package-cloudflare-kv-asset-handler-6009
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9488330209/npm-package-miniflare-6009
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9488330209/npm-package-cloudflare-pages-shared-6009
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9488330209/npm-package-cloudflare-vitest-pool-workers-6009

Note that these links will no longer work once the GitHub Actions artifact expires.


[email protected] includes the following runtime dependencies:

Package Constraint Resolved
miniflare workspace:* 3.20240605.0
workerd 1.20240605.0 1.20240605.0
workerd --version 1.20240605.0 2024-06-05

Please ensure constraints are pinned, and miniflare/workerd minor versions match.

Copy link
Contributor

@andyjessop andyjessop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@@ -0,0 +1,8 @@
export function* createBatches<T>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason this needs to be a generator rather than a standard function returning an array?

@petebacondarwin petebacondarwin self-assigned this Jun 12, 2024
@petebacondarwin petebacondarwin changed the title use createBatches to batch assets for Pages assets upload fix: reduce the number of parallel file reads on Windows to avoid EMFILE type errors Jun 12, 2024
@petebacondarwin petebacondarwin merged commit 169a9fa into main Jun 12, 2024
19 checks passed
@petebacondarwin petebacondarwin deleted the batch-pages-assets-upload branch June 12, 2024 19:25
@workers-devprod workers-devprod added the contribution [Holopin] Recognizes an open-source contribution, big or small label Jun 12, 2024
Copy link

holopin-bot bot commented Jun 12, 2024

Congratulations @RamIdeas, you just earned a holobyte! Here it is: https://holopin.io/holobyte/clxc80u4s12320cl7w3k99jbv

This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account.
Or if you're new to Holopin, you can simply sign up with GitHub, which will do the trick!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution [Holopin] Recognizes an open-source contribution, big or small
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants