Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: significantly increase build speed #114

Merged
merged 3 commits into from
Oct 8, 2023
Merged

Conversation

rustatian
Copy link
Member

@rustatian rustatian commented Oct 8, 2023

Reason for This PR

  • Slow builds.

Description of Changes

  • Parallelize network operations as much as possible.

  • Approx 10x faster builds:

  • Right - Old, Left - New velox (go clean -modcache), first run:
    image

18.14s vs 275.06s

  • Same, but second run with modcache:
    image

3.5s vs 36s

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the MIT license.

PR Checklist

[Author TODO: Meet these criteria.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

  • All commits in this PR are signed (git commit -s).
  • The reason for this PR is clearly provided (issue no. or explanation).
  • The description of changes is clear and encompassing.
  • Any required documentation changes (code and docs) are included in this PR.
  • Any user-facing changes are mentioned in CHANGELOG.md.
  • All added/changed functionality is tested.

Summary by CodeRabbit

Release Notes

  • New Feature: Upgraded dependencies to github.com/roadrunner-server/rpc/v4 and github.com/roadrunner-server/config/v4 in builder/templates/templateV2023.go for enhanced functionality and compatibility.
  • Refactor: Improved error handling in github/pool.go with the introduction of the appendErr method, providing thread-safe error accumulation.
  • Performance: Increased queueCh channel buffer size in github/pool.go from 1 to 100, allowing for better concurrency and throughput.
  • Bug Fix: Ensured thread safety in github/pool.go by returning copies of errs and modinfo slices in errors and moduleinfo methods respectively, preventing potential data races.

Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
@rustatian rustatian added the B-performance Bug: performance issues label Oct 8, 2023
@rustatian rustatian requested a review from wolfy-j October 8, 2023 15:57
@rustatian rustatian self-assigned this Oct 8, 2023
@coderabbitai
Copy link

coderabbitai bot commented Oct 8, 2023

Walkthrough

This update introduces new dependencies in templateV2023.go, enhances error handling, and improves data race conditions in pool.go by modifying the processor struct and its methods.

Changes

File Summary
builder/templates/.../templateV2023.go Added two new dependencies: github.com/roadrunner-server/rpc/v4 and github.com/roadrunner-server/config/v4.
github/pool.go Increased queueCh channel buffer size from 1 to 100. Enhanced error handling in run method using appendErr. errors and moduleinfo methods now return copies of slices.

🐇

In the land of code, where the shadows lie,

Two dependencies were added, under the sky.

In the pool of tasks, where the goroutines fly,

Buffer size increased, reaching high.

Error handling improved, no more sigh,

Copies of slices, we now supply.

Celebrate these changes, don't be shy,

For in the land of code, we aim to comply. 🎉


Tips

Chat with CodeRabbit Bot (@coderabbitai)

  • Mention @coderabbitai in any review comment for bot assistance.
  • Note: Review comments are made on code diffs or files, not on the PR overview.

Pause Incremental Reviews

  • Insert @coderabbitai: ignore in the PR description to halt the bot's ongoing reviews. Remove the line to resume.
  • Additionally, enter @coderabbitai pause to halt the bot's ongoing reviews as any issue or review comment. To
    resume reviews, enter @coderabbitai resume as any issue or review comment.

@rustatian rustatian merged commit e492f1a into master Oct 8, 2023
2 checks passed
@rustatian rustatian deleted the feature/build-speed branch October 8, 2023 15:57
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Commits Files that changed from the base of the PR and between ba6d0f0 and 3fe6b54.
Files selected for processing (2)
  • builder/templates/templateV2023.go (1 hunks)
  • github/pool.go (8 hunks)
Files skipped from review due to trivial changes (1)
  • builder/templates/templateV2023.go
Additional comments (Suppressed): 9
github/pool.go (9)
  • 43-49: The buffer size of queueCh has been increased from 1 to 100. This change allows more concurrent operations, which can improve performance. However, it also increases memory usage. Ensure that the system has enough memory to handle this increase.
    [APPROVE]

  • 69-74: The error handling has been improved by introducing the appendErr method. This method locks the mutex, appends the error to the errs slice, unlocks the mutex, and calls wg.Done(). This is a good practice as it encapsulates the error handling logic and makes the code cleaner.
    [APPROVE]

  • 79-89: The error handling logic has been replaced with the appendErr method. This change makes the code cleaner and more maintainable.
    [APPROVE]

  • 98-103: The error handling logic has been replaced with the appendErr method. This change makes the code cleaner and more maintainable.
    [APPROVE]

  • 109-114: The error handling logic has been replaced with the appendErr method. This change makes the code cleaner and more maintainable.
    [APPROVE]

  • 128-152: The error handling logic has been replaced with the appendErr method. This change makes the code cleaner and more maintainable.
    [APPROVE]

  • 170-174: The appendErr method is a good addition. It encapsulates the error handling logic, making the code cleaner and more maintainable.
    [APPROVE]

  • 182-188: The errors method now returns a copy of the errs slice instead of the slice itself. This change ensures thread safety by preventing external modifications to the errs slice.
    [APPROVE]

  • 190-195: The moduleinfo method now returns a copy of the modinfo slice instead of the slice itself. This change ensures thread safety by preventing external modifications to the modinfo slice.
    [APPROVE]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
B-performance Bug: performance issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant