Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up initial clone with --filter=blob:none #467

Open
fsackur opened this issue Mar 14, 2025 · 3 comments · May be fixed by #468
Open

Speed up initial clone with --filter=blob:none #467

fsackur opened this issue Mar 14, 2025 · 3 comments · May be fixed by #468
Assignees
Labels
cleanup Low impact changes

Comments

@fsackur
Copy link

fsackur commented Mar 14, 2025

On my box, cloning the tools repo and running make update takes 45 minutes. (My dev box is terrible and its on a terrible LAN.)

By adding --filter=blob:none, I've reduced that to 22 minutes.

Yes, it's twice as fast.

The tradeoff is that some git operations will take longer, as blobs will be downloaded on demand (e.g. doing git bisect or checking out earlier commits).

However, I expect this cost will never be incurred for the majority of workflows.

For workflows that dive into git history, I expect that the cost of fetching blobs on-demand will still be much less than the cost of fetching everything at clone time.

My understanding of the build system is very superficial, so I am open to having my assumptions corrected.

Ref: https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/

To implement, I think we just need to add the --filter=blob:none arg to the git_clone function in build/common.sh. I attach a PR to demonstrate: #468

@fsackur fsackur linked a pull request Mar 14, 2025 that will close this issue
@fichtner fichtner self-assigned this Mar 14, 2025
@fichtner
Copy link
Member

Just a question do you intend to build all from scratch? Partial updates are also an option with prefetching sets for kernel, base and packages even. If you even need an image...

@fichtner fichtner added the cleanup Low impact changes label Mar 14, 2025
@fsackur
Copy link
Author

fsackur commented Mar 14, 2025

Just a question do you intend to build all from scratch?

No, I'm all about writing a plugin. The reason I'm running make update is because it's on the way to make plugins PLUGINSLIST="devel/my_plugin".

I've already captured the repos into a VM template, so I personally don't need to clone again.

@fichtner
Copy link
Member

Ok I’m asking because you can use an OPNsense VM to build plugins just by cloning core and plugins into the VM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cleanup Low impact changes
Development

Successfully merging a pull request may close this issue.

2 participants