Does not search remote buckets

[jvtrigueros]

When running scoop-search on a bucket that I haven't added, it doesn't give suggestions like OG scoop search:

PS C:\Users\jvtrigueros> scoop-search fantasque
No matches found.
PS C:\Users\jvtrigueros> scoop search fantasque
Results from other known buckets...
(add them using 'scoop bucket add <name>')

'nerd-fonts' bucket:
    bucket/FantasqueSansMono-NF-Mono
    bucket/FantasqueSansMono-NF

PS C:\Users\jvtrigueros> scoop bucket add nerd-fonts
Checking repo... ok
The nerd-fonts bucket was added successfully.
PS C:\Users\jvtrigueros> scoop-search fantasque
'nerd-fonts' bucket:
    FantasqueSansMono-NF (2.1.0)
    FantasqueSansMono-NF-Mono (2.1.0)

[shilangyu]

Oh yeah, I forgot about this behavior of scoop search since I was testing on a machine that already has all buckets added. When I find time I will implement it. Thanks for reporting

[shilangyu]

For potential Hacktoberfest contributors:

To implement this feature you need to:

• Fetch all missing 'known' buckets see here
  • If there is no internet connection fail silently
• Conduct the search (this is already implemented)
• Print results (already implemented)
• Remove fetched buckets (ideally these would be fetched into windows' temporary directory)

See scoop's implementation to make sure you're printing exactly the same things.

Also, add new test cases to tests/integration.ps1.

[plicit]

I wanted to play with Go a bit more, so I implemented this, but I didn't like having to "git update" for every inactive repo on every failed search.

So, I looked at Scoop's implementation. Scoop uses the GitHub API to fetch a simple file list for each remote repo, but that limits us to roughly 60 requests per hour. I implemented it in go and then cached the results to avoid that limit, but I didn't care for it.

I thought that I might as well just download a repo zipball and cache that for the inactive buckets, so I can search the full manifests. So I implemented searching zip files.

Finally, I found rasa's bucket directory and decided to fetch and parse that on a failed search, since it not only includes inactive known buckets, but a lot of other buckets, too. So rasa is my current fallback with a 24 hour cache.

Anyhow, I ended up adding a bunch of ways to search bucket sources and refactored your code quite a bit in the process. I also changed the args and output to show a bit more, but that breaks tests/integration.ps1. FWIW, shovel has a different output format that shows more fields and is colorful, which might be worth mirroring.

I'm pretty much done with my experiments. Feel free to take what you like from my multisource branch. I can send a PR if you want, but I'm not sure you want it all, especially since it currently breaks identical output to Scoop and searches rasa, not just known buckets.

https://github.com/plicit/scoop-search/tree/multisource

[shilangyu]

@plicit forgive me for such a late response, your work is awesome. Refactor is also nice since it makes the code modular and more maintainable. The experiments you have done with efficient caching are very interesting, definitely a great improvement over scoops implementation (which is very susceptible to rate limits).

FWIW, shovel has a different output format that shows more fields and is colorful, which might be worth mirroring

Like you have said, "unfortunately" with scoop-search I am aiming at 100% compatibility with scoop itself, without any kind of improvements. This makes it a no-brain choice to replace existing scoop search without fearing that it might behave differently.

For a while I have been thinking about rewriting the whole scoop CLI. I find scoop to be way too slow, clunky (scoop's CLI args parsing is very unforgiving), has too little options, and it's output is rather meh. Building upon the existing ecosystem of standard manifests and buckets would be great since it would not force anyone to create new manifests just to work with this new tool. However I am no longer a windows user myself (heck, I wasn't using windows back when I first created scoop-search) so I don't have the motivation to do the rewrite. Initially scoop-search was created just out of frustration that every time I am forced to use windows, I eventually had to use scoop search and the wait for results to appear was unbearable.

So yeah, your work is very interesting, but unfortunately I won't be able to merge it into scoop-search any time soon

[plicit]

@shilangyu , Thank you so much for your kind words! :) No worries about the delay and not using my code -- especially since you're no longer using scoop or Windows.

I'll probably create a proper fork for my multisource version at some point, since I think people might find a binary release useful.

I agree about scoop's slow and clunky PowerShell CLI. I've similarly mused about replacing more functions of the CLI with Go or adding to their PowerShell version (or more invasive design changes), but scoop is in an odd place right now with Ash and the other scoop devs at odds and I'm not sure it's worth the investment.

Speaking of slow, every so often I get a slow response from my Go version of scoop-search. I profiled it using github.com/pkg/profile which traced the delay back to simply reading the file system, with most of the delay in cgocall, which isn't helpful.

I suspect that Windows security may be scanning files as I access them, since it sometimes seems like it has to do with where the scoop-search.exe binary is located. Perhaps that plagues the PowerShell version, too.

Anyhow, thanks again for your comment. Cheers :)

[shilangyu]

scoop-search being slow every so often is most likely related to either antivirus scans or NTFS caching, tho I have not investigated it closely. Either way it seems like something that cannot be avoided (maybe with some low level syscalls?) and windows related

[plicit]

Just to put a bow on my tangent for this issue, here is my fork with a binary release. Comments welcome! :)

https://github.com/plicit/scoop-search-multisource