Implement xlocate to sync with all the repos defined #231

Animeshz · 2022-03-20T13:33:22Z

Closes #230.

230, In short:

Currently, xlocate has hardwind path to inspect ${XLOCATE_REPO:=https://alpha.de.repo.voidlinux.org/xlocate/xlocate.git}. As a result, its currently difficult to search through all the active repositories in the system.

What it does:

Search for all the conf files defining the repository through /etc/xbps.d and /usr/share/xbps.d prioritizing the former.
Tries to pull the prebuilt indexes using the xlocate repo (at $XLOCATE_REPOS[i]) if exists. Or else use xbps-query --repository=<repo> -Mio to resolve all the subrepo (like multilib/nonfree) or the custom repos at local path(s).
Resolves common packages by priority order of the conf files, i.e. if same package is present in both, then the higher priority (e.g. 00-repository.conf > 01-repository.conf) is preferred to keep the package's file list.
Commits for each of the repository.
If any repository in middle is disturbed at any time, then it drops all the commits below and equal to that repo (in priority order), so that the common package file list gets preference by this repo over the other (lower priority) if added in this repo this time.

This makes -g obselete, so is removed. Shebang is changed to bash to utilize arrays and [[ checking builtin over the test command [.

Animeshz · 2022-03-20T15:29:28Z

I'm not sure about formatting guidelines, lemme know if any error is there regarding file formatting, max-line lengths etc.

Also lemme know if we need to force generate a new indexes for a remote repository (instead of syncing with xlocate.git, even if available)?

Duncaen · 2022-03-20T18:12:58Z

Using xbps repos is wrong, you can't assume that /xlocate is there.

The only sane way to support multiple repos is to allow the user to specify the git repos.

Duncaen · 2022-03-20T18:17:42Z

* Search for all the conf files defining the `repository` through `/etc/xbps.d` and `/usr/share/xbps.d` prioritizing the former.

Your implementation is broken, repository files can include multiple repositories. xbps-query -L should be used.

This makes -g obselete, so is removed. Shebang is changed to bash to utilize arrays and [[ checking builtin over the test command [.

So how are we supposed to create the actual index?

Animeshz · 2022-03-20T18:49:26Z

Using xbps repos is wrong, you can't assume that /xlocate is there.

Yes we're checking if there exists a git repository by git ls-remote, otherwise will fallback with xbps-query as usual :)

repository files can include multiple repositories.

Oh didn't knew about that, I thought the files are just shell scripts defining shell variables, like /etc/os-release, as they had = without spaces. xbps-query -L should be used I'll make sure to check about that.

So how are we supposed to create the actual index?

Oh if we mean about the official remote repository, then I was also worried about that post writing the changes, also stated in Also lemme know if we need to force generate a new indexes for a remote repository (instead of syncing with xlocate.git, even if available)?. I'll make sure to provide a better solution about that tomorrow 👀

Animeshz · 2022-03-20T18:54:21Z

What is number in front of result of xbps-query -L? I didn't found any mentions in the man page 👀

Duncaen · 2022-03-20T18:58:10Z

What is number in front of result of xbps-query -L? I didn't found any mentions in the man page eyes

     -L, --list-repos
         Lists repositories and the number of packages contained on them. If a
         repository is not available the number of packages will be -1.  The
         -v option can be used to show more detailed information of remote
         repositories.

Yes we're checking if there exists a git repository by git ls-remote, otherwise will fallback with xbps-query as usual :)

I'm talking about the urls, you can't assume the path of xlocate, xbps repositories are structured differently and mirros might put everything in more directories.

Animeshz · 2022-03-20T19:07:46Z

Thank you!

I'm talking about the urls, you can't assume the path of xlocate, xbps repositories are structured differently and mirros might put everything in more directories.

Um, but those repository don't have prebuilt xlocate indexes attached to them (or I'm missing something), a musl package may contain some more or maybe less file(s), their prebuilt indexes aren't in https://alpha.de.repo.voidlinux/xlocate/xlocate.git (main/glibc indexes) I assume. I think we'll use xbps-query -Mio for building their index in that case. Correct me if there's anything wrong.

Duncaen · 2022-03-20T19:11:33Z

How do you come from https://alpha.de.repo.voidlinux/current or https://alpha.de.repo.voidlinux/current/musl to https://alpha.de.repo.voidlinux/xlocate and from https://some-mirror.com/voidlinux/current or https://some-mirror.com/voidlinux/current/musl to https://some-mirror.com/voidlinux/xlocate.

The main point is that its not really possible to get the repo and/or assume where the xlocate directory is located it.
It would be a lot simpler to just have the user specify the xlocate git remotes and then use them, instead of trying to come up with some scheme that assumes a bunch of things about configured repositories.

Animeshz · 2022-03-20T19:14:39Z

By replacing /<last-part-of-url> with /xlocate/xlocate.git 👀 I don't think there's a https://alpha.de.repo.voidlinux/current/xlocate/xlocate.git so it should just move on to xbps-query for the /musl or other subrepo?

Duncaen · 2022-03-20T19:20:21Z

Thats the point, that doens't work for musl while it should work, and you can't replace everything since that would break mirrors.

There is no point in trying to guess xlocate repos or enforcing some directory layout if the user just configures them.

so it should just move on to xbps-query for the /musl or other subrepo?

No, it should work like it currently does, simply don't use xbps repositories to guess locations of xlocate repositories.

Animeshz · 2022-03-20T19:27:43Z

Ok I had some misconceptions lately, the https://alpha.de.repo.voidlinux/xlocate/xlocate.git is xlocate repo only for glibc x86_64 right?

If it does we should just be able to pull from the guess (saving lot of cpu work), if not we just used xbps-query to build the indexes locally for rest of custom repo / sub repo, so we can search through all of it?

Otherwise -g currently does the same except it just delegates everything to xbps-query, taking much time and cpu to build full index of both large official repo and possibly small custom repo.

Duncaen · 2022-03-20T19:30:49Z

Ok I had some misconceptions lately, the https://alpha.de.repo.voidlinux/xlocate/xlocate.git is xlocate repo only for glibc x86_64 right?

Its generated from x86_64-glibc, but that doesn't matter, we use it for every architecture and libc.

Otherwise -g currently does the same except it just delegates everything to xbps-query, taking much time and cpu to build full index of both large official repo and possibly small custom repo.'

Right, and we still need that since that is used to generate /xlocate on the repository server.

Animeshz · 2022-03-21T07:18:52Z

Its generated from x86_64-glibc, but that doesn't matter, we use it for every architecture and libc.

So if we club index of all repositories (including subrepos) coming from a certain base url into one single commit, will it be better?

Duncaen · 2022-03-21T16:14:16Z

xlocate already contains all repos, I don't get the purpose of the PR, you can't derive the xlocate git repo from the repository urls and there are no other xlocate git repo to even add. If you want to have xlocate on you own repo then it would be simpler to allow to add additional git repositories urls to xlocate than trying to assume things based on configured xbps repos.

Animeshz · 2022-03-26T08:35:34Z

Is it better now, -g and -S will work as they were, with added allowance of defining multiple-repos with any of

xlocate git repository url
or local xbps repo path like hostdir/binpkgs
or any remote xbps repository say mirrors

at env variable, and if one/more argument supplied to -g, it will build index only of that repo(s) instead of all.

Example:

It generates commit as:

  xtools on  master >>> git -C $HOME/.cache/xlocate.git log
commit c738d2ec1c86147e6be1fae1abf8d123158e9e90 (HEAD -> master)
Author: xupdategit <xupdategit@none>
Date:   Sat Mar 26 15:25:26 2022 +0530

    xupdategit: /home/animesh/Projects/void-packages/hostdir/binpkgs/xpack

commit 7cd8eb515eec512b04cd7aae29341dd4876dfb75
Author: xupdategit <xupdategit@none>
Date:   Sat Mar 26 15:25:16 2022 +0530

    xupdategit: https://alpha.de.repo.voidlinux.org/xlocate/xlocate.git

for each of the repository specified in the env variable. It also keeps track of the remote updates are present by hashing *-repodata for local repo or remote xlocate git's HEAD hash. As such if repo corresponding to 2nd commit has changes, it'll only update 2, 3, 4... and the first should be unaffected, making operation efficient to what's needed to be generated/pulled.

…nt variable to get it synced, adds optional repository argument to -g if present restrict to generate xlocate indexes only for the given repository(ies) rather than all the repositories defined in the system

Animeshz force-pushed the master branch from 4047c61 to 5037c52 Compare March 26, 2022 08:26

Animeshz force-pushed the master branch from 5037c52 to b649d6b Compare March 26, 2022 09:36

Allow multiple XLOCATE and XBPS repos to be supplied to the environme…

8f0cbd5

…nt variable to get it synced, adds optional repository argument to -g if present restrict to generate xlocate indexes only for the given repository(ies) rather than all the repositories defined in the system

Animeshz force-pushed the master branch from b649d6b to 8f0cbd5 Compare March 26, 2022 09:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement xlocate to sync with all the repos defined #231

Implement xlocate to sync with all the repos defined #231

Animeshz commented Mar 20, 2022 •

edited

Loading

Animeshz commented Mar 20, 2022

Duncaen commented Mar 20, 2022

Duncaen commented Mar 20, 2022

Animeshz commented Mar 20, 2022

Animeshz commented Mar 20, 2022

Duncaen commented Mar 20, 2022 •

edited

Loading

Animeshz commented Mar 20, 2022 •

edited

Loading

Duncaen commented Mar 20, 2022 •

edited

Loading

Animeshz commented Mar 20, 2022 •

edited

Loading

Duncaen commented Mar 20, 2022 •

edited

Loading

Animeshz commented Mar 20, 2022

Duncaen commented Mar 20, 2022

Animeshz commented Mar 21, 2022

Duncaen commented Mar 21, 2022

Animeshz commented Mar 26, 2022 •

edited

Loading

Implement xlocate to sync with all the repos defined #231

Are you sure you want to change the base?

Implement xlocate to sync with all the repos defined #231

Conversation

Animeshz commented Mar 20, 2022 • edited Loading

Animeshz commented Mar 20, 2022

Duncaen commented Mar 20, 2022

Duncaen commented Mar 20, 2022

Animeshz commented Mar 20, 2022

Animeshz commented Mar 20, 2022

Duncaen commented Mar 20, 2022 • edited Loading

Animeshz commented Mar 20, 2022 • edited Loading

Duncaen commented Mar 20, 2022 • edited Loading

Animeshz commented Mar 20, 2022 • edited Loading

Duncaen commented Mar 20, 2022 • edited Loading

Animeshz commented Mar 20, 2022

Duncaen commented Mar 20, 2022

Animeshz commented Mar 21, 2022

Duncaen commented Mar 21, 2022

Animeshz commented Mar 26, 2022 • edited Loading

Animeshz commented Mar 20, 2022 •

edited

Loading

Duncaen commented Mar 20, 2022 •

edited

Loading

Animeshz commented Mar 20, 2022 •

edited

Loading

Duncaen commented Mar 20, 2022 •

edited

Loading

Animeshz commented Mar 20, 2022 •

edited

Loading

Duncaen commented Mar 20, 2022 •

edited

Loading

Animeshz commented Mar 26, 2022 •

edited

Loading