Future plans

[DerGoogler]

Hey @IzzySoft, is there any plan that you'll use the new MRepo cli? Nothing has really changed — no categories, anti features etc.

What are your future plans?

[IzzySoft]

Thanks for reaching out, @DerGoogler – and apologies for 1) not having reported back earlier and 2) not even having had time to work on this…

With Sanmer's repo-util being archived now, this certainly moved up in priority on my roadmap. Quite busy with a few other tasks (yes, the same as before), so it will still take a little until I can get to this. Until now, apart from being quite busy, one thing holding me back was concerns about "future compatibility" with updates from Sanmer – concerns which are now rather moot.

As you ask, let me ask one thing back: Will you keep this very repo-util here maintained? Then I'd switch to your fork here – which would be much easier and faster than starting with an entirely different repo-builder, and thus could probably be accomplished faster and earlier. I have no ETA yet, but the "main task" keeping me occupied at the moment will hopefully be accomplished in a few weeks, giving me a break to take care for switching to your fork here. I might even find an earlier slot then, given it would probably work as drop-in replacements and the new fields could be added to the modules one-by-one later.

[DerGoogler]

The fork is stable. Fields like antifeatures, icon are placed inside tracks.json

This is how a module looks like

{
      "id": "acp",
      "name": "Audio Compatibility Patch",
      "version": "v2.5",
      "versionCode": 39,
      "author": "zackptg5, ahrion, John Fawkes",
      "description": "Fixes music and streaming apps (Spotify, Pandora, etc) that aren't processing audio effects for various equalizer applications through the modification of audio policy",
      "added": 1445340430.7142262,
/*
     "lastupdated": "2023-12-28",
     "lastupdated_ts": 1703786679,
     "lastbuild": "2023-12-28",
     "added_ts": 1703840636.160768,
*/
      "timestamp": 1643646579.0,
      "track": {
        "type": "GIT",
        "added": 1445340430.7142262,
        "license": "",
        "homepage": "",
        "source": "https://github.com/Magisk-Modules-Repo/acp.git",
        "support": "https://github.com/Magisk-Modules-Repo/acp/issues",
        "donate": "",
        "verified": false,
        "cover": "",
        "icon": "",
        "screenshots": [],
        "category": "",
        "categories": [
          "Audio Enhancements",
          "Advanced Audio Mods",
          "App Additions and Features"
        ],
        "readme": "https://raw.githubusercontent.com/Magisk-Modules-Repo/acp/master/README.md",
        "require": [
          "aml"
        ],
        "antifeatures": []
      },
      "versions": [
        {
          "timestamp": 1643646579.0,
          "version": "v2.5",
          "versionCode": 39,
          "zipUrl": "https://gr.dergoogler.com/gmr/modules/acp/v2.5_39.zip",
          "changelog": ""
        }
      ]
    }

I would like to add the commented out properties to. Do you agree with that?

[IzzySoft]

Sure! I'm always open to new fields, especially those filled automatically from existing details (like here pulling it up from the separate versions for the summary). My own scripts ("repo browser") should simply ignore fields they don't know about – and the more details are available on request for those browsing the modules e.g. in MMRL, the better! You might remember (or having seen) that I failed with such suggestions on the original repo.

Just wondering: that added_ts differs from added in the above example is not what it finally will be, right?

If you think it makes sense for the MMRL app to also have a "short description" (speaking "fastlane terms" here), you'd run through widely open gates with me. I'd happily implement that on my end then, given time and possibility (there's no such thing with the modules.prop or anything – but I could imagine setting short_description or summary here myself, by "extracting" it from the longer one). Might be helpful with lists. Just a suggestion, not a "request" (as I'm short on time to work on these things myself currently, how could I 😜)

[DerGoogler]

Sure! I'm always open to new fields, especially those filled automatically from existing details (like here pulling it up from the separate versions for the summary). My own scripts ("repo browser") should simply ignore fields they don't know about – and the more details are available on request for those browsing the modules e.g. in MMRL, the better! You might remember (or having seen) that I failed with such suggestions on the original repo.

Just wondering: that added_ts differs from added in the above example is not what it finally will be, right?

If you think it makes sense for the MMRL app to also have a "short description" (speaking "fastlane terms" here), you'd run through widely open gates with me. I'd happily implement that on my end then, given time and possibility (there's no such thing with the modules.prop or anything – but I could imagine setting short_description or summary here myself, by "extracting" it from the longer one). Might be helpful with lists. Just a suggestion, not a "request" (as I'm short on time to work on these things myself currently, how could I 😜)

added_ts and added are no difference, same as lastupdated_ts and timestamp

If you want to open a PR, do it!

[DerGoogler]

@IzzySoft. I've updated some things 😄

User can now create a file in their module called common/repo.json:

{
  "support": "str",
  "donate": "str",
  "cover": "str",
  "icon": "str",
  "license": "str",
  "homepage": "str",
  "screenshots": ["array"],
  "category": "str",
  "categories": ["array"],
  "antifeatures": ["array"],
  "require": ["array"]
}

This allows the devs to manage their module without opening any pr. Every property that doesn't exist will be a null value

[DerGoogler]

I've adde a sib command to generate sitemaps

cli.py sitemap --base-url "https://..."

[DerGoogler]

@IzzySoft, the development of MRepo has been completely stopped, even the new repo util.

I don't plan to fork the new repo util, the current does it job perfectly.

[IzzySoft]

As mentioned in the issue that just linked itself here (see above this comment), I'm now switching to your fork (finally 🙈). Having some questions there:

I'm now in the process of moving the "extra fields" for track.json (which Sanmer didn't want to implement and I thus had placed into iod_track.json) over to the corresponding track.json (will have to duplicate that for a little until I've adjusted my framework for the repo browser), and will also see to fill the other "gaps" (new and empty fields) wherever possible. Checking the modules.json generated, I wonder about some "duplicated fields" there – that is, fields that exist in the "top level" of each module (but stay empty there) as well as in modules[]->track. For example: license. I've set that in track.json and it shows up in track, but the license field parallel to track stays empty. Can I ignore that? Are those "global fields" accidental duplication, or what purpose do they have (guess they're supposed to come from the new repo.json, see below)? I also see the "global" support field has the value of track->source instead of track->support, while there seems to be no "global" support item. Am a little confused now.

    {
      "id": "73sydney.coloursatmod",
      "name": "Colour And Saturation Modifier",
      "version": "1.1",
      "versionCode": 1,
      "author": "adrianmmiller (73Sydney)",
      "description": "A Colour and Saturation Modifier Module - Default Settings for Pixel 6 Pro as per Justarandomguy's post at: https://forum.xda-developers.com/t/root-tune-saturation-white-balance-and-enable-hbm-on-your-pixel.4575403/"
,
      "added": 1703840636.160768,
      "timestamp": 1703786679.0,
      "size": 15144,
      "maxApi": null,
      "minApi": null,
      "category": null,
      "categories": null,
      "icon": null,
      "homepage": null,
      "donate": null,
      "support": "https://gitlab.com/adrian.m.miller/coloursatmodifier",
      "cover": null,
      "screenshots": null,
      "license": null,
      "readme": null,
      "require": null,
      "verified": false,
      "note": null,
      "track": {
        "type": "ONLINE_JSON",
        "added": 1703840636.160768,
        "license": "GPL-2.0-or-later",
        "homepage": "",
        "source": "https://gitlab.com/adrian.m.miller/coloursatmodifier",
        "support": "https://gitlab.com/adrian.m.miller/coloursatmodifier/-/issues",
        "donate": "",
        "verified": false,
        "cover": "",
        "icon": "",
        "screenshots": [],
        "category": "System",
        "categories": [],
        "readme": "",
        "require": [],
        "antifeatures": []
      },

I've only switched locally for now, didn't put it online yet – waiting for your advice concerning this confusion on my end 🙈 After all, it should go live correctly – and not with some "quirks" added by me due to misunderstanding. Will continue local update as soon as I know what to change and how, and push it online once "clean" and "tested" (that my local framework still can deal with it fine – only tested index generation so far).

PS: love you even thought about a "notes" field there 🤩 Guess that's intended for "maintainer notes"?

And one more question: the readme describes a common/repo.json (you also mention that above) that should be placed into the module's dir. Would that then be modules/moduleName/common/repo.json (why the additional subdir?) – or modules/moduleName/repo.json? And what's the purpose of this extra JSON that those details are not simply included with track.json?

This allows the devs to manage their module without opening any pr.

Ah. So that file would be found in the corresponding module's repo (maintained by the module's author), and needed to be "sync'd" to the Magisk repo using repo-util? How is that sync supposed to happen? I see no field in track.json pointing to a potential repo.json to pull/sync. Further, how are conflicts handled (e.g. a different license being specified in track.json and repo.json) by the client (and supposed to be handled by my repo browser)?

command to generate sitemaps

Will have to look at that later, once the general change is complete, thanks! One step at a time 😉

(PS: as you're in Germany like I am: we could have a chat on the phone if you like. Contact info can be found in the imprint of my business site. Just give me a call – or send me your number and I call you 😉)

[DerGoogler]

     "size": 15144,
      "maxApi": null,
      "minApi": null,
      "category": null,
      "categories": null,
      "icon": null,
      "homepage": null,
      "donate": null,
      "support": "https://gitlab.com/adrian.m.miller/coloursatmodifier",
      "cover": null,
      "screenshots": null,
      "license": null,
      "readme": null,
      "require": null,
      "verified": false,
      "note": null,

And one more question: the readme describes a common/repo.json (you also mention that above) that should be placed into the module's dir. Would that then be modules/moduleName/common/repo.json (why the additional subdir?) – or modules/moduleName/repo.json? And what's the purpose of this extra JSON that those details are not simply included with track.json?

This part can configure user with a file called common/repo.json. modules[]->license, modules[]->readme etc should always preferred over modules[]->track->license etc.
common/repo.json is by/on the remote. you don't have to maintain it

PS: love you even thought about a "notes" field there 🤩 Guess that's intended for "maintainer notes"?
Available colors are red, blue, green and yellow. It also supports titles

[IzzySoft]

Wait…

This file should be placed in the modules root directory

is maybe ambiguous then. I understood it as a file in the same modules dir as track.json – but reading between the lines, you probably rather mean

This file should be placed in the root directory of the module's repository (and if exists, is fetched together with the ZIP)

Do I get that right? In that case…

modules[]->license, modules[]->readme etc should always preferred over modules[]->track->license etc.

The former would be the ones coming from the module's repo, right? So what if the maintainer of the "index repo" (like me) needs to override something? For example, categories. We might end up with a mess of "custom categories" if multiple module authors make up their own (I've seen the categories devs put with their RFPs at F-Droid). I'm also not sure what note is supposed for then (especially colored ones), as the readme doesn't detail on it – and your comment only mentions what colors are available…

A few more questions if you permit, as those fields are not described in the readme (or in case of category and categories are not even present):

• verified is "bool", OK – and means what?
• categories is an array, or a comma-separated list?
• translation is not an existing field I guess? Was meant for things like Weblate, Crowdin, Transifex. Not that I remember having seen any module using such, so no prob if it's missing. Just asking as I had that in my iod_track.json. Can as well drop it.
• What is cover intended for? And where are things like cover, icon and screenshots expected to reside? Are those relative URLs to the modules in the "index repo", or rather absolute URLs pointing "anywhere" (and thus could be misused for e.g. tracking)?
• similarly, is readme supposed to be a file in the "index repo", a relative path of the same, or a full URL or relative path in the module's repo?
• what is supposed to go to require?

I guess if I just move the fields from my iod_track.json (antifeatures and category, if there's no translation field) over to track.json, the resulting index should be similar to what it was before, except for the new (and empty) fields from common/repo.json? I still wonder why modules[]->support has the value of modules[]->track->source – it cannot come from common/repo.json in the above example as the module's repo does not have that file (nor has mine). So maybe it's copied incorrectly, or should not have been copied at all?

[DerGoogler]

You just showed me that it need some additional work. Like categories can be misused if it is not validated.

Also a additional process to handle relative urls when using cover, icons, screenshots etc.

readme is supposed to show instructions of how module works or something else. Should be a url, I'll implant a relative variant over the days.

require is meant if a module is depending on a other module, like Systemless Node.js requires also Systemless Mkshrc

The repo doesn't support currently translation :(

[IzzySoft]

Thanks! I just took a look at your repo (the modules.json as well as the module's track.jsons). Here's what I gathered so far. If that's correct, and you want me to, I could make a PR to update the Readme with the details:

track.json

• categories: array of strings
• category: probably not needed (picks the first?)
• readme: full path to readme.md
• require: needs other module(s) (array)
• cover: full URL to (raw) img (svg, png, jppg) – corresponds to featureGraphic
• icon: same
• screenshots: same, but array of strings
• verified: true/false (though still not sure what is verified here; maybe feedback of the module working and doing what it should?)

other files

• sitemap.xml: collection of module URLs (from the repo), each with its full URL ("loc") and last modified ("lastmod")
• commons/repo.json is extracted from the module's ZIP file if it exists there

---

additional process to handle relative urls when using cover, icons, screenshots etc.

Yupp, that'd be great! Not sure what you've planned there, but I'd say relative URLs should be handled as relating to the "index repo" then?

The repo doesn't support currently translation :(

No prob, we can skip that (at least for now). I've just grep'd all the iod_track.json files in my repo, not a single module has an URL in there. Unlikely to be used by modules I guess.

[DerGoogler]

verified is just to mark that the module has good quality and is well maintained.

I also opened a new branch. This currently includes whitelisted categories, track fields like, categories are now excluded and now just putted into modules[]->categories no matter if the categories in common/repo.json are changed. track.json can override common/repo.json to prevent spam and abuse.

[IzzySoft]

Thanks, sounds good! How is category handled should only categories be filled? Picks the first one?

[DerGoogler]

Thanks, sounds good! How is category handled should only categories be filled? Picks the first one?

category can also be added from track.json, but I think I'm removing category because categories can be used

[IzzySoft]

Thanks for letting me know (and good I waited) – so I rather fill the array then. You plan to remove category in the index altogether – or have it simply pick the first category? Just asking so I can keep that in mind when updating my framework later.

[IzzySoft]

OK, now updating all the track.json files. Let me know when you'd have time to check an index if everything seems fine. I could attach a locally built one here (would prefer to only "deploy" once confirmed).

[IzzySoft]

All track.json updated, some graphics added, index generation works fine. Ran a sync (which crashed with a known error from the original repo-util; fixed on my end and PR open if you want to adopt the fix), then ran through. Before I push the results to the server, I'll need to update the hint on which client to use (done locally).

You want to test the index before to make sure, or shall I just push it out, @DerGoogler?

[DerGoogler]

yeah @IzzySoft

[IzzySoft]

yeah = push or yeah = review first? 😉

[DerGoogler]

I'll review that later ;)

[IzzySoft]

OK, before asking again I read that as "push it now, I'll take a look later". Pushed then, new index is live.

I've also opened a PR fixing a minor bug (if some module does not define any versionCode: do not crash, but report that module and then skip/ignore it). I'd offer another one if you'd accept it – to fill in the gaps in the description for track.json with the README. Here's what I would suggest:

Key	Attribute	Type	Description
id	required	Str	Id of Module (in module.prop)
enable	required	Bool	Whether to enable updates
update_to	required	Str	Follow examples below
source	optional	Str	Url of where the source code lives
homepage	optional	Str	URL
readme	optional	Str	URL with e.g. description, instructions
changelog	optional	Str	Markdown or Simple Text (no HTML)
support	optional	Str	URL to issue tracker/support forum
donate	optional	Str	URL to donation page
cover	optional	Str	URL to cover image (featureGraphic)
icon	optional	Str	URL to icon.png (squared, max 512x512 px)
screenshots	optional	Str[]	URLs to screenshots of the module
license	optional	Str	SPDX identifier
antifeatures	optional	Str[]
category	optional	Str	category the module belongs to (deprecated)
categories	optional	Str[]	array of categories the module belongs to
require	optional	Str[]	list of module_ids this module depends on
verified	optional	Bool	if module has good quality and is well maintained
max_num	optional	Int	Overload MAX_NUM in config.json
versions	auto	Int	how many versions are present (do not touch!)

I'd also ask for 2 additional fields if I might, corresponding to short_description.txt and full_description.txt in Fastlane. The former would override what desc in module.prop gives, the latter go to the module's details page (which in my repo browser does not yet exist, as that field does not exist either).

And btw, discovered 2 modules in my collection which indeed provide a value for translation, as they use e.g. Crowdin to provide localization – so maybe when adding fields, adding this one too?

[DerGoogler]

Any PR is welcome

[IzzySoft]

OK, then I'll prepare one. Though doing that now would conflict with #3 which already changes the same README section. Any ETA when you will have that merged into main, so I can pull it and base my PR on that?

[DerGoogler]

There are currently no plans to merge that branch due private work reasons. Current plan it to complete this here at the end of 2024

[DerGoogler]

I also currently fight around the MMRL and MMRL-CLI. MMRL needs a deploy automation and MMRL-CLI needs many fixes

[IzzySoft]

OK, no prob. If it's OK with you, I'd then simply open a PR against main with my suggestions, and we see from there?

[DerGoogler]

Yes.

[IzzySoft]

Thanks! PR opened.

[DerGoogler]

@IzzySoft, I began to fork MRepo and will use it as the base for MMRL V4 (including the same license and credits)

Current dev builds: https://github.com/DerGoogler/MMRL-V4/releases

Supported repos: https://github.com/DerGoogler/MMRL-V4

[IzzySoft]

Apologies for my delayed answer, I was on vacation… You want me to pick something up or anything? Any action needed on my part at this point?

[DerGoogler]

@IzzySoft you don't have have to do anything. Just wanted to tell you that.

[IzzySoft]

Thanks!

[DerGoogler]

@IzzySoft One thing you can do. You'll need to upgrade the repo util

[IzzySoft]

Apologies again for the delay (seems I'm either on vacation, or drowning in tasks) – done now, and went easy & smooth! Also ran an index update, so new index is online as well.

[DerGoogler]

No worries!

[IzzySoft]

PS, just as another thanks: before you took over, I had occasional crashes on sync (which is why there's still no automated run). Since I switched to your fork, not a single one anymore, so I could finally consider setting up a cron job for regular sync runs and more timely updates 🤩

[DerGoogler]

@IzzySoft https://github.com/Googlers-Repo/magisk-modules-repo-util/releases/tag/v2.7.8

[IzzySoft]

Ah! That was already included when I updated, or should I update again now? (checks commits) Oh, the latter then. OK, let me pull… Done, thanks! Up-to-date again here.