FR: Easy way to install Beets plugins

[edgar-vincent]

Hi!

Thanks a lot for this brilliant program. It really is the best frontend to Beets so far!

I use a couple of extra plugins for Beets, and I installed them in beets-flask by building a new Docker image like this:

FROM pspitzner/beets-flask:latest

USER root
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install beets-copyartifacts3 python3-discogs-client

COPY --from=pspitzner/beets-flask:latest /repo/entrypoint_fix_permissions.sh /repo/
COPY --from=pspitzner/beets-flask:latest /repo/entrypoint.sh /repo/

ENTRYPOINT ["/bin/sh", "-c", "./entrypoint_fix_permissions.sh && su beetle -c ./entrypoint.sh"]

However, it might be easier if one could simply provide a list of plugins, referenced by their pip package name, via an environment variable, that would be installed when the container is started. What do you think?

Thanks a lot,

EV

[pSpitzner]

Good suggestion!

Are plugins working well for your custom build? we haven't tested a lot of the plugins beyond Spotify, therefore adding a way to enable more of them was not important.

But you are right, this is easy to add. What about a pip requirements file in the config directory, that the user can place and pip installs after launch?

@semohr, what do you think?

[edgar-vincent]

Thank you for your reply.

I only use the discogs, copyartifacts and convert plugins so far, and they seem to be working fine.

A pip requirements file would probably the cleanest solution, albeit possibly not the easiest one to use, but perhaps I'm being a nit-picker :)
I don't know much about Python project building, but would a Docker environment variable do?

Anyway, it is not something that will need to be changed very often, so either solution would be fine!

Thanks again!

[semohr]

In my opinion, we should parse the beets config file and install all enabled plugins. This would remove one point of failure from a user's perspective. One needs to restart the container anyways for config changes to take effect so this seems like a reasonable solution. I.e. on startup install all enabled plugins if not existing yet.

I think this should be the desired and most user-friendly behavior. Shouldn't be too hard to implement, maybe a bit more difficult than a requirements file but still doable.

Not sure if this could yield some problems down the line with different plugin versions but we will cross that road once we run into an issue.

FYI: Am also using some plugins in my setup but I just start a terminal in the container and install them manually atm. Would be a nice thing to have for me too :D

[edgar-vincent]

That would be ideal, but how should Python dependencies by dealt with in the scenario you describe?
Some plugins are built-in, other are developed by third parties (and thus need to be installed). Some require extra Python dependencies, some don't.

[semohr]

I guess it could work as follows. On startup we do the following:

1. Ready config yaml file and get list of all plugins
2. Match plugins to dependencies
   • We need a mapping from plugin name to dependencies for that
   • Will take some time but we can find all plugins in the beets documentation
3. Install latest version of plugin

Additionally, we could still support a requirements file for custom plugins.

In code this could look like the following

@dataclass
class Dependency:
    name: str
    url: str
    thrid_party: bool = False

# TODO
plugins_to_dependencies = dict()

def get_plugins_from_yaml(file: Path) -> list:
    if not Path(file).exists():
        return []
    with open(file, "r") as f:
        plugins = yaml.safe_load(f).get("plugins", [])
    return plugins

if __name__ == "__main__":
    if beets_dir is None:
        log.warning("BEETSDIR not set! Skipping auto plugin installation...")
        exit(1)

    config_path = Path(beets_dir) / "config.yaml"
    plugins = get_plugins_from_yaml(config_path)

    for plugin in plugins:
        dep = plugins_to_dependencies.get(plugin)
        if dep is not None:
            # TODO
            # Check if is installed e.g. with importlib
            # Install
            pass
        else:
            log.debug(f"Dependency for {plugin} not required! Skipping...")

[pSpitzner]

Id like having both,

• requirements.txt (simple, robust, and you can fix versions)
• auto-detect if not in requirements.txt. Convenient but needs the mapping, and we should take care that our auto-magic does not get in the way of you want a bit more control!

Also, some plugins need extra deps not available via pip, like bpm and keyfinder.

[semohr]

Only using a requirements.txt will not be enough than. I just checked and for instance the keyfinder plugin needs https://github.com/mixxxdj/libkeyfinder which one would need to build manually.

How about:

• startup.sh (if exists this is run first)
• requirements.txt (if exists install deps from here)
• auto

[pSpitzner]

Hey @edgar-vincent,

we just pushed another version to latest (including both, the parenthesis fixes and plugin additions)
and added some docs for it: readthedocs

Could you give this a try? There were some changes in the Dockerfile, would be great if you can confirm that nothing obvios broke!

[lucapolesel]

Seems to work fine even tho it may be a bit difficult for not so experienced users.
Would it be such a bad idea to include all documented plugins (the ones from the main beets documentation) like linuxserver's container? (Or to include their dependencies at last so the user can always pull the new version of the plug-in at startup like it is right now).
startup.sh and requirements.txt can still be useful for those people that may want to import their own plugins.

startup.sh for Chromaprint/Acoustid:

#!/bin/sh

# Build chromaprint
apk update
apk add \
    build-base \
    cmake \
    gstreamer \
    py3-gobject3 \
    ffmpeg

git clone https://github.com/acoustid/chromaprint.git
cd chromaprint
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TOOLS=ON .
make
make install

# Install fpcalc
wget https://github.com/acoustid/chromaprint/releases/download/v1.5.1/chromaprint-fpcalc-1.5.1-linux-x86_64.tar.gz

tar -xzf chromaprint-fpcalc-1.5.1-linux-x86_64.tar.gz \
    --strip-components=1 \
    -C /usr/local/bin \
    chromaprint-fpcalc-1.5.1-linux-x86_64/fpcalc

requiremenents.txt (including other plugins I need):

beets[fetchart,embedart,lastgenre,lyrics,chroma,discogs,plexupdate]

Thank you for the cool project!

[semohr]

I agree it is not the most user friendly option. This issue will stay open until we add the automatic installer.

In my opinion we should not include dependencies that are not used by a majority of people.

Should be relatively straight forward to implement (e.g.). The most time consuming part would be to add the dependency mapping or think of a better way to do this in general.
Someone of you want to give it a shot? Otherwise we will come back to it once we have migrated to quart and polished the frontend a bit. Maybe some weeks, depends how much time we can free up.

[semohr]

FYI (keep in mind we also ship the files for the front-end and have some additional dependencies)

docker images lscr.io/linuxserver/beets:latest --format "{{.Repository}}:{{.Tag}} -> {{.Size}}"
lscr.io/linuxserver/beets:latest -> 602MB
docker images ghcr.io/pspitzner/beets-flask:latest --format "{{.Repository}}:{{.Tag}} -> {{.Size}}"
ghcr.io/pspitzner/beets-flask:latest -> 361MB

I think 361MB is way too much for us too.

[edgar-vincent]

Thanks a lot for this!

I think the docs could be a bit clearer. For non-developers, it is unclear which of requirements.txt or startup.sh should be used (the first being for Python dependencies, the second for Python as well as non-Python dependencies).

I agree that shipping too many plugins and dependencies would cause the image to be too big. However, since dependencies installed when the container starts aren't cached, startup times could be high.

Perhaps you could provide two images, one without any plugins and the second with all the built-in plugins as well as their dependencies? That way, users who don't need any plugins wouldn't be impacted, as they would just pull the default image, while those who do would just pull their images from a different tag. All that with the requirements.txt and custom.sh logic still in place, for users who want to use third party plugins.

I fear that mapping dependencies in order to auto-install plugins could be a huge task, which would be prone to break, but you know better. I thought about parsing the source of this section of the official Beets documentation: https://beets.readthedocs.io/en/stable/plugins/index.html#other-plugins, but some links point to a repo, and others to a single file, so I'm not sure how this could be automated without some kind of plugin repository with serialised recipes describing how to install each plugin. But that seems to me like a completely different endeavour, perhaps not in the scope of this project.

Again, I'm not a developer, so I'm just giving my Philistine opinion :)

[semohr]

Creating a "full" optional image sounds like an excellent compromise.

I was more thinking about auto installing python only dependencies (they are mostly on pypi) and add a supported list of plugins to the docs. Shouldn't be too bad to maintain. Of course the startup.sh and requirements.txt will still be around.

[pSpitzner]

+1 one a -with-plugins tag.

Have to look into how to do this, but I guess via another layer in the Dockerfile.

Thanks for the link to Linux server, this is a great reference to include everything. I honestly prefer including everything over mapping out the config and installing on run (seems more robust and easier to maintain)