Refactor module loader

[stijn-uva]

This is a more radical version of #425 that makes some significant changes to how configuration contexts are used and handled. TL;DR, in order of preference:

• Use self.config when in an object context to read settings
• Use the method argument config when in a constructor, or static or class method, to read settings
• Use the config imported from webtool when in a Flask view to read settings
• Use the global config variable to read settings otherwise. In principle this should only occur in two places in the code, bootstrap.py and Flask's __init__.py (there are currently a few other places where relying on injection would be too inconvenient)
• Do not pass around user objects. Instead pass around user-aware ConfigWrappers. Where you need to use User objects, pass a config as argument to their instructor so that they will read from their own configuration context (if desired).

The long version:

• Eliminate the passing of user objects as much as possible in favour of heavier usage of ConfigWrapper. This means that processors no longer need to be aware of users (with very few exceptions) - I think this is good because users are a front-end concept, not one the back-end should need to care about. Users were only ever really needed in the back-end to make sure the right configuration values were read.
• ConfigWrapper is used to create a configuration manager (reader) that reads configuration values taking into account a certain user and optionally a Flask request. This is then used to make sure that the right configuration tags are applied. The wrapping can happen in low-level parts of the code such as the main worker class, and the result can just be passed on to higher level functions such as processor process() etc. As long as a processor reads from self.config the right values should always be read.
• Because get_options(), validate_query() and is_compatible_with() are class methods they cannot access self.config. Instead they now have an argument config (replacing user) to which a reader or wrapped reader is passed. They then read from that for e.g. checking whether an API key is present or how many items can be processed.
• Instead of each set of views defining their own config as before, this is now done in the main Flask app, so that the front-end only ever instantiates a single (wrapped) configuration reader.
• In the backend, the configuration reader is instantiated as early as in bootstrap.py, and then passed all the way down to the individual modules where it is wrapped as needed. Thus in the backend there is also only a single reader that is wrapped in various ways as needed.
• ModuleCollector now also gets its config reader via injection rather than instantiating its own, to avoid some config managers being aware of module-defined settings and others not (this was the original goal of Refactor module loading v2: move writing of module_config.bin to bootstrap #425).
• If a processor ever still needs to access the user object, it is available via the ConfigWrapper().user property (which only exists for wrapped readers). This should be avoided but is necessary in some places e.g. the Telegram search worker which needs to behave differently when used by an anonymous user.

[dale-wahl]

I have been reviewing this and ran into an issue preventing logging in. That's cool that flask_login manages to update current_user with our User class, I am assuming, on login. But I am not sure how to handle this particular chicken/egg situation. Maybe I can give the not really a user current_user the config in the show_login function if it is a GET?

Fixed in below commits.

File "/usr/src/app/webtool/views/views_user.py", line 331, in show_login
... 
File "/usr/src/app/webtool/lib/template_filters.py", line 384, in inject_now
2024-11-11 16:32:09     "__notifications": current_user.get_notifications(),
2024-11-11 16:32:09   File "/usr/src/app/common/lib/user.py", line 447, in get_notifications
2024-11-11 16:32:09     raise ValueError("User not instantiated with a configuration reader. Provide a ConfigManager at "
2024-11-11 16:32:09 ValueError: User not instantiated with a configuration reader. Provide a ConfigManager at instantiation or use with_config().