Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow registering function-based entrypoints for mappings #235

Open
slint opened this issue Nov 18, 2024 · 0 comments · May be fixed by #234
Open

Allow registering function-based entrypoints for mappings #235

slint opened this issue Nov 18, 2024 · 0 comments · May be fixed by #234
Assignees

Comments

@slint
Copy link
Member

slint commented Nov 18, 2024

Currently we only support registering mappings via Python module "paths" that in turn contain all the mappings in raw JSON, following a certain directory hierarchy to assist with alias definition.

Motivation

This method has surfaced several problems over time:

  • Mapping definitions can get very large
    • "Native" composition capabilities for indices/mappings are not great at the moment
      • Index and component templates can be complex to reason about in terms of priorities and what the resulting mapping will be (the only way to find out, is by trying to create an index and see the result)
      • They're mostly focused on logs and monitoring storage use-cases
  • Very little reuse of existing logic, which ends in inconsistent definitions for the same type of objects across different indices (e.g. an organization/affiliation object looks different when used in a bibliographic record vs. an authority record, even though they share the same structure and types).
  • Difficult to implement "dynamic" mappings
    • Features like Dynamic templates can solve some use-cases, but they're also difficult to manage.
    • Implementing features that rely on config flags isn't possible, since any indexed fields have to be defined in the centrally managed JSON mappings.
  • No support for comments in JSON

Proposal

Allowing function-based entrypoints (similar to e.g. to the blueprint_loader) that e.g. return plain Python dictionaries for the mappings, would solve several of the above problems:

  • Mappings can be programmatically composed using Python code. That could be achieved
  • Configuration variables can be taken into account to include/exclude certain fields
@slint slint linked a pull request Nov 18, 2024 that will close this issue
@slint slint moved this to In progress in Sprint Q1/2025 Nov 18, 2024
@slint slint self-assigned this Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready
Development

Successfully merging a pull request may close this issue.

1 participant