Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/refactor sources #48

Merged
merged 49 commits into from
Jan 23, 2025
Merged

Feature/refactor sources #48

merged 49 commits into from
Jan 23, 2025

Conversation

leo-mazzone
Copy link
Collaborator

@leo-mazzone leo-mazzone commented Jan 16, 2025

Context

Changes proposed in this pull request

  • Removed SourceWarehouse, and generalised code to deal with multiple types of warehouse (at least Postgres and SQLite)
  • Removed datasets.toml and admin.py
  • Improved and simplified interfaces of Sources, Selector, SourceColumn, Match, and moved these to matchbox.common.sources (out of matchbox.common.db)
  • Sources now have an address, made up of their full name, and hashed representation of their warehouse
  • Multiple sources can now be mapped to the same resolution
  • Now storing column data in plain text on the server
  • Moving client-side code out of server logic for query function
  • Renamed some custom exceptions to clarify when they originate in the server
  • Removed ability to query without specifying resolution when selecting from multiple tables
  • Removed unused code in various places
  • Automatically reformatted some docstrings

Guidance to review

  • Further improvements to source columns are captured in MTCHBX-105
  • I haven't changed it, but are we sure that upon indexing a dataset we want to update the value of the source in the backend on conflict?

Checklist:

  • My code follows the style guidelines of this project
  • New and existing unit tests pass locally with my changes

test/fixtures/db.py Outdated Show resolved Hide resolved
src/matchbox/__init__.py Outdated Show resolved Hide resolved
@leo-mazzone leo-mazzone marked this pull request as ready for review January 23, 2025 09:08
Copy link
Collaborator

@wpfl-dbt wpfl-dbt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very well done!

@leo-mazzone leo-mazzone merged commit f8b0dcc into main Jan 23, 2025
5 checks passed
@leo-mazzone leo-mazzone deleted the feature/refactor-sources branch January 23, 2025 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants