Improve compare two records #2498
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RobinL
commented
Nov 12, 2024
| df_records_right, | ||
| pipeline, | ||
| in_tablename="__splink__compare_two_records_right_with_tf", | ||
| out_tablename="__splink__compare_two_records_right_with_tf_uid_fix", | ||
| uid_str="_right", | ||
| ) | ||
|
|
||
| sqls = block_using_rules_sqls( |
There was a problem hiding this comment.
If we're using cartesian blocking, we don't need to run any complex blocking code.
In addition, this code creates and materilises a list of pairwise Ids, which is uses for the join. This is unnecessary in the context of a handful of records
RobinL
commented
Nov 12, 2024
|
|
||
| This is needed e.g. when using linker.compare_two_records | ||
| or linker.inference.find_matches_to_new_records in which the user provides | ||
| new records which need tf adjustments computed | ||
| """ | ||
| tf_cols_already_populated = [] |
There was a problem hiding this comment.
The user can now include tf columns in the input data. If they do, do not recompute, and use whatever the user has provided
ADBond
approved these changes
Nov 12, 2024
splink/internals/realtime.py
Outdated
Comment on lines
146
to
149
| settings_obj._retain_matching_columns = retain_matching_columns | ||
| settings_obj._retain_intermediate_calculation_columns = ( | ||
| retain_intermediate_calculation_columns | ||
| ) |
There was a problem hiding this comment.
We probably don't need this in this case, as we only instantiate the Settings in this function
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR:
compare_two_recordsExample code demoing new features here
I have left the realtime module private for now i.e. essentially it's 'beta functionality'. I think we should start using this ourselves internally for a bit to make sure we're happy with it, before putting in the public api.
When using
compare_two_records, you previously had to precompute term frequency tables. You can now explicitly pass in term frequency adjustments in the input data e.g. a column calledtf_citywith hardcoded valueWhen using
compare_two_records, you previously could ONLY pass two records as dictionaries. You can now pass typed dataframes, e.g. a duckdbpyrelation if it’s a DuckDB linker. This fixes a bug whereby previously it was not possible to runcompare_two_recordswith complex data types e.g. a date type columncompare_two_recordspreviously did not produce afound_by_blocking_rulescolumn, this is now (optionally) outputted)compare_two_recordsmode previously was slow for two reasons:I’ve now included beta functionality of running
compare_two_recordswithout a linker, which will provisionally be atsplink.internals.realtime.compare_records. This ‘flavour’ of the function is deliiberately design for high performance (i.e. Core Person Record). It assumes (i.e. does not double check) that all data is present in the input records, INCLUDING term frequency adjustments. It memoizes generated SQL code so that it’s extremely fast (0.001seconds for a pariwise prediction, which is at least 10x faster. Previously most of the time taken was generating code)When these functions are called, the
retain_intermediate_calculation_columnsandretain_matching_columnssettings are always set toTrueirrespective of the settings in the main settings object. This means users can e.g. run predict using efficient settings, but generate a handful of waterfalls using this functionSupercedes #2426