Create out_zenodo_logs to track changes in downloads over time
#209
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
e-belfer
force-pushed
the
out_zenodo_logs
branch
from
05d984a to
d35771d
Compare
zaneselvans
approved these changes
Oct 25, 2024
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
zaneselvans
approved these changes
Oct 29, 2024
bendnorman
requested changes
Jan 3, 2025
| Column( | ||
| "version_id", | ||
| Integer, | ||
| primary_key=True, |
There was a problem hiding this comment.
This is a nit but could you move this up to top of this table definition to it's easy to understand the full primary key of the table please?
Just so I understand, the version number represents a release of a dataset? I've been out of the zenodo world for a while.
|
|
||
| new_df = df.stack() | ||
| # Rename the diff columns | ||
| new_df = new_df.rename({col: "new_" + col for col in metrics_cols}, axis=1) |
There was a problem hiding this comment.
Why did you decide to rename these columns?
| @@ -466,6 +466,144 @@ | |||
| Column("partition_key", String), | |||
| ) | |||
|
|
|||
| out_zenodo_logs = Table( | |||
| "out_zenodo_logs", | |||
There was a problem hiding this comment.
I think it would be helpful to add table descriptions as doc strings or via Table.comment so we can easily understand the differences between the core and out zenodo tables.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Closes #215.
What problem does this address?
Makes it simpler to track changes in Zenodo downloads by version over time. Plotting diffs in Superset makes it impossible to make the legend legible, or to account for the first day of downloads.
What did you change in this PR?
Add
out_zenodo_logs, which calculates the difference in downloads and views daily for each version. Also backfills to 0 where the version does not yet exist to make it simpler to handle new records.core_zenodo_logsinput to not drop Saturday data by making it a non-partitioned assetTesting
How did you make sure this worked? How can a reviewer verify this?
Generate and then inspect
out_zenodo_logs.To-do list
Tasks