Archive vcerare data
#438
Merged
Archive vcerare data
#438
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
e-belfer
changed the title
vceregen data and actually pass contributor type to Zenodo metadatavceregen data
zschira
approved these changes
Oct 15, 2024
| bucket = storage.Client().get_bucket(self.bucket_name) | ||
| blobs = bucket.list_blobs(prefix=f"{self.name}/") # Get all blobs in folder | ||
|
|
||
| for blob in blobs: |
There was a problem hiding this comment.
Not blocking: At some point it might be cool to use universal-pathlib which allows you to work with GCS objects like normal python Path's. I definitely don't think it's worth the effort right now though.
| # If data source was manually archived by us, specify that the | ||
| # data_source.path is a documentation link, rather than where we archived | ||
| # the data from. | ||
| if data_source_id in ["gridpathratoolkit", "vceregen"]: |
There was a problem hiding this comment.
Might be slightly cleaner to just set the title/description in the if statement and have a single return at the end since the rest of the fields are all the same.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Closes #434.
What problem does this address?
Adds
vceraredata to Zenodo. Because this data was provided to us and isn't available online, we upload our copy to a GCS bucket so we can archive from a stable source, as well as capture any file changes over time as new data is added.What did you change in this PR?
vcerare files fromsources.catalyst.coop` GCS bucket.pdfand.mdfiletypes to frictionlessvceraretorun-archiver.ymland add GCS configurationNote: Zenodo does not allow you to set which files are previewed from the API, and it automatically previews the "first file", which it determines by alphabetical sorting - this will always be
datapackage.jsonfor this repository. We'll need to manually check "preview" next to the README file before publishing when we update this dataset.Note 2: This PR contemplated adding contributor types to the Zenodo payload. We were setting them in
zenodo_rolebut not passing them to the Zenodo API. I configured this to specify type forcreator, as is possible in the GUI, but this is only possible for collaborators in the API. So, unless we reconfigure the way we handle creators this isn't possible - for now, we'll just made any adjustments manually in the GUI since we rarely refresh metadata.Testing
How did you make sure this worked? How can a reviewer verify this?
Run
pudl_archiver --datasets vcerare --sandbox --initialize. See also: https://zenodo.org/records/13919960To-do list
Tasks