You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If a file exists in the local cache, the file is not verifid on a call to getreferences. Using the example in this issue below, when run the first time (assuming no corresponding file exists) the file will be downloaded and verified as seen here
"""Download and verify file `name` under context `pipeline_context` to `localpath`."""
ifconfig.get_download_plugin():
self.plugin_download(name, localpath)
else:
generator=self.get_data_http(name)
self.generator_download(generator, localpath)
self.verify_file(name, localpath)
If run a second time (when the file exists) the file will be opened and not verified. This can be confirmed by replacing the file with a blank file of the same name.
@stscieisenhamer came up with the initial test (replacing the file with a blank file) and mentioned that this issue can result in an incorrect local crds cache if multiple processes attempt to fetch the same file (crds attempts to protect against this but may not be perfect). @zacharyburnett may have fixed the jwst regression tests to improve the reliability of crds syncing and these improvements might be port-able to roman to improve the reliability of regression tests.
The text was updated successfully, but these errors were encountered:
This issue came up again while testing with crds-test. The nrm reference file jwst_niriss_nrm_0002.fits was renamed on the server to jwst_niriss_nrm_0001.fits. Since I already had a local cache which contained the old jwst_niriss_nrm_0001.fits crds failed to download the new version.
If a file exists in the local cache, the file is not verifid on a call to
getreferences
. Using the example in this issue below, when run the first time (assuming no corresponding file exists) the file will be downloaded and verified as seen herecrds/crds/client/api.py
Lines 673 to 680 in aa0d8f3
If run a second time (when the file exists) the file will be opened and not verified. This can be confirmed by replacing the file with a blank file of the same name.
@stscieisenhamer came up with the initial test (replacing the file with a blank file) and mentioned that this issue can result in an incorrect local crds cache if multiple processes attempt to fetch the same file (crds attempts to protect against this but may not be perfect). @zacharyburnett may have fixed the jwst regression tests to improve the reliability of crds syncing and these improvements might be port-able to roman to improve the reliability of regression tests.
The text was updated successfully, but these errors were encountered: