DM-46185: Refactor and simplify. #29
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sometimes we may not want to constrain the data query
hsinfang
force-pushed
the
tickets/DM-46185
branch
3 times, most recently
from
7865ff7 to
f7130b3
Compare
hsinfang
force-pushed
the
tickets/DM-46185
branch
from
f7130b3 to
b4dbde9
Compare
The original test is not quite right, because when the embargo hours falls right on an exposure, it should unembargo it. Copy the original test to a new test test_copy_just_one. shift the "now" time test_nothing_copies to be 0.000001 second earlier, so all data still stay within the embargo period, and nothing should copy. 5596964.255774 sec is the time difference between 2020-01-17 16:55:11.322700 and 2019-11-13T22:12:27.066926, the start time of exposure 2019111300059.
|
@ktlim may you please take a look at my commits? Now all tests pass as before this PR (two I tried to preserve what I think the tests were written to mean, and leave test refactoring to a later ticket. |
test_main_copy_midnight_precision is idential to test_main_midnight_precision.
test_raw_and_calexp_should_copy and test_raw_and_calexp_should_copy_yaml have two, different now_time_embargo timestamps. With the new API, it is more difficult to test with multiple now_time_embargo timestamps in one call. Also we wouldn't use that in pratical use case anyway. So, use just the later timestamp, and shift the embargo period for the raw datasets that used the earlier timestamp. The amount of shifting is the difference between the two original now_time_embargo timestamps: 2022-11-13T03:35:12.836981 - 2020-01-17T16:55:11.322700 = 89030401.514281 seconds, or 24730.66708730028 hr. Instead of an embargo period of 0.1 hr, the new embargo period is 0.1 + 24730.66708730028 hr.
hsinfang
force-pushed
the
tickets/DM-46185
branch
from
19b9816 to
c5559ae
Compare
|
Sorry, pulling back one commit. |
Exposure 2020011700004 has a start time at 2020-01-17T16:55:10.322700 and end time at 2020-01-17T16:55:11.322700. 3827088.677301 seconds after 2020-01-17T16:55:10.322700 is 2020-03-02T00:00:00.000000 So, in test_main_midnight_precision, 2020011700004 _just_ got out of the embargo period.
|
Fixed |
transfer_dataset_type is used with batching. It is possible that no datasets match the data IDs in a particular batch from all possible exposure/visit records.
|
Notes: this branch doesn't work with some datasets at the moment, because of DM-48094 since |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.