automatically determine remote_load #53
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Merge pull request #45 from ldeo-glaciology/ongoing-work-jk-07-24
The NB contains a working function that implements all the things I was hoping to get working: - fft of mautli-dimensional datarrays - Georges's demean, detrend, drop bad chirps. - reference array - agrees with the code implemented in xapres currently (- the error in the padding described in an issue #48)
also adds an option to load all to use the legacy fft method and to uses a corrected padding procedure for comparison with the new methods. custom_fft_implementation.ipynb tests this out, but needs more work to make it understanable.
…oved the deliberate error
add a test of the new fft methods and compares the to the previous way of doing it and doing the same thing in different ways with the new methods, to make sure they all give the same answer. Also edited test_file_selection_methods so that it loads from local, not a google bucket. Just to speed things up a bit. All test run in 30s on my local machine.
this will need more testing. Currently I have not been using these features and by default they are turned off.
…eters for load_all
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Taking over from #29.
After this PR you will not need to specify remote_load=True when trying to laod from a google bucket.
A new function,
from_dats.is_this_a_remote_load, determines remote_load based on whether gs:// is in the directory you are trying to load from.Future work could get this working for other cloud storage locations.