Add region property to DAG #677
Conversation
|
The only issue I foresee is if someone overrides region after DAG already started. |
|
@spencerseale if I read the code correctly, the new property doesn't tell us for sure what region the DAG is running in, it tells us the current client configuration. Which is pretty much what you said above, right? I think this goes against the OO principle of encapsulation. DAG methods should be based on DAG data, not data in another class or module. |
|
@sgillies That's correct and you're right. I can take another stab at it so it's pinned to the DAG object. |
* Generalize directory ingest to cover the single file case Fixes the registration error in the directory case. Much less code and clearer about the intent of it. * Fix output URI pathlib loses the double slashes in our URIs, so we can't use that everywhere. * Append input_uri
* Defer loading of cloud.json file This is achieved by deferring the initialization of low-level instances until they are needed and by customizing attribute access for the confg.py module. * Explicitly call both the methods from the original constructor
* docstr/comment typos * uDAG arg validation: raise `ValueError`s instead of `TileDBCloudError`s * type nit
* Defer loading of cloud.json file This is achieved by deferring the initialization of low-level instances until they are needed and by customizing attribute access for the confg.py module. * Explicitly call both the methods from the original constructor * Fix failing config test, add more notes to config.py. Also xfail two consistently failing task graph tests. * Remove duplicated function
* Expose deadline argument for DAG * Use 86400 as default instead of None --------- Co-authored-by: Sean Gillies <[email protected]>
Fixes SC-60567
* Vendor cloudpickle * Install missing cloudpickle test components * Add cloudpickle back to CI requirements Needed by a pickle compatibility test * In module code, import vendored cloudpickle as tdbcp
Also fix the logger
Add
DAG.regionfor better detection of DAG host region