Keep transport open for quick tasks #6595
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
EDIT: For now, restricting this to handling only the first task, by keeping track of the new
_last_close_time
attribute, and submitting a job immediately if this isNone
.transports.py
_last_request_special
attribute toTransportQueue
to track special requests that didn't respectsafe_open_interval
, to not overload SSH._last_open_time
and_last_close_time
to keep timing information when transports were opened/closed, same reason as above -> Use this timing information to evaluate when an open callback can be scheduled (immediately, after time difference between last open andsafe_open_interval
, or after the fullsafe_open_interval
(as done so far))transports.py
intotasks.py
, as otherwise we cannot reuse opened transports, as they are always closed after the task for which the request was created has finished, prohibiting the use of one transport for multiple tasks that finish quickly (to be thought about, as not always closing the transport might have undesired side effects...)open_transports
attribute toTransportQueue
similar to thetransport_requests
dictionary, with thepk
of theAuthInfo
instance as key and the transport as value. This can be accessed outside, intasks.py
to retrieve existing, open transports.tasks.py
do_
methods, see if an existing open transport for theAuthInfo
contained inTransportQueue._open_transports
, and use it. If not, create newTransportRequest
.TransportRequest
to be able to re-use an openTransport
). Logic for when they should be closed? When topop
theTransportRequest
s and open transports? After every successful re-use of aTransport
? Would be nice if an open transport could be used for "arbitrarily" many tasks that finish quickly.do_update
for now, as this is done in batches to poll multiple jobs (isn't that for the wholeTransportQueue
the case, at least according to the file docstring?)Notes from discussion with @khsrali and @unkcpz
execmanager.upload_calculation
intask_upload_job
is actually blocking, whiletask_upload_job
anddo_upload
defined inside are async -> Possibly Ali's async PR could fix the issue, but likely not, as everytask_
function still requests an individual transportclose
is being called in everyrequest_transport
call for every transport task