Fix errors due to non-unpicklable Exception when "enqueue=True" #963
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix #329.
Fix #504.
There are instances where a custom
Exception
might contain attributes that cannot be serialized or is incorrectly implemented (see Cannot unpickle Exception subclass). If this occurs withenqueue=True
, an error will be raised by the background thread when it callsqueue.get()
. Due to the failure of de-serialization, therecord
isNone
, resulting in an inability to identify the origin of the error.To address this concern, we can take a proactive step by attempting to unpickle the
Exception
beforehand. In the event of a failure, we replace its value withNone
to make sure it can be unpickled. This strategy mirrors what we already do for the serialization part. Actually, this approach was already implemented in the past. However, it was temporarily removed due to concerns raised about its safety (#563). Subsequent assessments have demonstrated its complete safety, allowing us to reintroduce it. Furthermore, we have expanded its scope to encompass all types of exceptions, not solely limited toUnpicklingError
. This broader implementation serves to mitigate the aforementioned issues effectively.