RSDK-12274: Streaming file copy: write to temp file & rename on success #5424
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
viambot
added
the
safe to test
Member
|
LGTM @aldenh-viam. I like the idea of ensuring the file copy is completed before naming the file the expected name. That will fix the state we are seeing. |
cheukt
reviewed
Oct 29, 2025
| } else { | ||
| // Since file may be streamed (see copyFile in copy_rpc.go), write to a temp file (filename.download) and rename after the download | ||
| // has completed. This way a temporary network blip won't leave a corrupted partial file in the expected location. | ||
| // Technically the temp file can be deleted upon any Copy error, but it will also be clobbered next time we retry. |
Member
There was a problem hiding this comment.
let's do a best effort clean up here, would suck if the partially downloaded module is very large and we just leave it hanging around
viambot
added
safe to test
aldenh-viam
force-pushed
the
push-trlktntwqurw
branch
from
8efaa34 to
91ff31b
Compare
viambot
added
safe to test
viambot
added
safe to test
viambot
added
safe to test
cheukt
approved these changes
Nov 3, 2025
Member
cheukt
left a comment
cheukt
left a comment
There was a problem hiding this comment.
LGTM - thanks for adding the test!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
shellRPCFileReadCopier / shellFileCopyWriterin services/shell/copy_rpc.go uses a gRPC stream to upload a large file in chunks directly into the specified filename. If there is a network blip midway through the transfer, the partial file is left on the destination. If this happens during aviam module reload, RDK'sreconfigurewill repeatedly try loading the corruptedmodulename.tar.gzand failing.This change makes
shellRPCFileReadCopierwrite files to a temporaryfilename.downloadand rename it tofilenameonly if the stream completes without error.Note: this
Copymethod also supports directories, but if my understanding is correct, it only creates empty directories for future files, so it doesn't need the same write-to-temp-and-rename logic.cc @michaellee1019 @cheukt: team discussed offline and we believe we don't want to skip failed packages and proceed with regular startup to avoid the possibility of the robot getting stuck in a bad config state.