SCP freezes wthen downloading large files:

[tgeek77]

This is the command that I am using at the output:

> ~/.ssh/bsshscp -P 222 root@foo:/root/logs.tgz ~/temp
*------------------------------------------------------------------------------*
|THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.|
|ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW.        |
*------------------------------------------------------------------------------*
 will try the following accesses you have:
  - group-member of admin with ED25519-256 key SHA256:nNbqefYIKSOMTha2XMXpMFu8KDx3Ws0GulB7VIRIBps [2024/06/19] 

MFA_TOKEN=notrequired
Connection to dev-bastion closed.
*------------------------------------------------------------------------------*
|THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.|
|ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW.        |
*------------------------------------------------------------------------------*
>>> Hello user, transferring your file through the bastion from [email protected]:222...
logs.tgz                                                                                                                                                  96% 2079MB  97.1MB/s   00:00 ETA

As you can see, the file is over 2GB. If I try this with a small file, it works. If I try it with directly scp (not using the bastion), it also works. This seems to be a hard limit to upload/download sizes that is not documented. I checked here and here.

[Kladislav]

same issue

[Kladislav]

@tgeek77 mb already solve?

[tgeek77]

No, it's still happening on my system.

[Kladislav]

idk, it looks like huge bug and no activity here, sad...

[speed47]

Hello,

I had some trouble reproducing it, but indeed it seems that on large files, it stalls at the end of the file regardless of its size, as long as it's big enough. Seems like some kind of race condition, I'll dig into it.

[speed47]

OK, it can actually happen with files as small as 50 MiB, but the bigger the file, the most likely it'll happen. This seems to happen mostly on downloads, not so much on uploads.

After some stracing, it appears that once the file has been read on the sending side, the scp server closes its file descriptor and sends an EOF as it should, but then waits for confirmation that the file has indeed been transferred forever: this is where sometimes, the deadlock happens. When it does, you'll be missing 64 KiB (the size of the read buffer) of the file on the receiving side, no matter what.

I've changed the exec mode of the scp plugin to remove some buffering that happens on the bastion itself, as I guessed this would be where the race condition could happen. With this change, I can no longer reproduce the issue. The only noticeable drawback is that the bastion can no longer report how many bytes have flowed through scp. The line:

>>> Done, 4739 bytes uploaded, 294878 bytes downloaded

will be replaced by:

>>> Done, scp exited successfully.

which seems a negligible compromise to make.

[tgeek77]

That sounds like an acceptable compromise to me. My team is using The Bastion in a our production environment and we have people asking to add their own SSH keys to the server as a workaround when manually gathering logs. It would be good to see this fix.

[speed47]

fixed in v3.18.00, if you could confirm that the issue does disappear on your environment too with this version, that would be great!

[tgeek77]

I have scheduled a change window for 1hr 15m from now. I will update with my test results.

[tgeek77]

Works for me! Test file was > 100MB

$ bscp -i ~/.ssh/key_ed25519 [email protected]:/var/log/messages.1 ./
*------------------------------------------------------------------------------*
|THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.|
|ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW.        |
*------------------------------------------------------------------------------*
 will try the following accesses you have:
  - group-member of dev with ED25519-256 key SHA256:RdoCbMCJ+ORa23l4dLRfHeqXjgSg64x8AtEWlpyq3XE [2024/06/19] 
  - group-member of admin with ED25519-256 key SHA256:nNbqefYIKSOMTha2XMXpMFu8KDx3Ws0GulB7VIRIBps [2024/06/19] 

MFA_TOKEN=notrequired
Connection to dev-bastion.dev.office.ad closed.
*------------------------------------------------------------------------------*
|THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.|
|ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW.        |
*------------------------------------------------------------------------------*
>>> Hello tgeek77, transferring your file through the bastion from [email protected]:22...
messages.1                                                                                                                                                   100%  126MB  19.1MB/s   00:06    
>>> Done, scp exited successfully.

[speed47]

Good, internally some people had the issue and they confirm that the issue disappeared too. Closing as resolved then!

[niallguerin]

Just confirming this scp fix works for scp download of files up to 5GB so far for us, so thank you for fixing. I notice rsync pull (and only rsync pull) seems to have a similar pattern of behaviour in that I can push large files via rsync fine from system A to system B via bastion, but when I test rsync pull via bastion, it seems to be ok up to about 100mb max and then it won't let me pull down larger files. Even below 100mb, it sometimes gives errors when it is over 45-50mb for the rsync pull and I had similar test case results with scp download until this fix and that issue is resolved. Let me know if you prefer I open a separate issue for tracking for rsync pull test, if you can recreate. Testing (for scp download) and rsync pull done against bastion version v3.18.00.

Error Result

unexpected tag 90 [receiver/inc]
rsync error: error in rsync protocol data stream (code 12) at io.c(1705) [receiver=3.4.1]
rsync: connection unexpectedly closed (54 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(232) [generator=3.4.1]

[tgeek77]

@niallguerin I think it would be a good idea to also give an example of the command that you are using. You can obfuscate real hostnames with example.com or use x.x.x.x for IP addresses.

[speed47]

@niallguerin could you open a new issue indeed? I'll see if I can reproduce. When this happens, does rsync appear to just be stuck, so you CTRL+C it and then it shows the error lines you've pasted, or does this give out an error directly, instead of getting stuck?

If it's stuck, it's probably a similar issue and the fix might be the same, otherwise, it can be a different issue. You might want to include the whole command you've typed and the whole output, removing any sensitive information as @tgeek77 suggested. Thanks!

[niallguerin]

Sorry for delay in responding. I'll open a separate issue as requested. I just want to set up some stuff to reproduce on another server at home as well as work. To answer your last reply, unlike the old scp behaviour (that was fixed), the rsync fails immediately without input from me (i.e. I do not hit CTRL+C in terminal) and the rsync pull fail case only occurs when the file is large whereas in the old scp large file download issue it did just become "stuck", rsync large file errors out immediately for me. Smaller files are fine though for rsync pull. I'll send full command and log output with hosts obfuscated as requested and I will log it as separate issue as scp issue is fixed for large files. Thank you.