-
-
Notifications
You must be signed in to change notification settings - Fork 191
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CI: SFTP mkdir sometimes fails #491
Comments
Hello,as you could guess, the problem is most likely related to the already created folders at the time of building I found issue in ssh2-sftp-client which you use in sftp-upload. Also i try to reproduce this error via this bash script: #!/bin/bash
echo "UPLOADER"
randomInt=$((1 + $RANDOM % 10))
git_sha=$(echo $randomInt | md5sum | head -c 20;)
# Our custom function
cust_func(){
env \
INPUT_HOST=your-host \
INPUT_LOCAL-FILE-GLOBS=*.json \
INPUT_PRIVATE-KEY="$(cat ~/.ssh/id_rsa)" \
INPUT_REMOTE-DIRECTORY=/home/opensource/tmp/$git_sha/manual/ \
INPUT_USER=opensource \
node index.js
}
# For loop 10 times
for i in {1..10}
do
cust_func $i & # Put a function in the background
done
## Put all cust_func in the background and bash
## would wait until those are completed
## before displaying all done message
wait
echo "All done" I think we need upgrade ssh2-sftp-client to 7.2.x |
Why would the failure happen sometimes, but not all the time? |
I do not quite understand yet what the problem is. logs from uploader.sh:UPLOADER
::debug::followSymbolicLinks 'true'
::debug::implicitDescendants 'true'
::debug::omitBrokenSymbolicLinks 'true'
::debug::Search path '/root/opensource/sftp-upload'
::debug::followSymbolicLinks 'true'
::debug::implicitDescendants 'true'
::debug::omitBrokenSymbolicLinks 'true'
::debug::Search path '/root/opensource/sftp-upload'
note: Found 2 files to upload
note: Found 2 files to upload
note: Uploading to remote directory: /home/opensource/tmp/48a24b70a0b376535542/manual/
note: Uploading to remote directory: /home/opensource/tmp/48a24b70a0b376535542/manual/
Error: Bad path: /home/opensource/tmp/48a24b70a0b376535542 permission denied
at Object.cb (/root/opensource/sftp-upload/node_modules/ssh2-sftp-client/src/index.js:845:27)
at 101 (/root/opensource/sftp-upload/node_modules/ssh2/lib/protocol/SFTP.js:2622:11)
at SFTP.push (/root/opensource/sftp-upload/node_modules/ssh2/lib/protocol/SFTP.js:278:11)
at CHANNEL_DATA (/root/opensource/sftp-upload/node_modules/ssh2/lib/client.js:525:23)
at 94 (/root/opensource/sftp-upload/node_modules/ssh2/lib/protocol/handlers.misc.js:859:16)
at Protocol.onPayload (/root/opensource/sftp-upload/node_modules/ssh2/lib/protocol/Protocol.js:2025:10)
at AESGCMDecipherBinding.decrypt (/root/opensource/sftp-upload/node_modules/ssh2/lib/protocol/crypto.js:1086:26)
at Protocol.parsePacket [as _parse] (/root/opensource/sftp-upload/node_modules/ssh2/lib/protocol/Protocol.js:1994:25)
at Protocol.parse (/root/opensource/sftp-upload/node_modules/ssh2/lib/protocol/Protocol.js:293:16)
at Socket.<anonymous> (/root/opensource/sftp-upload/node_modules/ssh2/lib/client.js:713:21) {
code: 'ERR_BAD_PATH'
}
Error: mkdir: Bad path: /home/opensource/tmp/48a24b70a0b376535542 permission denied
at fmtError (/root/opensource/sftp-upload/node_modules/ssh2-sftp-client/src/utils.js:55:18)
at SftpClient.mkdir (/root/opensource/sftp-upload/node_modules/ssh2-sftp-client/src/index.js:896:13)
at processTicksAndRejections (internal/process/task_queues.js:97:5) {
code: 'ERR_BAD_PATH',
custom: true
}
note: Uploading: /root/opensource/sftp-upload/package-lock.json
note: Uploading: /root/opensource/sftp-upload/package-lock.json
note: Uploading: /root/opensource/sftp-upload/package.json
note: Uploading: /root/opensource/sftp-upload/package.json
All done uploader.sh#!/bin/bash
echo "UPLOADER"
randomInt=$((1 + $RANDOM % 10))
git_sha=$(echo $randomInt | md5sum | head -c 20;)
# Our custom function
cust_func(){
env \
INPUT_HOST=your-host \
INPUT_LOCAL-FILE-GLOBS=*.json \
INPUT_PRIVATE-KEY="$(cat ~/.ssh/id_rsa)" \
INPUT_REMOTE-DIRECTORY=/home/opensource/tmp/$git_sha/manual/ \
INPUT_USER=opensource \
node index.js
}
# For loop 2 times
for i in {1..2}
do
cust_func $i & # Put a function in the background
done
## Put all cust_func in the background and bash
## would wait until those are completed
## before displaying all done message
wait
echo "All done" The problem appears if you create parralel connection for ssh2-sftp-client as i know ssh2-sftp-client does not support concurrent operations without good configured promises... ...
INPUT_REMOTE-DIRECTORY=/home/opensource/tmp/$git_sha/manual/ auditd from Linux:type=SYSCALL msg=audit(1640363056.766:1766): arch=c00000b7 syscall=34 success=yes exit=0 a0=ffffffffffffff9c a1=aaaaf154dfa0 a2=1ff a3=ffffb69cc6d0 items=2 ppid=10248 pid=10270 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=139 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=PATH msg=audit(1640363056.766:1766): item=0 name="/home/opensource/tmp/" inode=522270 dev=08:01 mode=040775 ouid=1002 ogid=1002 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1640363056.766:1766): item=1 name="/home/opensource/tmp/48a24b70a0b376535542" inode=522494 dev=08:01 mode=040775 ouid=1002 ogid=1002 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(1640363056.834:1767): arch=c00000b7 syscall=34 success=no exit=-17 a0=ffffffffffffff9c a1=aaaaf8699fa0 a2=1ff a3=ffffa7afa6d0 items=1 ppid=10269 pid=10271 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=141 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=PATH msg=audit(1640363056.834:1767): item=0 name="/home/opensource/tmp/" inode=522270 dev=08:01 mode=040775 ouid=1002 ogid=1002 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(1640363056.862:1768): arch=c00000b7 syscall=34 success=yes exit=0 a0=ffffffffffffff9c a1=aaaaf154dfa0 a2=1ff a3=ffffb69cc6d0 items=5 ppid=10248 pid=10270 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=139 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=PATH msg=audit(1640363056.862:1768): item=0 name="/home/opensource/tmp/48a24b70a0b376535542/" inode=522494 dev=08:01 mode=040775 ouid=1002 ogid=1002 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(1640363056.954:1769): arch=c00000b7 syscall=56 success=yes exit=3 a0=ffffffffffffff9c a1=aaaaf154ce60 a2=241 a3=1b6 items=2 ppid=10248 pid=10270 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=139 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=PATH msg=audit(1640363056.954:1769): item=0 name="/home/opensource/tmp/48a24b70a0b376535542/manual/" inode=522495 dev=08:01 mode=040775 ouid=1002 ogid=1002 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1640363056.954:1769): item=1 name="/home/opensource/tmp/48a24b70a0b376535542/manual/package-lock.json" inode=522496 dev=08:01 mode=0100664 ouid=1002 ogid=1002 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(1640363057.034:1770): arch=c00000b7 syscall=56 success=yes exit=3 a0=ffffffffffffff9c a1=aaaaf8698e60 a2=241 a3=1b6 items=2 ppid=10269 pid=10271 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=141 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=PATH msg=audit(1640363057.034:1770): item=0 name="/home/opensource/tmp/48a24b70a0b376535542/manual/" inode=522495 dev=08:01 mode=040775 ouid=1002 ogid=1002 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1640363057.034:1770): item=1 name="/home/opensource/tmp/48a24b70a0b376535542/manual/package-lock.json" inode=522496 dev=08:01 mode=0100664 ouid=1002 ogid=1002 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(1640363057.046:1771): arch=c00000b7 syscall=52 success=yes exit=0 a0=3 a1=1b6 a2=ffffe1f349d0 a3=ffffb69cc6d0 items=1 ppid=10248 pid=10270 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=139 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=SYSCALL msg=audit(1640363057.130:1772): arch=c00000b7 syscall=52 success=yes exit=0 a0=3 a1=1b6 a2=ffffe6d74280 a3=ffffa7afa6d0 items=1 ppid=10269 pid=10271 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=141 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=SYSCALL msg=audit(1640363057.422:1773): arch=c00000b7 syscall=56 success=yes exit=3 a0=ffffffffffffff9c a1=aaaaf154ce60 a2=241 a3=1b6 items=2 ppid=10248 pid=10270 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=139 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=PATH msg=audit(1640363057.422:1773): item=0 name="/home/opensource/tmp/48a24b70a0b376535542/manual/" inode=522495 dev=08:01 mode=040775 ouid=1002 ogid=1002 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1640363057.422:1773): item=1 name="/home/opensource/tmp/48a24b70a0b376535542/manual/package.json" inode=522497 dev=08:01 mode=0100664 ouid=1002 ogid=1002 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(1640363057.518:1774): arch=c00000b7 syscall=52 success=yes exit=0 a0=3 a1=1b6 a2=ffffe1f349d0 a3=ffffb69cc6d0 items=1 ppid=10248 pid=10270 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=139 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=SYSCALL msg=audit(1640363057.518:1775): arch=c00000b7 syscall=56 success=yes exit=3 a0=ffffffffffffff9c a1=aaaaf8698e60 a2=241 a3=1b6 items=2 ppid=10269 pid=10271 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=141 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp"
type=PATH msg=audit(1640363057.518:1775): item=0 name="/home/opensource/tmp/48a24b70a0b376535542/manual/" inode=522495 dev=08:01 mode=040775 ouid=1002 ogid=1002 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1640363057.518:1775): item=1 name="/home/opensource/tmp/48a24b70a0b376535542/manual/package.json" inode=522497 dev=08:01 mode=0100666 ouid=1002 ogid=1002 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(1640363057.614:1776): arch=c00000b7 syscall=52 success=yes exit=0 a0=3 a1=1b6 a2=ffffe6d74280 a3=ffffa7afa6d0 items=1 ppid=10269 pid=10271 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=(none) ses=141 comm="sftp-server" exe="/usr/lib/openssh/sftp-server" subj=unconfined key="tmp" _P.S I also was try to pre create all needed dirs before create ssh2-sftp-client connections and all files migrated successfully without this errors. |
In quick-lint-js' builds, artifact uploads often fail [1]: note: Found 1 files to upload note: Uploading to remote directory: ***/builds/d32117cb896385a05fdc68c727535f5b9f3f9a33/manual/ error: Error: _mkdir: Failure ***/builds/d32117cb896385a05fdc68c727535f5b9f3f9a33 at fmtError (/Users/github-builder/github-actions-runner/_layout/_work/_actions/quick-lint/sftp-upload/master/node_modules/ssh2-sftp-client/src/utils.js:53:18) at /Users/github-builder/github-actions-runner/_layout/_work/_actions/quick-lint/sftp-upload/master/node_modules/ssh2-sftp-client/src/index.js:729:20 at SFTPStream._transform (/Users/github-builder/github-actions-runner/_layout/_work/_actions/quick-lint/sftp-upload/master/node_modules/ssh2-streams/lib/sftp.js:417:17) at SFTPStream.Transform._write (node:internal/streams/transform:184:23) at writeOrBuffer (node:internal/streams/writable:389:12) at _write (node:internal/streams/writable:330:10) at SFTPStream.Writable.write (node:internal/streams/writable:334:10) at Channel.ondata (node:internal/streams/readable:754:22) at Channel.emit (node:events:394:28) at addChunk (node:internal/streams/readable:315:12) This issue may have been fixed in ssh2-sftp-client upstream [2]. Upgrade to get the potential fix. The ssh2 package, used by ssh2-sftp-client, added an optional Node.js native dependency. Make sure to not install it by running 'yarn install --ignore-optional' instead of just 'yarn install'. [1] quick-lint/quick-lint-js#491 [2] theophilusx/ssh2-sftp-client#360 Co-Authored-By: Rishat Sultanov <[email protected]>
I landed @Rishats's possible solution in commit b2939e2 (which uses quick-lint/sftp-upload@f91d9df). I assume the problem has been fixed. I will reopen this issue if I see the problem again. |
I think this issue is still happening:
|
Another flake: https://github.com/quick-lint/quick-lint-js/actions/runs/6206686929/job/16851193547
|
This issue is not worth keeping open. |
Another flake: https://github.com/quick-lint/quick-lint-js/actions/runs/6992496178/job/19024186235
|
EDIT: I think the below issue is persistent, not one-off. Will investigate. https://github.com/quick-lint/quick-lint-js/actions/runs/7257535608/job/19771502097
|
Probably a race condition.
e.g. https://github.com/quick-lint/quick-lint-js/runs/4122606582?check_suite_focus=true
The text was updated successfully, but these errors were encountered: