Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Actions artifact upload stalls when using self hosted runners #623

Open
shaik2419 opened this issue Sep 30, 2024 · 5 comments
Open

Actions artifact upload stalls when using self hosted runners #623

shaik2419 opened this issue Sep 30, 2024 · 5 comments
Labels
bug Something isn't working v4

Comments

@shaik2419
Copy link

What happened?

When uploading artifacts using self hosted runners, artifacts gets stalled with below error

With the provided path, there will be 550127 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
/home/runner/_work/_actions/actions/upload-artifact/v4/dist/upload/index.js:3312
                    throw new Error('Upload progress stalled.');
                    ^

Error: Upload progress stalled.
    at Timeout._onTimeout (/home/runner/_work/_actions/actions/upload-artifact/v4/dist/upload/index.js:3312:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)

Node.js v[20](https://github.com/CBRE-GCSO/gcso-iam-eadc/actions/runs/11113819006/job/30878983707#step:10:21).13.1

What did you expect to happen?

Artifacts are uploaded in less than a minute

How can we reproduce it?

I am running self hosted action runner using the image below. Upload an artifact with 500000 files which in total size is less than 100MB. It times out with the above error.

image: actions-runner:2.319.1

Run the same in ubuntu managed runner, it works

Anything else we need to know?

No response

What version of the action are you using?

2.319.1

What are your runner environments?

self-hosted, linux

Are you on GitHub Enterprise Server? If so, what version?

No response
@shaik2419 shaik2419 added the bug Something isn't working label Sep 30, 2024
@shaik2419
Copy link
Author

Hi is there any resolution to this. Ironically version 3 even though was slow, works. Please provide some steps to debug this in case v4 has to be used

@yacaovsnc yacaovsnc added the v4 label Jan 6, 2025
@yacaovsnc
Copy link
Contributor

Please let us know if this is still occurring, and if it still does, could you please try with ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY env and see if setting it to 10 helps? e.g.:

        - uses: actions/upload-artifact@v4
          with:
            name: my-artifact
            path: artifact-path/*
          env:
            ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY: 10

@mhx
Copy link

mhx commented Feb 3, 2025

Please let us know if this is still occurring

FWIW, after about two months I've started running workflows again on self-hosted runners. The same worksflows that ran completely fine during most of 2024 are now basically unable to successfully complete because a large fraction of the jobs (probably around 1/5 jobs) are failing during upload-artifact. The more common case is Upload progress stalled.:

2025-02-03T12:43:20.4206727Z ##[group]Run actions/upload-artifact@v4
2025-02-03T12:43:20.4207020Z with:
2025-02-03T12:43:20.4207333Z   name: dwarfs-0.10.2-130-gfb09a555e2-Linux-x86_64-clang-debug-stacktrace.tar.zst
2025-02-03T12:43:20.4208056Z   path: /home/mhx/git/github/actions-runner2/_work/_temp/artifacts/dwarfs-0.10.2-130-gfb09a555e2-Linux-x86_64-clang-debug-stacktrace.tar.zst
2025-02-03T12:43:20.4208652Z   if-no-files-found: error
2025-02-03T12:43:20.4208879Z   compression-level: 0
2025-02-03T12:43:20.4209103Z   overwrite: false
2025-02-03T12:43:20.4209336Z   include-hidden-files: false
2025-02-03T12:43:20.4209571Z env:
2025-02-03T12:43:20.4210372Z   binary_tarball: dwarfs-0.10.2-130-gfb09a555e2-Linux-x86_64-clang-debug-stacktrace.tar.zst
2025-02-03T12:43:20.4210979Z   universal_binary: dwarfs-universal-0.10.2-130-gfb09a555e2-Linux-x86_64-clang-debug-stacktrace
2025-02-03T12:43:20.4211402Z ##[endgroup]
2025-02-03T12:43:20.6082135Z With the provided path, there will be 1 file uploaded
2025-02-03T12:43:20.6085889Z Artifact name is valid!
2025-02-03T12:43:20.6087099Z Root directory input is valid!
2025-02-03T12:43:21.5731645Z Beginning upload of artifact content to blob storage
2025-02-03T12:48:21.5913262Z ##[error]Upload progress stalled.
2025-02-03T12:51:08.4058890Z Uploaded bytes 8388608
2025-02-03T12:51:09.4306325Z Uploaded bytes 16777216
2025-02-03T12:51:11.5264467Z Uploaded bytes 25165824
2025-02-03T12:51:11.9642021Z Uploaded bytes 33554432
2025-02-03T12:51:12.9225381Z Uploaded bytes 41943040
2025-02-03T12:51:13.4822431Z Uploaded bytes 50331648
2025-02-03T12:51:14.4381572Z Uploaded bytes 58720256
2025-02-03T12:51:14.5062381Z Uploaded bytes 67108864
2025-02-03T12:51:15.0201574Z Uploaded bytes 75497472
2025-02-03T12:51:16.4179881Z Uploaded bytes 83886080
2025-02-03T12:51:22.1023584Z Uploaded bytes 92274688
2025-02-03T12:51:57.5842984Z Uploaded bytes 92901746
2025-02-03T12:51:57.6174173Z Post job cleanup.

I've change the workflow to set

+        env:
+          ACTIONS_ARTIFACT_UPLOAD_TIMEOUT_MS: 1800000

for every invocation of upload-artifact hope that this would at least help the jobs to finish. However, I'm now running into Server failed to authenticate the request.:

2025-02-03T13:31:24.2002135Z ##[group]Run actions/upload-artifact@v4
2025-02-03T13:31:24.2002807Z with:
2025-02-03T13:31:24.2003498Z   name: dwarfs-0.10.2-131-g848ac5204e-Windows-AMD64-debug-stacktrace.7z
2025-02-03T13:31:24.2004665Z   path: C:\actions-runner\_work\_temp\build\dwarfs-0.10.2-131-g848ac5204e-Windows-AMD64-debug-stacktrace.7z
2025-02-03T13:31:24.2005437Z   if-no-files-found: error
2025-02-03T13:31:24.2005745Z   compression-level: 0
2025-02-03T13:31:24.2006014Z   overwrite: false
2025-02-03T13:31:24.2006286Z   include-hidden-files: false
2025-02-03T13:31:24.2006579Z env:
2025-02-03T13:31:24.2006775Z   do_install: true
2025-02-03T13:31:24.2007044Z   CommandPromptType: Native
2025-02-03T13:31:24.2007578Z   DevEnvDir: C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\
2025-02-03T13:31:24.2008431Z   ExtensionSdkDir: C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs
2025-02-03T13:31:24.2012335Z   EXTERNAL_INCLUDE: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um
2025-02-03T13:31:24.2016298Z   Framework40Version: v4.0
2025-02-03T13:31:24.2016695Z   FrameworkDir: C:\Windows\Microsoft.NET\Framework64\
2025-02-03T13:31:24.2017224Z   FrameworkDir64: C:\Windows\Microsoft.NET\Framework64\
2025-02-03T13:31:24.2017696Z   FrameworkVersion: v4.0.30319
2025-02-03T13:31:24.2018038Z   FrameworkVersion64: v4.0.30319
2025-02-03T13:31:24.2021605Z   INCLUDE: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um
2025-02-03T13:31:24.2027332Z   LIB: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\lib\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22000.0\\um\x64
2025-02-03T13:31:24.2031878Z   LIBPATH: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\lib\x86\store\references;C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.22000.0;C:\Program Files (x86)\Windows Kits\10\References\10.0.22000.0;C:\Windows\Microsoft.NET\Framework64\v4.0.30319
2025-02-03T13:31:24.2034662Z   NETFXSDKDir: C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\
2025-02-03T13:31:24.2046880Z   Path: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools\DiagnosticsHub\Collector;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22000.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\;C:\Program Files\Python312\Scripts\;C:\Program Files\Python312\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\bin;C:\Program Files\Git\cmd;C:\msys64\usr\bin;C:\Program Files\fio\;C:\Program Files\CMake\bin;C:\WINDOWS\ServiceProfiles\NetworkService\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Microsoft Visual Studio\Installer;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg
2025-02-03T13:31:24.2057726Z   Platform: x64
2025-02-03T13:31:24.2057998Z   UCRTVersion: 10.0.22000.0
2025-02-03T13:31:24.2058433Z   UniversalCRTSdkDir: C:\Program Files (x86)\Windows Kits\10\
2025-02-03T13:31:24.2059194Z   VCIDEInstallDir: C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\
2025-02-03T13:31:24.2060015Z   VCINSTALLDIR: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\
2025-02-03T13:31:24.2060767Z   VCPKG_ROOT: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg
2025-02-03T13:31:24.2061665Z   VCToolsInstallDir: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\
2025-02-03T13:31:24.2062806Z   VCToolsRedistDir: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Redist\MSVC\14.40.33807\
2025-02-03T13:31:24.2063496Z   VCToolsVersion: 14.41.34120
2025-02-03T13:31:24.2063989Z   VisualStudioVersion: 17.0
2025-02-03T13:31:24.2064608Z   VS170COMNTOOLS: C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\
2025-02-03T13:31:24.2065390Z   VSCMD_ARG_app_plat: Desktop
2025-02-03T13:31:24.2065708Z   VSCMD_ARG_HOST_ARCH: x64
2025-02-03T13:31:24.2066001Z   VSCMD_ARG_TGT_ARCH: x64
2025-02-03T13:31:24.2066272Z   VSCMD_VER: 17.11.5
2025-02-03T13:31:24.2066711Z   VSINSTALLDIR: C:\Program Files\Microsoft Visual Studio\2022\Community\
2025-02-03T13:31:24.2067788Z   WindowsLibPath: C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.22000.0;C:\Program Files (x86)\Windows Kits\10\References\10.0.22000.0
2025-02-03T13:31:24.2068831Z   WindowsSdkBinPath: C:\Program Files (x86)\Windows Kits\10\bin\
2025-02-03T13:31:24.2069471Z   WindowsSdkDir: C:\Program Files (x86)\Windows Kits\10\
2025-02-03T13:31:24.2069944Z   WindowsSDKLibVersion: 10.0.22000.0\
2025-02-03T13:31:24.2070509Z   WindowsSdkVerBinPath: C:\Program Files (x86)\Windows Kits\10\bin\10.0.22000.0\
2025-02-03T13:31:24.2071090Z   WindowsSDKVersion: 10.0.22000.0\
2025-02-03T13:31:24.2071787Z   WindowsSDK_ExecutablePath_x64: C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\
2025-02-03T13:31:24.2072843Z   WindowsSDK_ExecutablePath_x86: C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\
2025-02-03T13:31:24.2073529Z   __DOTNET_ADD_64BIT: 1
2025-02-03T13:31:24.2073827Z   __DOTNET_PREFERRED_BITNESS: 64
2025-02-03T13:31:24.2077185Z   __VSCMD_PREINIT_PATH: C:\Program Files\Python312\Scripts\;C:\Program Files\Python312\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\bin;C:\Program Files\Git\cmd;C:\msys64\usr\bin;C:\Program Files\fio\;C:\Program Files\CMake\bin;C:\WINDOWS\ServiceProfiles\NetworkService\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Microsoft Visual Studio\Installer
2025-02-03T13:31:24.2080001Z   binary_tarball: dwarfs-0.10.2-131-g848ac5204e-Windows-AMD64-debug-stacktrace.7z
2025-02-03T13:31:24.2080943Z   universal_binary: dwarfs-universal-0.10.2-131-g848ac5204e-Windows-AMD64-debug-stacktrace.exe
2025-02-03T13:31:24.2081675Z   ACTIONS_ARTIFACT_UPLOAD_TIMEOUT_MS: 1800000
2025-02-03T13:31:24.2082047Z ##[endgroup]
2025-02-03T13:31:25.2562974Z With the provided path, there will be 1 file uploaded
2025-02-03T13:31:25.2568755Z Artifact name is valid!
2025-02-03T13:31:25.2572125Z Root directory input is valid!
2025-02-03T13:31:26.0476099Z Beginning upload of artifact content to blob storage
2025-02-03T13:40:01.2367820Z Uploaded bytes 8388608
2025-02-03T13:40:09.7644646Z Uploaded bytes 16777216
2025-02-03T13:40:29.5528922Z Uploaded bytes 25165824
2025-02-03T13:41:06.5561288Z Uploaded bytes 33554432
2025-02-03T13:41:20.8554167Z Uploaded bytes 41943040
2025-02-03T13:41:27.8802429Z Uploaded bytes 50331648
2025-02-03T13:42:01.8638232Z Uploaded bytes 58720256
2025-02-03T13:42:02.6360989Z Uploaded bytes 67108864
2025-02-03T13:42:04.1687051Z Uploaded bytes 75497472
2025-02-03T13:42:06.9060308Z Uploaded bytes 83886080
2025-02-03T13:50:28.0110998Z Uploaded bytes 92274688
2025-02-03T13:51:56.3404930Z Uploaded bytes 100663296
2025-02-03T13:54:47.6299540Z Uploaded bytes 109051904
2025-02-03T14:02:43.0426805Z Uploaded bytes 117440512
2025-02-03T14:15:58.5436042Z Uploaded bytes 125829120
2025-02-03T14:36:01.2867528Z Uploaded bytes 129458835
2025-02-03T14:36:01.7491028Z ##[error]Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:08b97915-701e-0077-3c48-76f596000000
Time:2025-02-03T14:36:01.7766638Z

It takes more than one hour to try and upload the artifact.

@yacaovsnc
Copy link
Contributor

@mhx please give

ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY:5

a try instead of increasing the timeout. If you increase the timeout, you will run into long upload time if the problem isn't with the network layer.

mhx added a commit to mhx/dwarfs that referenced this issue Feb 3, 2025
@mhx
chore: try limiting artifact upload concurrency
6b003d7 
See actions/upload-artifact#623
@mhx
Copy link

mhx commented Feb 3, 2025

Doesn't seem to make much of a difference; at least, there are still cases like this:

2025-02-03T20:45:51.1352178Z ##[group]Run actions/upload-artifact@v4
2025-02-03T20:45:51.1352758Z with:
2025-02-03T20:45:51.1353404Z   name: dwarfs-0.10.2-131-g6b003d77e4-Linux-x86_64-clang-debug-stacktrace.tar.zst
2025-02-03T20:45:51.1355020Z   path: /home/mhx/git/github/actions-runner2/_work/_temp/artifacts/dwarfs-0.10.2-131-g6b003d77e4-Linux-x86_64-clang-debug-stacktrace.tar.zst
2025-02-03T20:45:51.1356295Z   if-no-files-found: error
2025-02-03T20:45:51.1356769Z   compression-level: 0
2025-02-03T20:45:51.1357341Z   overwrite: false
2025-02-03T20:45:51.1357762Z   include-hidden-files: false
2025-02-03T20:45:51.1358213Z env:
2025-02-03T20:45:51.1358895Z   binary_tarball: dwarfs-0.10.2-131-g6b003d77e4-Linux-x86_64-clang-debug-stacktrace.tar.zst
2025-02-03T20:45:51.1360573Z   universal_binary: dwarfs-universal-0.10.2-131-g6b003d77e4-Linux-x86_64-clang-debug-stacktrace
2025-02-03T20:45:51.1361644Z   ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY: 5
2025-02-03T20:45:51.1362252Z ##[endgroup]
2025-02-03T20:45:51.4776798Z With the provided path, there will be 1 file uploaded
2025-02-03T20:45:51.4780334Z Artifact name is valid!
2025-02-03T20:45:51.4781922Z Root directory input is valid!
2025-02-03T20:45:52.2595875Z Beginning upload of artifact content to blob storage
2025-02-03T20:50:52.2665021Z ##[error]Upload progress stalled.
2025-02-03T20:53:25.3930046Z Uploaded bytes 8388608
2025-02-03T20:53:28.8172445Z Uploaded bytes 16777216
2025-02-03T20:53:30.0730257Z Uploaded bytes 25165824
2025-02-03T21:00:30.4390350Z Uploaded bytes 33554432
2025-02-03T21:00:58.8885425Z Uploaded bytes 41943040
2025-02-03T21:01:00.3092662Z Uploaded bytes 50331648
2025-02-03T21:01:08.8773172Z Uploaded bytes 58720256
2025-02-03T21:08:35.8809254Z Uploaded bytes 67108864
2025-02-03T21:08:36.3731828Z Uploaded bytes 75497472
2025-02-03T21:08:50.2854892Z Uploaded bytes 83886080
2025-02-03T21:08:52.2322026Z Uploaded bytes 92274688
2025-02-03T21:09:28.3954065Z Uploaded bytes 92901729

@ezekg ezekg mentioned this issue Feb 4, 2025
Open
mhx added a commit to mhx/dwarfs that referenced this issue Feb 12, 2025
@mhx
chore: try limiting artifact upload concurrency
776f26b 
See actions/upload-artifact#623
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working v4
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mhx @yacaovsnc @shaik2419