[BUG] Using DownloadSDK stil fails to download files #373

OEI-Cgray · 2024-05-16T16:18:21Z

Describe the bug
DownloadSDK passes its tests during the Set-AksHciConfig, but still fails to actually download the entire file. When it fails, it completely closes the existing powershell window as well, with no mention of a log file... To even see the error message, I had to run powershell within powershell so on the first process would be killed, leaving the error visible to see.

To Reproduce
Steps to reproduce the behavior:

Setup new AKSHCI cluster.

Expected behavior
Files to actually download.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

OS: Windows 2019 Datacenter
Version 1.1.104

Collect log files

Set-AksHciConfig -imageDir "c:\clusterstorage\containerstorage\AKS-Images" -cloudConfigLocation "c:\clusterstorage\containerstorage\AKS-Config" -workingDir "c:\clusterstorage\containerstorage\AKS-WorkingDir" -vnet $vnet -cloudservicecidr "10.42.0.129/23"
True
DownloadSdk Tests Execution Started 16.05.2024 09:03:44
===============================================================================

Test (1 of 1): "Validate DownloadSDK Host Firewall URL Requirements". Category: DownloadSDK Host
Testing firewall url requirements
Connection to https://msk8s.api.cdp.microsoft.com ... Succeeded
Connection to http://msk8s.b.tlu.dl.delivery.mp.microsoft.com ... Succeeded

Host is able to reach list of URLs requirements

Test Succeeded
Details: Host is able to reach list of URLs requirements
Recommendation:
Test execution time: 244.9995 milliseconds

DownloadSdk Tests Execution Ended 16.05.2024 09:03:44

=====================================================
       All DownloadSdk Validation tests are successful
=====================================================

Check the test report(downloadsdk_validation_report.html) in current directory

Kva Tests Execution Started 16.05.2024 09:03:45
===============================================================================

Test (1 of 1): "Validate KVA". Category: KVA

panic: totalWritten != stat.Size()

goroutine 145 [running]:
github.com/vbauerster/getparty.Session.concatenateParts({{0x1c0002c8690, 0xef}, {0x1c0002c8690, 0xef}, {0x1c00001e489, 0x6e}, {0x0, 0x0}, {0x1c0001efec5, 0x5}, ...}, ...)
        /home/vsts/go/pkg/mod/github.com/vbauerster/[email protected]/session.go:146 +0x15a5
github.com/vbauerster/getparty.(*Cmd).Run(0x1c000407080, {0x1c00011e100, 0x5, 0x8}, {0x0, 0x0}, {0x0, 0x0})
        /home/vsts/go/pkg/mod/github.com/vbauerster/[email protected]/getparty.go:318 +0x1d85
msazure.visualstudio.com/msazure/msk8s/downloadsdk.git/sdk/pkg/http.DownloadFile({0x7ffc2e2ac348?, 0x1c0001874a0}, {0x1c0002c8690, 0xef}, {0x1c00027e070, 0x6e}, 0xa, {0x0, 0x0}, 0x1c0001b6160)
        /home/vsts/work/1/s/sdk/pkg/http/http.go:85 +0x72f
msazure.visualstudio.com/msazure/msk8s/downloadsdk.git/sdk/pkg/sfsclient.(*SFSClient).GetVerifiedFiles(0x1c0002bd008, {0x7ffc2e2ac348, 0x1c0001874a0}, {{{0x1c000111760, 0x1c}, {0x7ffc2e1d76f0, 0x7}, {0x1c000498144, 0xc}}}, {0x1c0001e3680, ...}, ...)
        /home/vsts/work/1/s/sdk/pkg/sfsclient/sfsclient.go:187 +0x85b
msazure.visualstudio.com/msazure/msk8s/downloadsdk.git/sdk/pkg/download/provider/sfs.(*sfsProvider).getReleaseInternal(0x1c0002bd020, {0x7ffc2e2ac348, 0x1c0001874a0}, {0x1c000111760?, 0x0?}, {0x1c000498144?, 0x0?}, {0x1c0001e3680, 0x46}, 0xa, ...)
        /home/vsts/work/1/s/sdk/pkg/download/provider/sfs/release.go:49 +0x454
msazure.visualstudio.com/msazure/msk8s/downloadsdk.git/sdk/pkg/download/provider/sfs.(*sfsProvider).GetRelease(0x0?, {0x7ffc2e2ac348?, 0x1c0001874a0?}, {0x1c000111760?, 0x0?}, {0x1c000498144?, 0x0?}, {0x0?, 0x0?}, {0x0, ...}, ...)
        /home/vsts/work/1/s/sdk/pkg/download/provider/sfs/release.go:16 +0x96
msazure.visualstudio.com/msazure/msk8s/downloadsdk.git/sdk/pkg/download.(*downloadClient).getRelease(0x1c0003107b0, {0x7ffc2e2ac348, 0x1c0001874a0}, {{0x1c000111760, 0x1c}, {0x1c000498144, 0xc}, {0x0, 0x0}, {0x0, ...}, ...})
        /home/vsts/work/1/s/sdk/pkg/download/release.go:72 +0x2ad
msazure.visualstudio.com/msazure/msk8s/downloadsdk.git/sdk/pkg/download.(*downloadClient).GetRelease(0x0?, {0x7ffc2e2ac348?, 0x1c0001874a0?}, {{0x1c000111760, 0x1c}, {0x1c000498144, 0xc}, {0x0, 0x0}, {0x0, ...}, ...})
        /home/vsts/work/1/s/sdk/pkg/download/release.go:55 +0x13d
main.GetRelease.func2()
        /home/vsts/work/1/s/sdk/main.go:175 +0x97
created by main.GetRelease
        /home/vsts/work/1/s/sdk/main.go:174 +0x5ec

The text was updated successfully, but these errors were encountered:

Elektronenvolt · 2024-05-16T18:40:10Z

@OEI-Cgray I'm familiar with quite some error messages at stage "Validate KVA" but I've never seen URLs *.github.com and *.visualstudio.com at AKS Arc setups. What exactly are you trying to do?

OEI-Cgray · 2024-05-16T22:35:34Z

@Elektronenvolt , trying to install it. I couldn't even set the config, let alone run install-akshci.

This is the same issue I've seen before, downloads in my environment, with no restrictions on outgoing traffic, fail to download some file. It's not always the same file. If I enable logging and verbose output, I can find the URL that fails to download, and then download it with invoke-webrequest from the same machine, even the same powershell session. But using the scripts, which download with the DownloadSDK module, fails.

Because the download size did not match the expected download size, downloadsdk panicked, and killed the process.

The file paths are for the DownloadSDK module, which I assume at the moment is a DLL, those would be the local source folders of whatever machine built the release DLL, or something along those lines. They are there because of the panic crash.

Elektronenvolt · 2024-05-21T08:50:37Z

@OEI-Cgray - I never have seen that issue in any of the setups I did. Looks like an issue with the underlying storage. Your download with Invoke-Webrequest to the same storage you've set for AKS Arc? And these days - endpoint protections got better - download from PS may treated as malicious activity.
What are the specs of the storage? SSDs only?

OEI-Cgray · 2024-05-22T01:04:30Z

@Elektronenvolt

The storage is all enterprise class SSDs, and works fine when we were able to install this over a year ago before these same download issues made us abandon it. And yes, downloaded to the same storage.

Defender is our endpoint protection, and it does not stop the invoke-webrequest test. That's not to say the activity wasn't flagged when downloading via DownloadSDK, but when checking, Defender currently says it's never detected a threat on the three machines that make up the cluster. The machines have no threat detection history according to the various Azure portals either.

I'm happy to generate logs or whatever else needed to troubleshoot this... meanwhile we'll also be setting up all the things to basically do this manually by installing k8s on some vms and then installing the arc integrations afterwards... which is all less ideal than hopefully getting this issue resolved.

Elektronenvolt · 2024-05-23T17:36:14Z

@OEI-Cgray Interesting issue - I'm always curious to know the root cause, a lot of issues I've seen with AKS Arc the last years had been caused by our infrastructure or 'limitations' like Firewalls, permissions, ... In case it comes down to be infra related, would be nice to know the root cause.
In which Azure region do you run the setup? I'm in West Europe

Well - try to run with debug and verbose flags on, may you see anything interesting in the debug and verbose output.
$VerbosePreference = "continue"
$DebugPreference = "continue"

If that doesn't point you to something interesting, looks like it's time for a support case.

Elektronenvolt · 2024-06-03T11:02:15Z

@OEI-Cgray - by searching for something else I've seen that AKS Arc is right now supported on these three regions only: Azure-Regions
You may run the setup in a non-supported region?

OEI-Cgray · 2024-06-05T19:33:04Z

@Elektronenvolt The resource group is in west us, but the download error occurs on the Set-AksHciConfig, as far as I'm aware, you specify the resource group on the next command, Set-AksHciRegistration.

I'll have some time finally towards the end of this week for more testing.

Elektronenvolt · 2024-06-13T12:57:46Z

@OEI-Cgray According to docs West US is not supported. Yes, true - would also expect a region-based error at the step where you set the resource group at Set-AksHciRegistration

Continue testing - the June release is out and, in your case - try the offline download and offsite setup

If that works, consider Internet connectivity issues like #355

OEI-Cgray · 2024-06-22T00:02:51Z

Hello again, I finally got some time to try the offline download. Attached is the end of the results. It did pass all the 9 tests without any issues.

Test-OfflineDownloadFiles : Cannot bind argument to parameter 'DifferenceObject' because it is null.
At C:\Program Files\WindowsPowerShell\Modules\AksHci\1.2.4\AksHci.psm1:7656 char:17
+ ...             Test-OfflineDownloadFiles -destination $destination -vers ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Test-OfflineDownloadFiles], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Test-OfflineDownloadFiles

aks-log.txt

OEI-Cgray · 2024-06-25T20:58:33Z

Digging into this more, this function doesn't seem to exist.

It's referenced a few lines before the line 7656 error.

    $updateReleases = Get-ProductReleasesUptoVersion -Version $startVersion -moduleName $moduleName
    $updateReleases | ForEach-Object  {
        $version = $_.Version
        if ([System.Version]$version -ge [System.Version]$startVersion)
        {
            $destination = $global:config[$modulename]["stagingShare"] + "\" + $version
            if (Test-Path $destination) {
                Test-OfflineDownloadFiles -destination $destination -version $version
            } else {
                New-Item -Path $destination -ItemType Directory
                Get-ReleaseContent -version $version -activity $activity -destination $destination -moduleName $moduleName -mode $mode
            }
        }
    }

This does not seem to exist.

Get-ProductReleasesUptoVersion
Get-ProductReleasesUptoVersion : The term 'Get-ProductReleasesUptoVersion' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path
was included, verify that the path is correct and try again.
At line:1 char:1
+ Get-ProductReleasesUptoVersion
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Get-ProductReleasesUptoVersion:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Elektronenvolt · 2024-06-26T14:23:38Z

Line 7656 is Test-OfflineDownloadFiles -destination $destination -version $version and it complains that DifferenceObject is null.
If you go to the function in line 8450 - it wants to get the name of the downloaded files:
$missingFiles = Compare-Object -ReferenceObject $releaseFiles.Name -DifferenceObject $downloadedFiles.Name

In your log the files I see a user profile path - like: Saving to: "C:\\Users\\myuser-~1\\AppData\\Local\\Temp\\dsdk-330135\\manifest.cab May you simply have permission issues on file access.

Did you specify directories like in the docs? https://learn.microsoft.com/en-us/azure/aks/hybrid/offline-download-22h2#step-4-configure-the-deployment-onsite
I don't remember these files ending up in user profile temp folders e.g. -> did you use new / non existing folder names like in the docs?

OEI-Cgray · 2024-06-26T18:59:07Z

Connect-AzAccount -TenantId '6e0faf7f-ffe0-4345-bf85-9f1011650754' -Subscription '80f4e056-b7b8-4f94-b84a-469cdfd59b48'
 
Import-Module AksHci

$vnet = New-AksHciNetworkSetting -name k8sdhcpvnet -vswitchName "10.42.x.x" -vipPoolStart "10.42.24.40" -vipPoolEnd "10.42.25.240" -vlanid 24

Set-AksHciConfig -offlineDownload $true -mode full -stagingShare 'C:\clusterstorage\containerstorage\AKS-Staging' -imageDir 'c:\clusterstorage\containerstorage\AKS-Images' -cloudConfigLocation 'c:\clusterstorage\containerstorage\AKS-Config' -workingDir 'c:\clusterstorage\containerstorage\AKS-Working' -vnet $vnet -cloudservicecidr '10.42.0.129/23'

The folders were all empty when starting.

I don't see this function, which is called before Test-OfflineDownloadFiles, in the PSM: Get-ProductReleasesUptoVersion

I can get the initial version, but then it fails due to the non-existing (as far as I can tell) function:

PS C:\Temp> Get-AksHciVersion
VERBOSE: [06/25/2024 13:53:43] [AksHci] Initializing environment
VERBOSE: [06/25/2024 13:53:43] [AksHci] Importing Configuration
VERBOSE: [06/25/2024 13:53:43] [Moc] Importing Configuration
VERBOSE: [06/25/2024 13:53:43] [Moc] Importing Configuration Completed
VERBOSE: [06/25/2024 13:53:44] [Moc] Validating configuration
VERBOSE: [06/25/2024 13:53:44] [Moc] Get MOC Configuration
VERBOSE: [06/25/2024 13:53:44] [Moc]   Installation state is: NotInstalled
VERBOSE: [06/25/2024 13:53:44] [Kva] Importing Configuration
VERBOSE: [06/25/2024 13:53:44] [Kva] Importing Configuration Completed
VERBOSE: [06/25/2024 13:53:44] [Kva] Getting configuration for Kva
VERBOSE: [06/25/2024 13:53:44] [Kva]   Installation state is: NotInstalled
VERBOSE: [06/25/2024 13:53:44] [AksHci] Importing Configuration Completed
VERBOSE: [06/25/2024 13:53:44] [AksHci] Saving Configuration for Module AksHci to configuration file
VERBOSE: [06/25/2024 13:53:44] [Moc] Discovering configuration
VERBOSE: [06/25/2024 13:53:44] [Moc] Importing Configuration
VERBOSE: [06/25/2024 13:53:45] [Moc] Importing Configuration Completed
VERBOSE: [06/25/2024 13:53:45] [Moc] Validating configuration
VERBOSE: [06/25/2024 13:53:45] [Moc] Applying configuration
VERBOSE: [06/25/2024 13:53:45] [Moc] Saving Configuration for Module Moc to configuration file
VERBOSE: [06/25/2024 13:53:45] [Moc] Saving Configuration for Module Moc to configuration file
VERBOSE: [06/25/2024 13:53:45] [AksHci] Uninitializing environment
VERBOSE: [06/25/2024 13:53:45] [AksHci] Saving Configuration for Module AksHci to configuration file
1.0.23.10605
PS C:\Temp>     $startVersion = (Get-AksHciVersion)
VERBOSE: [06/25/2024 13:55:18] [AksHci] Initializing environment
VERBOSE: [06/25/2024 13:55:18] [AksHci] Importing Configuration
VERBOSE: [06/25/2024 13:55:18] [Moc] Importing Configuration
VERBOSE: [06/25/2024 13:55:18] [Moc] Importing Configuration Completed
VERBOSE: [06/25/2024 13:55:18] [Moc] Validating configuration
VERBOSE: [06/25/2024 13:55:18] [Moc] Get MOC Configuration
VERBOSE: [06/25/2024 13:55:18] [Moc]   Installation state is: NotInstalled
VERBOSE: [06/25/2024 13:55:19] [Kva] Importing Configuration
VERBOSE: [06/25/2024 13:55:19] [Kva] Importing Configuration Completed
VERBOSE: [06/25/2024 13:55:19] [Kva] Getting configuration for Kva
VERBOSE: [06/25/2024 13:55:19] [Kva]   Installation state is: NotInstalled
VERBOSE: [06/25/2024 13:55:19] [AksHci] Importing Configuration Completed
VERBOSE: [06/25/2024 13:55:19] [AksHci] Saving Configuration for Module AksHci to configuration file
VERBOSE: [06/25/2024 13:55:19] [Moc] Discovering configuration
VERBOSE: [06/25/2024 13:55:19] [Moc] Importing Configuration
VERBOSE: [06/25/2024 13:55:19] [Moc] Importing Configuration Completed
VERBOSE: [06/25/2024 13:55:19] [Moc] Validating configuration
VERBOSE: [06/25/2024 13:55:19] [Moc] Applying configuration
VERBOSE: [06/25/2024 13:55:20] [Moc] Saving Configuration for Module Moc to configuration file
VERBOSE: [06/25/2024 13:55:20] [Moc] Saving Configuration for Module Moc to configuration file
VERBOSE: [06/25/2024 13:55:20] [AksHci] Uninitializing environment
VERBOSE: [06/25/2024 13:55:20] [AksHci] Saving Configuration for Module AksHci to configuration file
PS C:\Temp>
PS C:\Temp>     $updateReleases = Get-ProductReleasesUptoVersion -Version $startVersion -moduleName $moduleName
Get-ProductReleasesUptoVersion : The term 'Get-ProductReleasesUptoVersion' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path
was included, verify that the path is correct and try again.
At line:1 char:23
+     $updateReleases = Get-ProductReleasesUptoVersion -Version $startV ...
+                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Get-ProductReleasesUptoVersion:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Elektronenvolt · 2024-07-02T17:58:30Z

At New-AksHciNetworkSetting I usually add settings for -ipAddressPrefix, -gateway, -dnsServers like its documented here: https://learn.microsoft.com/en-us/azure/aks/hybrid/kubernetes-walkthrough-powershell#step-2-create-a-virtual-network
Otherwise I don't have any connectivity to other networks or Internet. May that's the missing thing to get it working.

OEI-Cgray · 2024-07-02T20:18:49Z

@Elektronenvolt that's for a static IP setup. The existing powershell process that's failing the download is running using the OS's current DNS servers. These servers do have unrestricted outgoing internet access, and I've checked the firewall to ensure it's not blocking anything due to any of it's internal threat protection mechanisms.

https://learn.microsoft.com/en-us/azure/aks/hybrid/reference/ps/new-akshcinetworksetting#deploy-with-a-dhcp-environment-and-a-vlan

Elektronenvolt · 2024-07-03T14:17:48Z

Ok, I'll try the offline mode on one of my test setups - can combine it with a few other topics. But, try offline and offsite.
https://learn.microsoft.com/en-us/azure/aks/hybrid/offline-download-22h2#use-offline-download-to-install-offsite
You can download the images somewhere else and then copy the image to the stagingshare. It would be interesting if this works.

OEI-Cgray · 2024-07-11T18:48:03Z

To even get the version from another machine, I have to initialize it? Even though it won't be installed on this machine?

Get-AksHciVersion
[AksHci] This machine does not appear to be configured for deployment.
At C:\Program Files\WindowsPowerShell\Modules\akshci\1.2.4\AksHci.psm1:1242 char:9
+         throw [CustomException]::new(($([System.String]::Format([Syst ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], CustomException
    + FullyQualifiedErrorId : [AksHci] This machine does not appear to be configured for deployment.

Regardless of that, the function seems to do nothing:

Set-AksHciOffsiteConfig -version '1.0.23.10605' -stagingShare "F:\AKS-Staging" -Verbose -Debug
VERBOSE: [07/11/2024 11:46:20] [AksHci] Saving Configuration for Module AksHci to configuration file
VERBOSE: [07/11/2024 11:46:20] [AksHci] Saving Configuration for Module AksHci to configuration file
VERBOSE: [07/11/2024 11:46:20] [AksHci] Saving Configuration for Module AksHci to configuration file
VERBOSE: [07/11/2024 11:46:20] [AksHci] Saving Configuration for Module AksHci to configuration file
VERBOSE: [07/11/2024 11:46:20] [AksHci] Saving Configuration for Module AksHci to configuration file
VERBOSE: [07/11/2024 11:46:20] [AksHci] Saving Configuration for Module AksHci to configuration file
VERBOSE: [07/11/2024 11:46:20] [AksHci] Saving Configuration for Module AksHci to configuration file
VERBOSE: [07/11/2024 11:46:21] [AksHci] Saving Configuration for Module AksHci to configuration file

No files were downloaded, no errors.

OEI-Cgray · 2024-07-11T18:50:15Z

The offline download needs to occur from a Server OS with Hyper-V installed?

Initialize-AksHciNode
WinRM is already set up to receive requests on this computer.
WinRM has been updated for remote management.
WinRM firewall exception enabled.

WinRM service is already running on this machine.
WinRM is not set up to allow remote access to this machine for management.
The following changes must be made:

Enable the WinRM firewall exception.

WinRM has been updated for remote management.

WinRM firewall exception enabled.
The target of the specified cmdlet cannot be a Windows client-based operating system.
    + CategoryInfo          : DeviceError: (localhost:String) [Get-WindowsFeature], Exception
    + FullyQualifiedErrorId : WindowsClient_NotSupported,Microsoft.Windows.ServerManager.Commands.GetWindowsFeatureCommand
    + PSComputerName        : CGRAY03
Windows feature 'Hyper-V' does not seem to be present in this OS version and therefore cannot be enabled.
At C:\Program Files\WindowsPowerShell\Modules\Moc\1.2.1\Common.psm1:2614 char:9
+         throw [CustomException]::new($_, ([ErrorTypes]::IsInfraErrorF ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], CustomException
    + FullyQualifiedErrorId : Windows feature 'Hyper-V' does not seem to be present in this OS version and therefore cannot be enabled.

Elektronenvolt · 2024-07-15T09:14:36Z

By reading the offline docs to the end, it looks like the offsite feature is supporting upgrades only.
Anyway - back to the initial issue. I've tested the offline setup and ran into a hash mismatch, but it downloaded the images and I didn't run into a missing method issue.
May you run into an issue with old PS modules from previous releases? Try this clean up script in a fresh PS window with nothing loaded: #130 (comment)
And - choose a separate folder as stagingshare like in the docs, not a "clusterstorage" subfolder. Not sure if it makes a difference, but this is how it's in the docs.

I'll test it again on a fresh demo setup with offline mode. Using this feature from time to time and so far never had issues like that.

Elektronenvolt · 2024-07-25T21:49:40Z

@OEI-Cgray These steps for using the offline mode worked fine on a clean single node machine. Means, all AKS Arc folders deleted, all old PS modules removed, and in the user profile .ssh .azure .kube folders deleted.

Import-Module AksHci -Force

Connect-AzAccount -DeviceCode
Set-AzContext -Subscription "mysub"
$tenant = (Get-AzContext).Tenant.Id

Initialize-AksHciNode

$vnet = New-AksHciNetworkSetting -name mynet -vswitchName "myswitch" -k8sNodeIpPoolStart "ip" -k8sNodeIpPoolEnd "ip" -vipPoolStart "ip" -vipPoolEnd "ip" -ipAddressPrefix "ip-range" -gateway "ip" -dnsServers "ips"

$csvPath = 'C:\clusterstorage'
Set-AksHciConfig -offlineDownload $true -mode full -stagingShare c:\aksarcoffline -imageDir $csvPath\Images -workingDir $csvPath\ImageStore -cloudConfigLocation $csvPath\Config -vnet $vnet

$credential = Get-Credential
Set-AksHciRegistration -subscriptionId "mysub" -resourceGroupName "MY-RG" -TenantId $tenant -Credential $credential

Install-AksHci

If you do a $VerbosePreference = "continue" before Set-AksHciConfig, you can watch the image download in detail.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Using DownloadSDK stil fails to download files #373

[BUG] Using DownloadSDK stil fails to download files #373

OEI-Cgray commented May 16, 2024

Elektronenvolt commented May 16, 2024

OEI-Cgray commented May 16, 2024 •

edited

Loading

Elektronenvolt commented May 21, 2024

OEI-Cgray commented May 22, 2024 •

edited

Loading

Elektronenvolt commented May 23, 2024

Elektronenvolt commented Jun 3, 2024

OEI-Cgray commented Jun 5, 2024

Elektronenvolt commented Jun 13, 2024

OEI-Cgray commented Jun 22, 2024 •

edited

Loading

OEI-Cgray commented Jun 25, 2024 •

edited

Loading

Elektronenvolt commented Jun 26, 2024 •

edited

Loading

OEI-Cgray commented Jun 26, 2024

Elektronenvolt commented Jul 2, 2024

OEI-Cgray commented Jul 2, 2024

Elektronenvolt commented Jul 3, 2024

OEI-Cgray commented Jul 11, 2024

OEI-Cgray commented Jul 11, 2024

Elektronenvolt commented Jul 15, 2024

Elektronenvolt commented Jul 25, 2024

[BUG] Using DownloadSDK stil fails to download files #373

[BUG] Using DownloadSDK stil fails to download files #373

Comments

OEI-Cgray commented May 16, 2024

Elektronenvolt commented May 16, 2024

OEI-Cgray commented May 16, 2024 • edited Loading

Elektronenvolt commented May 21, 2024

OEI-Cgray commented May 22, 2024 • edited Loading

Elektronenvolt commented May 23, 2024

Elektronenvolt commented Jun 3, 2024

OEI-Cgray commented Jun 5, 2024

Elektronenvolt commented Jun 13, 2024

OEI-Cgray commented Jun 22, 2024 • edited Loading

OEI-Cgray commented Jun 25, 2024 • edited Loading

Elektronenvolt commented Jun 26, 2024 • edited Loading

OEI-Cgray commented Jun 26, 2024

Elektronenvolt commented Jul 2, 2024

OEI-Cgray commented Jul 2, 2024

Elektronenvolt commented Jul 3, 2024

OEI-Cgray commented Jul 11, 2024

OEI-Cgray commented Jul 11, 2024

Elektronenvolt commented Jul 15, 2024

Elektronenvolt commented Jul 25, 2024

OEI-Cgray commented May 16, 2024 •

edited

Loading

OEI-Cgray commented May 22, 2024 •

edited

Loading

OEI-Cgray commented Jun 22, 2024 •

edited

Loading

OEI-Cgray commented Jun 25, 2024 •

edited

Loading

Elektronenvolt commented Jun 26, 2024 •

edited

Loading