JFrog Nuget Task Fails To Restore Package Due to perceived UTF-8 parsing error (Reopens #464)

[fourpastmidnight]

Describe the bug

This issue serves to re-open #464 since we are still experiencing this issue. When this ticket was first filed, we were using v2.9.0 of the JFrog Plugin. As noted below, the new plugin version is v2.9.2. Additionally, the build pipeline task this ticket is addressing is the .NET Core JFrog task. The original ticket was against v1.8.0. With ADOS extension v2.9.2, this task is versioned v1.9.2. The bug is still present in the updated version of this task.

#464 verbatim is repeated in this ticket immediately below.

Our existing Azure DevOps Server build pipelines use the built-in Nuget task to restore NuGet packages from our internally hosted Baget server. We are implementing JFrog Artifactory in a test environment.

To this end, I have updated one of our build pipeline to use the JFrog Nuget task to replace the built-in Nuget get task (to restore solution package dependencies).

When using the built-in task, all packages are restored with no issues. However, using the JFrog Nuget task, a warning is encountered: <Package> couldn't be parsed due to: XML syntax error on line 1: invalid UTF-8. Skipping the package dependency. Furthermore, the task completes successfully, which is another problem with this task since, the build is downloading dependencies which, obviously without all dependencies, the build could not possibly be successful. So this type of issue, even if deemed sever enough to continue to emit a warning, should in fact be an error and fail the build.

Current behavior

After replacing the built-in Nuget task in order to perform package restores in preparation for a build, the JFrog Nuget task produces the following warning and completes successfully:

Executing JFrog CLI Command:

D:\Agent01\_work\_tool\_jf\current\jf.exe nuget restore -PackagesDirectory "D:\\Agent01\\_work\\3\\packages\\" -Verbosity "Normal" --build-name="My Application - Test Artifactory Integration" --build-number="My-Application_$_My-Application_Main_App_2023-12-14T12.50.24"
13:02:55 [Info] Running nuget...
All packages listed in packages.config are already installed.
13:03:00 [Warn] Package: microsoft.web.infrastructure:1.0.0 couldn't be parsed due to: XML syntax error on line 1: invalid UTF-8 . Skipping the package dependency.
13:03:01 [Info] nuget finished successfully.

It should be noted that no such warning or error is ever emitted by the built-in NuGet task and my build completes successfully. However, with the JFrog Nuget task, this warning is emitted, skipping the package dependency; and because this step does not outright fail, an attempt to build proceeds to occur which ultimately ends in failure due to the missing dependency.

Reproduction steps

This is probably the result of the fact that the *.nuspec file embedded in the NuGet package does not have a UTF-8 BOM (or, perhaps the other way around--or perhaps an invalid one?).

1. Attempt to produce a NuGet package whose *.nuspec file contains invalid UTF-8 on line 1 (again, probably something related to the BOM)
2. Source the package from Artifactory. In our case, the package lives in a Baget Nuget server which is configured as a remote repository in Artifactory, surfaced via a virtual repository.
3. Consume the dependency via a .NET Framework project
4. Attempt to restore packages using the JFrog Nuget task
5. The task will emit the warning, but otherwise complete "successfully". The Build of the project will fail due to missing dependencies.

Expected behavior

The real world is messy. Bad data. I expect that the JFrog Nuget task would function exactly like the built-in Nuget task and that all project dependencies would be restored.

I further expect that any issue with restoring a dependency (especially one more severe than a UTF-8 issue, such as physically unable to write package files to some location) would result in not just a warning, but an error such that the JFrog Nuget step would fail, resulting in a failed build even before attempting an actual build which would never succeed due to missing dependencies!

Azure DevOps extension name and version

2.9.2

JFrog CLI version

2.50.4

Operating system type and version

Windows Server 2022

JFrog Artifactory version (if relevant)

7.63.7

JFrog Xray version (if relevant)

N/A

JFrog Distribution version (if relevant)

N/A

[fourpastmidnight]

Just an additional detail to add to this. I learned after initially submitting #464 that the package in question: microsoft.web.infrastructure-1.0.0, it's NUSPEC file is actually in UTF-16 format even though the XML prolog specifies it's UTF-8.

While this is an issue with the package itself, Microsoft is unlikely to fix this issue with the package. And therefore, Jfrog needs to deal with it.

I would expect that the best way to deal with this is to check that the text can be read as UTF-8, failing that, perform a check to see if the package spec can be read as UTF-16, and where there's a mismatch, emit at least a Warning (and hopefully, it is possible to "ignore" warnings). It's good to alert that there's such an issue, but in cases where the end-user is not in control of the package, it needs to be ignorable; and at the end of the day, this step needs to work EXACTLY like the built-in Dot Net Build restore step.

[sverdlov93]

Hi @fourpastmidnight ,
can you share the nuspec file?

[fourpastmidnight]

It's not my package! It's Microsoft's. You can easily get it from Nuget.org and extract it.

[fourpastmidnight]

Awesome, glad to see the PR associated with this issue has been merged! Has this been formally released (also asked on the PR--answering in either place will be fine with me! 😄)?

[sverdlov93]

@fourpastmidnight
https://github.com/jfrog/jfrog-azure-devops-extension/releases/tag/2.9.3 is Released and should fix the above issue!

[fourpastmidnight]

@sverdlov93 Thank you!!

[fourpastmidnight]

I can confirm that in the lastest JFrog CLI version, v2.53.2, either the resolution to #464 or the PR associated with this issue fixed the problem. I thought it wasn't working with the fix for #464, but now I'm not so sure since the JFrog Azure DevOps Server extension doesn't bundle the latest JFrog CLI with the release!—and the issue lies in the CLI, not the JFrog Azure DevOps extension itself! So this is very hard for customers to understand and JFrog really should change how it composes the release of the extension to eliminate this distinction. (Not to mention the fact that the extension contains a task to get any version of the CLI that you want.... so, why not ship the latest? Someone can always say, well, no, use v2.33 or whatever instead. This just causes _so much confusion, and the last thing you want is confusing CI/CD tooling. It's hard enough already!)