Add checksums (SHA256) to GitHub releases for all files

[o-l-a-v]

Related command

Add checksums (SHA256) to releases for all files, be it files hosted on GitHub (MSI, source code), and files from azcliprod.blob.core.windows.net (ZIP, MSI etc.).

Is your feature request related to a problem? Please describe.

• Integrity, know that what you downloaded hasn't been corrupted.
• Make it faster for package managers like Scoop to get checksums for new releases without having to download the artifact(s).
  • Which is an even bigger pain with Azure CLI because azcliprod.blob.core.windows.net is so slow.
    • 2021-09-03 [Feature Request] Use CDN to distribute MSI package #19447
    • 2024-04-30 Add ZIP release to GitHub releases too #28856

Describe the solution you'd like

Add a list to GitHub releases with SHA256, like Beaver Notes does:

• https://github.com/Beaver-Notes/Beaver-Notes/releases/tag/3.5.0

Describe alternatives you've considered

None.

Additional context

None.

[yonzhan]

Thank you for opening this issue, we will look into it.

[jiasli]

Hi @o-l-a-v, I noticed https://github.com/Beaver-Notes/Beaver-Notes/releases/tag/3.5.0 puts SHA256s in the release description:

https://github.com/PowerShell/PowerShell/releases/tag/v7.4.4 uses a different format:

How does Scoop extract this information from plaintext?

[o-l-a-v]

I have a PR on Beaver Notes for Scoop that shows the info Scoop needs to fetch the SHA256 checksums:

• https://github.com/ScoopInstaller/Extras/pull/13661/files

Here's the manifest for pwsh, looks like it fetches SHA256 info from release info too:

• https://github.com/ScoopInstaller/Main/blob/master/bucket/pwsh.json

Here's info on the Scoop app manifest:

• https://github.com/ScoopInstaller/Scoop/wiki/App-Manifests

Here's Scoop main manifest repos, see if you can find more examples there:

• main (command line programs only): https://github.com/ScoopInstaller/Main/tree/master/bucket
• extras (can be GUI too): https://github.com/ScoopInstaller/Extras/tree/master/bucket

---

Edit: And I believe the logic for getting new versions is found inside here:

• https://github.com/ScoopInstaller/Scoop/blob/master/bin/checkver.ps1

Which calls Invoke-AutoUpdate from here:

• https://github.com/ScoopInstaller/Scoop/blob/master/lib/autoupdate.ps1

[jiasli]

Thanks @o-l-a-v, but I think you misunderstand my question.

In other words, how is 6461dd3fda39fc65e30c7642f863b9e1dabe32885043094e1d8a79dffcef1dcb extracted from https://github.com/PowerShell/PowerShell/releases/tag/v7.4.4 to https://github.com/ScoopInstaller/Main/blob/75c4fa28f734a4849633a8bd920013bf7d93911a/bucket/pwsh.json#L16 ? Is this a manual process? Or is there a standard format? I would like to know which format you want us to use to provide the SHA256s.

BTW, I noticed Scoop computes the SHA256 of https://azcliprod.blob.core.windows.net/zip/azure-cli-2.63.0-x64.zip by itself: https://github.com/ScoopInstaller/Main/blob/75c4fa28f734a4849633a8bd920013bf7d93911a/bucket/azure-cli.json#L13

[o-l-a-v]

As the feature request states: Scoop can calculate SHA256 by downloading the artifact. But if checksum is already made available and Scoop is told were to look, it does not have to 1) download the artifact and 2) calculate hash.

Scoop has logic to autoupdate manifests. If we add hash ("autoupdate":{"hash": {}}) property to the manifest JSON, Scoop will try to find the checksums depending on the logic. So for Beaver-Notes https://github.com/ScoopInstaller/Extras/pull/13661/files:

{
    ...
    "autoupdate": {
        "architecture": {
            "64bit": {
                "url": "https://github.com/Beaver-Notes/Beaver-Notes/releases/download/$version/Beaver-notes.$version.portable.exe#/dl.7z"
            },
            "arm64": {
                "url": "https://github.com/Beaver-Notes/Beaver-Notes/releases/download/$version/Beaver-notes.$version.portable.arm64.exe#/dl.7z"
            }
        },
        "hash": {
            "url": "https://github.com/Beaver-Notes/Beaver-Notes/releases/tag/$version",
            "regex": "$sha256.*?$basename"
        }
    }
}

And for pwsh https://github.com/ScoopInstaller/Main/blob/master/bucket/pwsh.json:

{
    ...
    "autoupdate": {
        "architecture": {
            "64bit": {
                "url": "https://github.com/PowerShell/PowerShell/releases/download/v$version/PowerShell-$version-win-x64.zip"
            },
            "32bit": {
                "url": "https://github.com/PowerShell/PowerShell/releases/download/v$version/PowerShell-$version-win-x86.zip"
            },
            "arm64": {
                "url": "https://github.com/PowerShell/PowerShell/releases/download/v$version/PowerShell-$version-win-arm64.zip"
            }
        },
        "hash": {
            "url": "$baseurl/hashes.sha256"
        }
    }
}

[o-l-a-v]

There are som find_hash_in_x functions inside here:

• https://github.com/ScoopInstaller/Scoop/blob/master/lib/autoupdate.ps1

You could probably also just add checksum files to the GitHub release instead of adding it as plain text, if you prefer that.

[jiasli]

Thank you for the detailed explanation. find_hash_in_textfile looks a little bit fragile. I think creating a hashes.sha256 with sha256sum --binary * is a more formal and reliable implementation. We will consider this as a feature request.

BTW, we can reuse the code from https://github.com/PowerShell/PowerShell/blob/a1774fd9332925f7635e0832b64b2d158e3a3745/.pipelines/templates/release-githubtasks.yml#L88-L104

[jiasli]

I noticed today that Python provides very convenient functions to compute file hashes:

https://docs.python.org/3/library/hashlib.html#file-hashing

import io, hashlib, hmac
with open(hashlib.__file__, "rb") as f:
    digest = hashlib.file_digest(f, "sha256")

digest.hexdigest()  

[jiasli]

Our release pipeline is on Linux, so the below command will do the job:

sha256sum --binary azure-cli* > hashes.sha256

hashes.sha256 will look like

723cf2ac1d252ee086841dece0fff10538e15926d2c7ed1085d02c0f82b7393e *azure-cli-2.67.0-x64.msi
b8b31530939d8016ff23da4d2196a4286e9db6477117733c770b805b6fa64162 *azure-cli-2.67.0-x64.zip
c4cdd13157968052bc2ffeecb4459d466de95d4b9ab8194963ab601d9f613c44 *azure-cli-2.67.0.msi

* indicates binary mode.

According to https://manpages.ubuntu.com/manpages/noble/man1/sha256sum.1.html

Note: There is no difference between binary mode and text mode on GNU systems.

But it is still better to specify --binary to align with hashes.sha256 from https://github.com/PowerShell/PowerShell/releases

[jiasli]

@o-l-a-v, we have updated the latest release (2.67.0) to include SHA256 hashes for the release artifacts in the release description: https://github.com/Azure/azure-cli/releases/tag/azure-cli-2.67.0

This is only experimental, so please let us know if this suits your needs. If you want us to change its format, feel free to let us know. 😊

Also, do you want us to publish a hashes.sha256 file? If so, any preference on the file name?

[o-l-a-v]

Nice! That should make it easy for both humans and automation like Scoop to find the checksums.

I think I prefer one <filename.ext>.sha256 file per artifact, but one file for all hashes like PowerShell do works too:

• https://github.com/PowerShell/PowerShell/releases/tag/v7.4.6

[jiasli]

I am glad it works for you. 😊

Actually, I noticed https://github.com/Beaver-Notes/Beaver-Notes/releases doesn't have a dedicated file for hashes.

Does putting hashes only in release description work for you? Or is it still necessary for us to release dedicated hash file(s)?

Also, there doesn't seem to be an industry standard on what should be inside that file.

[o-l-a-v]

Hashes in description only works.

If creating checksum file(s) it'd be great it they work with sha256sum:

• https://www.man7.org/linux/man-pages/man1/sha256sum.1.html

sha256sum --check --ignore-missing checksums.sha256

Would be great if Microsoft had a standard across projects which can be managed by package managers, like Azure CLI, Bicep, PowerShell etc. Or if the GitHub releases API added checksum to each artifact.

---

Edit: Example on how to mimic SHA256SUM on Windows with PowerShell:

# Assets
$FilePath = [string] 'C:\Users\olavb\Desktop\azure-cli-2.67.0-x64.zip'
$OutputFilePath = [string] $FilePath + '.sha256'

# Using PowerShell native
<#
    Examples where it's used:
    * <https://github.com/pbek/QOwnNotes/blob/main/build-systems/github/windows/build-zip.ps1>
#>
Out-File -Encoding 'utf8' -Force -FilePath $OutputFilePath -InputObject (
    [string]::Concat(
        (Get-FileHash -Path $FilePath -Algorithm 'SHA256').'Hash',
        ' *',
        $([System.IO.FileInfo]($FilePath)).'Name'
    )
)

[jiasli]

@o-l-a-v, thanks for the additional information. As I mentioned in #29656 (comment), our pipeline is on a Linux agent, so a single sha256sum --binary azure-cli* > hashes.sha256 command will do the job.

There is even no convention on the hash file name: hashes.sha256 vs checksums.sha256. Also, as PowerShell does, if we add hashes.sha256, the hashes.sha256 needs its own hash (https://github.com/PowerShell/PowerShell/releases/tag/v7.4.6). I prefer not to complicate things too much.

As "Hashes in description only works", I will keep it this way. If a dedicated hash file is needed someday, please don't hesitate to create a new issue.

[o-l-a-v]

Perfect, thanks @jiasli. 😊