Add SLSA provenance via build script

[pnacht]

Hey, I'm Pedro and I'm working on behalf of Google and the Open Source Security Foundation (OpenSSF). Given the significant increase in supply-chain attacks, the OpenSSF is focused on improving the security of the open-source ecosystem as a whole. For example, #792 was based on feedback from Scorecards, an OpenSSF tool.

The OpenSSF has also developed the SLSA specification for projects to attest to a published artifact's provenance, allowing its consumers to ensure that the artifact comes from a trusted source. There are also GitHub workflows to securely generate this provenance and CLI tools to verify an artifact's authenticity.

Given how Jackson is almost synonymous with JSON in the Java ecosystem, the OpenSSF has placed Jackson on its list of the 100 most important open-source projects. I'd therefore like to offer to help jackson-core incorporate SLSA into its deploy workflow.

Would you be interested in a PR to adopt SLSA?

[cowtowncoder]

Hi @pnacht! I don't know much about SLSA, but in principle it sounds like something that could prove useful.
Usually the biggest question is effort involved in workflow; how much change is needed etc.
If there is little additional overhead from maintainer perspective I think we'd be +1.

So I think we can definitely start discussion. I hope to read a bit more about SLSA over the holidays.

Looking forward to collaboration!

[pnacht]

Hey @cowtowncoder, good to hear!

Hope you had a nice Thanksgiving and some time to recover!

Java provenance is currently generated using the SLSA generic generator. This involves adding two "parts" to the workflow: one step immediately after the deploy to calculate the hashes for the relevant jars, and a separate job (for security reasons) to generate the provenance.

If you'd be interested, I've actually already made the requisite changes in a fork, I'd be happy to submit a PR for you to take a look.

---

Or, if you'd first like to take a look at how the workflow works, see a successful run here. That run was on a slightly different version of the workflow than would be in the PR since my fork naturally can't deploy, so it instead uploads the jars as a workflow artifact.

The .intoto.jsonl file is the provenance. It's a JSON file, but the most important value ("payload") is a base64'd json blob. If you want to glance at it, I'd suggest using jq:

jq -r '.payload | @base64d' jackson-core-2.14.0-SNAPSHOT.intoto.jsonl | jq

There's a lot of information there, so if you want, I can write up a little summary of the most relevant bits.

[cowtowncoder]

A summary would be nice. I think PR would also make sense as it'd be a bit easier to see how CI changes look like (and how is triggering done).

Thank you!

[pnacht]

@cowtowncoder I've submitted the PR!

So, using the .intoto.jsonl provenance file from the run I mentioned earlier, we can run the following command (or use the Rekor link mentioned at the very bottom):

jq -r '.payload | @base64d' jackson-core-2.14.0-SNAPSHOT.intoto.jsonl | jq

This outputs the entire provenance, the most relevant parts of which are:

• "subject" identifies both jars (jackson...jar and jackson...-sources.jar), along with their SHAs
• "predicate" shows a bunch of information regarding how the jars were built, most interestingly:
  • "builder" shows the provenance was generated with the "generic generator" (used for languages without a SLSA builder)
  • "invocation" shows the jars were created from my fork's 2.14 branch, at a specific commit, using the main.yml workflow
• "materials" at the end also describes the "subjects", but by pointing at the "ingredients" used to generate them, not just their names. In the case of jackson-core, that's just the repo itself.

{
  "#": "...",
  "subject": [
    {
      "name": "jackson-core-...-sources.jar",
      "digest": {
        "sha256": "4be0c..."
      }
    },
    {
      "name": "jackson-core-....jar",
      "digest": {
        "sha256": "f7f0e..."
      }
    }
  ],
  "predicate": {
    "builder": {
      "id": "https://.../generator_generic_slsa3.yml@refs/tags/v1.2.2"
    },
    "buildType": "https://.../slsa-github-generator/generic@v1",
    "invocation": {
      "configSource": {
        "uri": "git+https://github.com/pnacht/jackson-core@refs/heads/2.14",
        "digest": {
          "sha1": "51259..."
        },
        "entryPoint": ".github/workflows/main.yml"
      },
      "#": "...",
    },
    "materials": [
      {
        "uri": "git+https://github.com/pnacht/jackson-core@refs/heads/2.14",
        "digest": {
          "sha1": "51259..."
        }
      }
    ]
  }
}

To ensure this information can't be falsified, the provenance is also published to an append-only transparency log. For example, looking at the compiled jar: https://rekor.tlog.dev/?hash=f7f0ee170a600c1f6d34916990c51bc88dbbd0dbb216351cbbdeed4ff3de8e6e

Here we can again see the same provenance attestation (click "Attestation" at the end) as well as a bunch of other information confirming this "release" was published via the "expected" channel (a GitHub Action on my fork).

[cowtowncoder]

Doh! Forgot to use the release script for 2.15.0-rc1.... need to make sure to use it for 2.15.0-rc2 just to verify before 2.15.0 final.

[pnacht]

Happens to the best of us!

…

On Sat, Mar 18, 2023 at 5:27 PM Tatu Saloranta ***@***.***> wrote: Doh! Forgot to use the release script for 2.15.0-rc1.... need to make sure to use it for 2.15.0-rc2 just to verify before 2.15.0 final. — Reply to this email directly, view it on GitHub <#844 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADUEE3XWKWBUR65OBWA632TW4YLBDANCNFSM6AAAAAASJNLA6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cowtowncoder]

@pnacht Ok, hmmh. So, we are missing a Secret for workflow to access it seems:

https://github.com/FasterXML/jackson-core/actions/runs/4546942534

with this being the error:

    gpg: no default secret key: No secret key
    gpg: signing failed: No secret key

wonder what would be the fix here? Locally it... "just works".

EDIT I deleted jackson-core-2.15.0-rc2 tag. Depending on timing, may need to push this one, too, manually. But will wait a bit first.

[cowtowncoder]

Oh. I have never uploaded GPG secret since I've never needed it in Github action. I only have them locally....

Hopefully I'll be able to do what this says:

https://stackoverflow.com/questions/61096521/how-to-use-gpg-key-in-github-actions

(trying it out now)

[cowtowncoder]

Looks like things get bit further, now failing with:

Error: ROR] Failed to execute goal org.apache.maven.plugins:maven-gpg-plugin:3.0.1:sign (sign-artifacts) on project jackson-core: Unable to decrypt gpg passphrase: org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException: java.io.FileNotFoundException: /home/runner/.m2/settings-security.xml (No such file or directory) -> [Help 1]

[pnacht]

I unfortunately can't test this for you, but my understanding is that actions/setup-java with the arguments you gave creates a local settings.xml that looks something like

<settings  ...>
  <servers>
    <server>
      <id>maven</id>
      <username>${env.CI_DEPLOY_USERNAME}</username>
      <password>${env.CI_DEPLOY_PASSWORD}</password>
    </server>
    <server>
      <id>gpg.passphrase</id>
      <passphrase>${env.MAVEN_GPG_PASSPHRASE}</passphrase>
    </server>
  </servers>
</settings>

You then need to define these environment variables yourself. Try making this change to the workflow:

      - name: Perform release
        # The following command will only succeed if the preparation was done via the
        # release.sh script.
        run: ./mvnw -B -q -ff -ntp release:perform -DlocalCheckout=true
        env:
          CI_DEPLOY_USERNAME: ${{ secrets.CI_DEPLOY_USERNAME }}
          CI_DEPLOY_PASSWORD: ${{ secrets.CI_DEPLOY_PASSWORD }}
          MAVEN_GPG_PASSPHRASE: ${{ secrets.MAVEN_GPG_PASSPHRASE }}  # or whatever you called it in the secrets

By placing the env within the "Perform release" step, you ensure that it isn't available and at risk of leaking from any other step.

This is similar to what you did for the SNAPSHOT releases in main.yml:

jackson-core/.github/workflows/main.yml

Lines 54 to 60 in 3f889b9

	- name: Deploy snapshot
	if: github.event_name != 'pull_request' && matrix.java_version == '8' && endsWith(steps.projectVersion.outputs.version, '-SNAPSHOT')
	env:
	CI_DEPLOY_USERNAME: ${{ secrets.CI_DEPLOY_USERNAME }}
	CI_DEPLOY_PASSWORD: ${{ secrets.CI_DEPLOY_PASSWORD }}
	# MAVEN_GPG_PASSPHRASE: ${{ secrets.MAVEN_GPG_PASSPHRASE }}
	run: ./mvnw -B -q -ff -DskipTests -ntp source:jar deploy

[cowtowncoder]

Thank you @pnacht that makes sense. Was about to try something along these lines, but this definitely helps confirm the direction.

[cowtowncoder]

@pnacht And it works! Had some trouble thanks to Emacs & tabs, but otherwise just adding ENV declarations for Maven to find did the trick (and had earlier figured out gpg secret addition for specific repo).

So this could conceivably be used for jackson-annotations and jackson-databind too I suppose.

[cowtowncoder]

@pnacht Hmmh. Ok, so publishing of artifacts did succeed, but workflow had some remaining issues that might affect provenance checking:

https://github.com/FasterXML/jackson-core/actions/runs/4548247411

so I thought you might be interested -- not sure if it's a real fail or not.

[pnacht]

Hrmm, yeah, it didn't generate the provenance. Looks like it couldn't find the jars...

Oh God, does release:perform delete the jars when it's done?

[cowtowncoder]

@pnacht Ok, that's too bad.

As release:perform, I wouldn't think it deletes anything. I think I've had Javadocs generated and usable after Maven release plug-ins work, locally.

[pnacht]

Hrmm, then it's weird, not sure what happened. Before sending #896, I'd managed to create the provenance after ./mvnw [...] package (see this run and the workflow at the time, which is basically the same as the current one other than the Maven command).

Could you add the following to the workflow at the end of the "Generate hash" step (or I can send a PR, if you prefer)? Need to understand the file structure after the release...

ls 
ls ./target
echo "$ARTIFACT_NAME"

We can see what's happening when you create -rc3 or the final release (super annoying we can't debug this without creating actual releases...).

[cowtowncoder]

@pnacht Created #981 feel free to comment or create better PR. Happy to merge something for rc3 which is planned to be the last RC for 2.15.0.

[cowtowncoder]

@pnacht Ok, so failure:

https://github.com/FasterXML/jackson-core/actions/runs/4703096295/jobs/8341172237

 sha256sum: 'jackson-core-2.15.0-rc3*.jar': No such file or directory
DEBUG: After SLSA hash generation we have:
checkout
ls: cannot access './target': No such file or directory

[pnacht]

Ah! release:perform created a directory ./checkout and then did everything in there. This is unlike mvn package which already has the source code right there and therefore naturally runs in the current directory.

The following should now work, I believe:

      - name: Generate hash
        id: hash
        run: |
          ARTIFACT_NAME="$( \
            ./mvnw help:evaluate \
              -Dexpression=project.artifactId -q -DforceStdout)-$( \
            ./mvnw help:evaluate \
              -Dexpression=project.version -q -DforceStdout)"
          echo "artifact_name=$ARTIFACT_NAME" >> "$GITHUB_OUTPUT"

          cd ./checkout/target
          echo "hash=$( \
            sha256sum $ARTIFACT_NAME*.jar | \
            base64 -w0 \
          )" >> "$GITHUB_OUTPUT"

          echo "DEBUG: After SLSA hash generation we have:"
          echo "DEBUG: ARTIFACT_NAME = $ARTIFACT_NAME"
          ls ./checkout
          ls ./checkout/target

Hopefully the final release will actually have provenance! And thanks a bunch for keeping at this!

[cowtowncoder]

Alas, still no go:

https://github.com/FasterXML/jackson-core/actions/runs/4780477741

for 2.15.0 release. With:

/home/runner/work/_temp/5fe1eec1-d6cb-4e0d-9200-b63399e5edbb.sh: line 8: cd: ./checkout/target: No such file or directory

so I guess directory isn't quite right still.

[pnacht]

Urgh... sorry for this.

I'm going to ramp up a repo to publish a toy package and figure this out without haranguing you. Didn't think there'd be so many trapdoors...

[cowtowncoder]

@pnacht Np, debugging GH actions is tricky. Just glad you are helping here, much appreciated!

[cowtowncoder]

@pnacht Sorry to bug you, but I was wondering if you might have any chance to look into this. I pushed 2.15.1 but turns out there were a few things I need to solve so will probably release 2.15.2 relatively soon. Although lack of SLSA content hashes is not a blocker by any means, it'd be nice to get it working at some point and I can wait for 2.15.2 if there was progress.
And if not then I can just proceed otherwise.

[pnacht]

Hey @cowtowncoder. Don't apologize! I've been meaning to work on this but admit I haven't had the time to do so yet. It's absolutely on my to-do list, but I've just always had something else appear on my plate. I'll unfortunately also be offline next week, so I don't think I'll be able to get back to you with a solid answer in the short term.

So I do suggest you release 2.15.2 without the attestation... I legitimately feel bummed out that this has been such a bumpy ride.

[cowtowncoder]

@pnacht That sounds like a plan. We'll get it done when we get there -- I just wanted to make sure I didn't push something only to find waiting a bit would have allowed improvements.

[pnacht]

Hey @cowtowncoder, sorry for the radio silence. I forgot to let you know that I'd put this on pause because we had some work going on for a Maven-specific SLSA builder, which has just been released. I'm going to take a look at it to see if it's really applicable for your use-case.

In the meantime, I'll keep sending any other security improvements I detect for the project.

[cowtowncoder]

Ok. Thank you for the update @pnacht ! I'll re-open this issue, easier to remember.