Build and publish Snap image for releases

[Doctor-love]

"Snap" is a popular package format that enables Linux users to install and run applications cross-distro.
In order to help potential users discover and use fq, it would be neat to automatically build and publish a snap during the release process.

fq utilizes GoReleaser in the "release" workflow, which according to the document supports building and publishing snaps.

I don't have any experience packaging snaps, so help would be greatly appreciated.

[Doctor-love]

How is confinement handled in Snaps? fq should be very easy to restrict (syscalls, network access, etc), except for which file paths it needs to access for probing.

[wader]

Yeap i think fq has very basic needs, read filsystem, read/write stdout and stderr maybe some ioctls for readline. All interaction with the os (exceptions for some profiling and fuzz things) are done thru the stdOS type https://github.com/wader/fq/blob/master/pkg/cli/cli.go#L36

Maybe first step i get a snap config with good confinement working and then look into goreleaser things?

I can do -rc tag releases once we have something to try

[Akaame]

name: fq
summary: jq for binary formats 
description: |
    fq is inspired by the well known jq tool and language and allows you to work with binary formats the same way you would using jq. 
    In addition it can also present data similar to a hex viewer, transform, slice and concatenate binary data, supports nested formats and has an interactive REPL with auto-completion.
version: git
grade: stable
base: core20

confinement: classic # or strict

apps:
  fq:
    command: bin/fq
    plugs:
      - home # should be enough
parts:
  fq:
    plugin: go
    source-type: git
    source: https://github.com/wader/fq

Do not have a linux system to test this on but should not something like this be enough for a basic golang based IO program?

[wader]

Thanks, i guess one maybe can test via docker or emulated if on some non-snap:able platform? what is needed more, some kind of packaging and publish? @Doctor-love any ideas?

@Akaame BTW I noticed your name didn't show up as contributor on github for fq, maybe git email configured differently somehow?

@Akaame Second BTW if your into binary tools there is a conference going on this weekend https://binary-tools.net/summit.html. I gave a talk about fq yesterday that was recorded so hopefully will show up somewhere soon.

[Akaame]

First BTW: I just realized that I committed to this repo using another git profile.
Second BTW: Thanks for the heads-up.

I just realized snapcraft is available on Mac and I already have multipass anyway. Will try if this works and if it does we can simply extend goreleaser.

i guess one maybe can test via docker or emulated if on some non-snap:able platform? what is needed more, some kind of packaging and publish? @Doctor-love any ideas?

I think we should not scope creep this particular issue but adding flatpak to out release outputs would also be nice in some other issue.

[Akaame]

I finally got snapcraft working on Mac. We have three problems

Snapcraft Go plugin has this line:

https://github.com/snapcore/snapcraft/blob/b81550376df7f2d0dfe65f7bfb006a3107252450/snapcraft/plugins/v1/go.py#L307

Which results in the following statement:

go install -p 2 -ldflags -linkmode=external ./...

However, we want:

go install -p 2 -ldflags -linkmode=external .

Luckily, we have the override-build option to handle the build/install however we want to.

build-environment:
      - CGO_ENABLED: '0'
override-build: |
      go install -p 2 -trimpath -ldflags "-s -w -linkmode=external" .

The second problem is that I had to populate build-environment and the line in override-build parameters to pass in our build environment variables, flags and ldflags. These values are already a part of goreleaser.yaml and I do not know how to confirm that GoReleaser Snapcraft plugin will indeed pass them along as environment variables so that we do not have to hardcode the flags as I did for override-build.

Third problem is we need to be able to push created snaps to snapstore for which there is this action:
https://github.com/snapcore/action-publish

But I have not got around to trying that out.

@wader do we have a Slack, Gitter or Discord for faster communication. We can pair up for this issue when you are free. Should not take more than an hour.

[wader]

I finally got snapcraft working on Mac. We have three problems

Snapcraft Go plugin has this line:

https://github.com/snapcore/snapcraft/blob/b81550376df7f2d0dfe65f7bfb006a3107252450/snapcraft/plugins/v1/go.py#L307

Which results in the following statement:

go install -p 2 -ldflags -linkmode=external ./...

However, we want:

go install -p 2 -ldflags -linkmode=external .

Luckily, we have the override-build option to handle the build/install however we want to.

build-environment:
      - CGO_ENABLED: '0'
override-build: |
      go install -p 2 -trimpath -ldflags "-s -w -linkmode=external" .

The second problem is that I had to populate build-environment and the line in override-build parameters to pass in our build environment variables, flags and ldflags. These values are already a part of goreleaser.yaml and I do not know how to confirm that GoReleaser Snapcraft plugin will indeed pass them along as environment variables so that we do not have to hardcode the flags as I did for override-build.

Ok hmm does snapcraft have some tools or sanity checks to make sure that the binaries will actually work in different envs? i'm not familiar with how it works but i guess similar to macOS apps (bring all your deps)?

Maybe possible to run goreleaser with some verbose logging to see what is going on?

Any ide how common snapcraft is on macOS? maybe a start it to just do it for linux for now?

Third problem is we need to be able to push created snaps to snapstore for which there is this action: https://github.com/snapcore/action-publish

Push not done by goreleaser?

@wader do we have a Slack, Gitter or Discord for faster communication. We can pair up for this issue when you are free. Should not take more than an hour.

Good idea. No official channel etc at the moment, any suggestion what to use?

[wader]

Maybe can sneak peak at some other go project that uses snapcraft?

[Akaame]

any suggestion what to use?

All of them are solid options Kubernetes Terraform etc uses slack. Rust's Tokio uses discord and most other projects use gitter. Totally depends on your taste.

Push not done by goreleaser?

Oh, I re-read the config yaml for goreleaser and it has a channel_templates parameter, so I am assuming it handles this for us. Still not clear how to pass our Snapstore API Token though (that we have to get after registering to the platform).

Any ide how common snapcraft is on macOS? maybe a start it to just do it for linux for now?

Snapcraft is the builder which is available on macos however snapd itself is not available so you can build snaps (thanks to multipass vm) but cannot use them. So our whole target with these is linux boxes.

Maybe possible to run goreleaser with some verbose logging to see what is going on?

That is also what I would suggest.

[wader]

any suggestion what to use?

All of them are solid options Kubernetes Terraform etc uses slack. Rust's Tokio uses discord and most other projects use gitter. Totally depends on your taste.

Hard to choose :) The GNU poke ppl are LiberaChat IRC, i'm there also but not very active. Think i would go for something web-able with archive support etc.

Push not done by goreleaser?

Oh, I re-read the config yaml for goreleaser and it has a channel_templates parameter, so I am assuming it handles this for us. Still not clear how to pass our Snapstore API Token though (that we have to get after registering to the platform).

Yeah had a look also, didn't find anything about tokens. But it do say at the bottom of the snapcraft docuementation:

GoReleaser will not install snapcraft nor any of its dependencies for you.

Is that CLI tools? does that mean token etc handled outside of goreleaser maybe?

Any ide how common snapcraft is on macOS? maybe a start it to just do it for linux for now?

Snapcraft is the builder which is available on macos however snapd itself is not available so you can build snaps (thanks to multipass vm) but cannot use them. So our whole target with these is linux boxes.

Aha ok that explains things. Ok but then maybe we can ignore supporting running goreleaser on macOS as i think it will only be used by the github release action? maybe can use docker container on macOS while testing?

[Akaame]

One of my OSS friends recommended Zulip.

https://waderfq.zulipchat.com/join/eiti77ojarx52b3fzek33mho/

[wader]

@Doctor-love have time to chat with me and @Akaame about snap stuff during the weekend?

[wader]

Found a project that uses goreleaser and snapcraft that seem to publish https://github.com/dosco/graphjin/blob/master/.github/workflows/build.yml. It installs snapcraft and logs in on the action host but how can the goreleaser/goreleaser-action@v2 action use the installed snapcraft? it runs in its own container or?

[wader]

Ah the goreleaser action is not a container https://github.com/goreleaser/goreleaser-action/blob/master/action.yml