Build suitable base images.

[zenhack]

We will want some base docker images that folks can work from when developing apps; probably something that comes with sandstorm-http-bridge, and we may want to try to replicate vagrant-spk's stacks.

[zenhack]

Started working on these in the base-images directory.

[zenhack]

The sandstorm-http-bridge base image is now published on dockerhub. There's an example under examples/ using it as a base to build a trivial python app (we could probably abstract this into a suitable python-specific base image itself).

@JamborJan, @ocdtrekkie, would appreciate opinions on the example.

[JamborJan]

Hey @zenhack, awesome work thanks! I’ll take a look at it in detail next week. I really like the approach. One big point for me is the possibility to automate stuff which should be much easier with Docker than classic VMs which are used with vagrant-spk.

[zenhack]

Great; I look forward to hearing.

[ocdtrekkie]

I just wanted to say I am super interested in testing this but have a lot on my plate elsewhere, and don't know if I will look at it extremely quickly. :| My backlog is getting long.

[zenhack]

Know the feeling; no worries if it doesn't happen.

[JamborJan]

I was not able to test too much yet, only to think about some points and one issue I see is, that sandstorm packages need really specific adjustments to work properly and even mor if they should work great. Fro example the wordpress port is running with SQLLite, some mandatory plugins and a quite a lot of code adjustments. Inn this case I didn't yet find the easier process to build the app compared to vagrant. Both are exhausting right now. I still would like to find a way automating the many manual steps and right now it feels like this would be "more possible" with docker.

[zenhack]

Yeah, I don't think there's really anything docker can do for us to alleviate the pain points of sandstorm being a fundamentally weird environment to run in. If there were a silver bullet for porting legacy apps to a cap security environment we might all be running secure OSes on our laptops :(.

What I think docker can give us is:

• A larger library of images to work from.
• Faster build times.
• The potential for small images without spk-dev's file system monitoring, which makes reproducibility really hard. Results building off of the alpine image have been encouraging. Doing this with vagrant doesn't work as well, because being a VM there's a lot of stuff that has to be there (kernel/bootloader/etc) that the sandstorm app doesn't need, so stripping that out is annoying.

I also think not being tied to a specific base image will make it easier to experiment with useful images to build off of. There's nothing about Vagrant that makes that harder than with Docker, but vagrant-spk is kinda hard-coded to the one image.

I think docker-spk is really only a small piece of the puzzle here.

[ocdtrekkie]

@zenhack How attached is vagrant-spk to Ubuntu besides the setup scripts themselves when you're first making a package? Most of the code which actually manipulates your package should be pretty release-independent.

[zenhack]

The image is actually debian, not Ubuntu. But yeah, the only real hard requirement is that sandstorm is installed in the VM. The stacks themselves are more tied to debian (and jessie in particular), as the use apt to install (specific versions of) packages. But docker-spk doesn't really help with that.

I don't think getting vagrant-spk to be able to use alternate images would be major surgery.

[ocdtrekkie]

Right, Debian. Stretch now though, not Jessie. ;)

So the question would be do you:

• Somehow move the box selection and setup to the stack, so you can define specific stacks as using specific boxes?
• Somehow modify all of the stacks to either use version-independent repositories, or include logic that selects them intelligently? (I presume, for instance, rather than changing out jessie for stretch in the repo URLs, we could've had it check which version it needed.) Though that would likely require we individually add support for each possible flavor of Linux we want to support, to each and every stack.

This is mostly pointless talk for a docker-spk issue, but it's interesting. :P

[zenhack]

My take would be to make the Vagrantfile part of the stack. With docker-spk our "stacks" are just vanilla docker images; you put FROM baseimage in a Docker file just like you would if you were building a normal docker container. One of the nice things that docker gives us is that staged builds are really easy, and much more efficient than similar logic would be with a VM, so rather than having global-setup.sh, setup.sh, build.sh, with fiddly rules about when each one gets run, you just have sequential RUN commands, each of which creates an intermediate image, and when you do docker build, it starts from the first one that was actually modified.

I really think the main advantage of using docker is that the existing tooling around docker images is easier to work with, and faster.

I'm also making a few different design decisions that I think are mostly independent of vagrant vs. docker:

• No filesystem monitoring. Using a small base images is "good enough" I think, and I don't like the lack of reproducability that comes with spk dev's magic file selection.
• No dependency on sandstorm itself. This involved re-implementing bits of the spk tool, but wasn't ultimately that hard (spk dev would be a little trickier). This is nice in that we get portability without having to deal with the dev tools being in our images, which simplifies the logic of stripping out unneeded bits in the final image.

[JamborJan]

Hey @zenhack, I was starting to test abit around but I'm blocked quite quickly. I tried to follow your instructions:

git clone https://github.com/zenhack/docker-spk
cd docker-spk
export PATH=$(pwd)/docker-spk:$PATH
cd ..
mkdir my_new_project
cd my_new_project
docker-spk init
docker save my-image > my-image.tar
docker-spk pack -imagefile my-image.tar

But I cannot convince my command line to know `docker-spk.

My plan is to build the gitlab sandstorm app with docker as the current manually build app is super complex and has a ton of dependencies.

Thanks for any comments and hints!

[ocdtrekkie]

@JamborJan I assume you need to create a symbolic link to the docker-spk executable in your bin. This is from vagrant-spk's instructions:

sudo ln -s $PWD/vagrant-spk /usr/local/bin

[zenhack]

Did you compile it? I don't see that in your command listing. Unlike vagrant-spk, docker-spk is written in a compiled language. See Building in the readme.

[JamborJan]

Ahh okay @zenhack, I didn't compile it. I thought that was the step for when I want to hack the tool. Is it possible to include a pre-build version in the git repo?

[zenhack]

Would probably be good to ship binaries too; part of why I've delayed is because I want to work out how to make the builds reproducible. I'm a bit uncomfortable telling folks to run an unreproducible blob I've created with access to their user account -- too much responsibility. But I'd definitely like to make that happen.

[JamborJan]

I completely understand your point. As I don’t have a ready to use go-environment I cannot simply build the app and as time is rare and costly: can you please provide a build? I know all the risks and I’m willing to take them in this case. If you don’t want to publish the build here on GitHub we can connect via Keybase and share it there.

Do you think this will be possible? Of course without selling all your values and putting too much pressure on you :)

[zenhack]

Here you go: https://mirror.zenhack.net/pub/docker-spk Quoting JJ (2018-10-02 11:25:14)

…

I completely understand your point. As I don�t have a ready to use go-environment I cannot simply build the app and as time is rare and costly: can you please provide a build? I know all the risks and I�m willing to take them in this case. If you don�t want to publish the build here on GitHub we can connect via Keybase and share it there. Do you think this will be possible? Of course without selling all your values and putting too much pressure on you :) -- You are receiving this because you were mentioned. Reply to this email directly, [1]view it on GitHub, or [2]mute the thread. Verweise 1. #3 (comment) 2. https://github.com/notifications/unsubscribe-auth/AA18PkZk0ZnxBC327XLnzH08nek_3fl-ks5ug4VagaJpZM4WCZq8

[zenhack]

There are now "official" binaries for MacOS and Linux uploaded to the releases page:

https://github.com/zenhack/docker-spk/releases