vagrant-spk dev not working apps in Sandstorm apps

[ssivachidambaram]

vagrant-spk vm up is working properly and can able to access site using IP.

Now we're following this tutorial to package our app https://docs.sandstorm.io/en/latest/vagrant-spk/packaging-tutorial/. But while running vagrant-spk dev, its not creating anything in Sandstorm apps page.

We've committed our files here https://github.com/RestyaPlatform/board/tree/dev/.sandstorm.

[ocdtrekkie]

When you run vagrant-spk dev, are you seeing "App is now available from Sandstorm server. Ctrl+C to disconnect." in the CLI?

If so, the most likely issue is that somehow you are not accessing the same Sandstorm server as vagrant-spk is creating. This could happen if you are running Sandstorm locally on the server and then also using vagrant-spk, which spins up it's own Sandstorm server inside the VM, or if you have multiple vagrant-spk vms running for some reason.

[ocdtrekkie]

I will see if I can find some time to clone your repo and bring it up on my Sandstorm dev box.

[ssivachidambaram]

@ocdtrekkie

When you run vagrant-spk dev, are you seeing "App is now available from Sandstorm server. Ctrl+C to disconnect." in the CLI?

Yes. I got this message.

If so, the most likely issue is that somehow you are not accessing the same Sandstorm server as vagrant-spk is creating. This could happen if you are running Sandstorm locally on the server and then also using vagrant-spk, which spins up it's own Sandstorm server inside the VM, or if you have multiple vagrant-spk vms running for some reason.

Yes. I am running in local server but single VM is there.

I will see if I can find some time to clone your repo and bring it up on my Sandstorm dev box.

Thank you very much. Once its worked i will create spk and submit to display in Sandstorm for our users.

[ocdtrekkie]

If you're running Sandstorm locally on port 6080, you'd need to stop your Sandstorm server (sudo service sandstorm stop) before you bring your vagrant-spk vm up so that the vagrant-spk VM can use port 6080.

[ssivachidambaram]

@ocdtrekkie

If you're running Sandstorm locally on port 6080, you'd need to stop your Sandstorm server (sudo service sandstorm stop) before you bring your vagrant-spk vm up so that the vagrant-spk VM can use port 6080.

1. Stopped sandstorm service.
2. Run vagrant-spk vm up
   ==> default: Machine 'default' has a post vagrant up message. This is a message
   ==> default: from the creator of the Vagrantfile, and not from Vagrant itself:
   ==> default:
   ==> default: Vanilla Debian box. See https://app.vagrantup.com/debian for help and bug reports
3. Run vagrant-spk vm ssh
   Checked IP address by using command ip a and then can able to access URL successfully.
4. Run vagrant-spk dev
   Calling 'vagrant' 'ssh' '-c' '/opt/app/.sandstorm/build.sh && cd /opt/app/.sandstorm && spk dev --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef' in /usr/share/nginx/html/RestyaPlatform/board/.sandstorm
   App is now available from Sandstorm server. Ctrl+C to disconnect.
5. Now i have stopped Sandstorm service, now where i have to check the app? Also when press Ctrl+C its not storing all the files in this file sandstorm-files.list. Storing only this file sandstorm-manifest.

[ocdtrekkie]

It should be at local.sandstorm.io:6080 (or localhost:6080) when you have the Sandstorm service stopped. Basically your Sandstorm service and vagrant-spk are both trying to run Sandstorm on port 6080, and so if you have the Sandstorm service running, vagrant-spk can't use it, and you see the non-dev Sandstorm instance.

[ssivachidambaram]

@ocdtrekkie

It should be at local.sandstorm.io:6080 (or localhost:6080) when you have the Sandstorm service stopped. Basically your Sandstorm service and vagrant-spk are both trying to run Sandstorm on port 6080, and so if you have the Sandstorm service running, vagrant-spk can't use it, and you see the non-dev Sandstorm instance.

Thank you very much for the information.

Now i can able to access http://local.sandstorm.io:6080/ and saw the apps but when i create grain its showing error,

http://ui-b06ce6adde47ed7c05b8bfe41c39e7ee.local.sandstorm.io:6080/_sandstorm-init?sessionid=a7a7d7cfdc6f6d9e170045b0427bf2b41599087103df40bcf64617d90ce4bee6&path=/

ERROR: The server threw an exception. Details:

sandstorm/gateway.c++:299: failed: expected parsed.query[1].name == "path"
stack: 5ebc89 7171ed 716e52 50f1f1 71cc20

[ocdtrekkie]

Well, we definitely made progress, because you are now able to troubleshoot your Sandstorm package!

@JamborJan Any chance you can help troubleshoot the package at all? I am probably leaving my depth around there.

[ssivachidambaram]

@ocdtrekkie

Well, we definitely made progress, because you are now able to troubleshoot your Sandstorm package!

Yes. I am using ubuntu machine and is there anything i have to add for subdomain ui-b06ce6adde47ed7c05b8bfe41c39e7ee.local.sandstorm.io:6080 in local machine?

[ocdtrekkie]

No, that's just a random subdomain Sandstorm is using for that grain. If you make another one it'll be different. Something in your grain isn't starting up correctly.

[ssivachidambaram]

@ocdtrekkie

No, that's just a random subdomain Sandstorm is using for that grain. If you make another one it'll be different. Something in your grain isn't starting up correctly.

Ok. Is there any command to debug that grain? Also while create grain which shell script will execute setup.sh, build.sh, launcher.sh?

[JamborJan]

Hey guys,

Debugging a running app is possible with nsenter:

• run app in spk dev
• create or open a grain
• try spk enter-grain
• select grain id
• wait for the error message + remember pid
• ssh into the running vm
• run nsenter

(Source is this issue comment: sandstorm-io/vagrant-spk#213 (comment))

If you need more detailed help and instructions to debug the issue, please let me know. Good luck!

[ssivachidambaram]

@JamborJan

vagrant-spk enter-grain
Looking for apps in dev mode...
Looking for grains...
No Sandstorm supervisor processes found.
Make sure to open a grain of this app in your web browser.

http://ui-aff367f14059bf4373e663d9cf1e4345.local.sandstorm.io:6080/_sandstorm-init?sessionid=1ee5ff4afeb65643f68fbf34e3048f53ff7af778cc123d13bb75f724f9b3c573&path=/

Redirecting to http://ui-aff367f14059bf4373e663d9cf1e4345.local.sandstorm.io:6080/

But its displaying error message.

[JamborJan]

Do you have a public repository which I could clone and run the single steps on my own? It’s often easier to find the issue this way.

[ssivachidambaram]

@JamborJan Yes. You can find it here https://github.com/RestyaPlatform/board/tree/dev/.sandstorm.

[ssivachidambaram]

@JamborJan Did you get a chance to check the issue we faced? We've completed 95% but stuck here due to grain not worked. Can you please help us to proceed further?

[JamborJan]

hey @ssivachidambaram I'm sorry, I'm packed with work right now and was not able to look into this yet. I'll do my best to give you feedback until the end of the week, okay?

[ssivachidambaram]

@JamborJan Okay. Thank you.

[JamborJan]

Hey @ssivachidambaram, partially good news: I am able to reproduce the issue which means I can debug it. So I'm working on that tomorrow and hopefully get an answer for you. I'll keep you posted.

[ssivachidambaram]

@JamborJan Thank you very much.

[ssivachidambaram]

@JamborJan Did you get a chance today to look it into the issue?

[JamborJan]

I‘m on it right now. There are some things missing in the spin up process plus you cannot use cron in sandstorm. But let me first try to fix it completely and then I’ll send you a pull request and detailed explanations. Okay?

[ssivachidambaram]

@JamborJan Okay. Thanks for your help.

[JamborJan]

I did make progress today but am not yet finished. The Nginx configuration needed adjustment and I streamlined some parts. I’ll contiune later this evening and if necessary tomorrow and let you know the results.

[ssivachidambaram]

Okay.

[JamborJan]

hey @ssivachidambaram,

please take a look here where I forked your repository and changed some stuff. I still have a problem that the php-fpm sock ist not properly working. Unfortunately I've no clue yet why this happens. Maybe you have an idea? I'll continue to investigate during the upcoming days. maybe you have also an idea.

[ssivachidambaram]

@JamborJan

https://github.com/JamborJan/board/blob/4112420f0f6d5a60cc1e1e28affca0cc65c40cdd/.sandstorm/launcher.sh#L10

In launcher.sh, it was mentioned as php5-fpm.sock. Is this the problem?

[JamborJan]

Unfortunately this is only a check I included to trace down the issue. Thanks for the hint but it’s not the cause. The whole block from while until done is optional and could be removed.

[ocdtrekkie]

@JamborJan Are you noting that that check is for php-fpm5, but the repo uses php-fpm7.2?

[JamborJan]

Hey @ocdtrekkie, after @ssivachidambaram pointed me at this: yes. But as it is only a check and optional, i don’t know yet why fpm is not starting.

[JamborJan]

I removed the check again (it didn't help or destroy anything).

When starting the app in dev mode I see in the app console:

...** SANDSTORM SUPERVISOR: Starting up grain. Sandbox type: userns
[26-Nov-2018 06:57:41] ERROR: unable to bind listening socket for address '/var/run/php/php7.2-fpm.sock': No such file or directory (2)
[26-Nov-2018 06:57:41] ERROR: FPM initialization failed
** HTTP-BRIDGE: App server exited with status code: 70
** SANDSTORM SUPERVISOR: App exited with status code: 1
** SANDSTORM SUPERVISOR: Starting up grain. Sandbox type: userns
[26-Nov-2018 06:57:41] ERROR: unable to bind listening socket for address '/var/run/php/php7.2-fpm.sock': No such file or directory (2)
[26-Nov-2018 06:57:41] ERROR: FPM initialization failed
** HTTP-BRIDGE: App server exited with status code: 70
** SANDSTORM SUPERVISOR: App exited with status code: 1
** SANDSTORM SUPERVISOR: Starting up grain. Sandbox type: userns
[26-Nov-2018 06:57:42] ERROR: unable to bind listening socket for address '/var/run/php/php7.2-fpm.sock': No such file or directory (2)
[26-Nov-2018 06:57:42] ERROR: FPM initialization failed
** HTTP-BRIDGE: App server exited with status code: 70
** SANDSTORM SUPERVISOR: App exited with status code: 1
** SANDSTORM SUPERVISOR: Starting up grain. Sandbox type: userns

When I ssh into the vm and try to start the fpm process with this command:

/usr/sbin/php-fpm7.2 --fpm-config /etc/php/7.2/fpm/php-fpm.conf -c /etc/php/7.2/fpm/php.ini

I get this error message:

[26-Nov-2018 06:59:33] ALERT: [pool www] user has not been defined
[26-Nov-2018 06:59:33] ERROR: failed to post process the configuration
[26-Nov-2018 06:59:33] ERROR: FPM initialization failed

[ssivachidambaram]

@JamborJan

I have also received this error when starting from SSH. When searching it, after removing semi colon ; from this file /etc/php/7.2/fpm/pool.d/www.conf

;user = www-data
;group = www-data

I can able to start php-fpm.

[JamborJan]

I did find that too now, this was a mandatory change which is applied to the configs in Sandstorms use case. See: https://github.com/sandstorm-io/vagrant-spk/blob/master/stacks/lemp/setup.sh#L23-L28

The crazy thing is, that all other configurations and apps I have are working with this change.

The change is not working either:

...** SANDSTORM SUPERVISOR: Starting up grain. Sandbox type: userns
[26-Nov-2018 07:27:56] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[26-Nov-2018 07:27:56] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[26-Nov-2018 07:27:56] ERROR: [pool www] cannot get uid for user 'www-data': No such file or directory (2)
[26-Nov-2018 07:27:56] ERROR: FPM initialization failed

[JamborJan]

@ssivachidambaram I can investigate into this further but before that I want you to consider & think about two things:

1. Sandstorm apps don't have anything like cron. So everything which is scheduled is not working right now. There are plans to include something like this in the platform but this is not done yet. So if you really need that, we have a blocker here.

2. communication to from inside a grain to the outside world and vice versa is very limited. Any apps for your boards will not work out of the box. There will be a lot of development required to achieve that.

What do you think?

[ssivachidambaram]

@JamborJan

Sorry for the delayed response.

Sandstorm apps don't have anything like cron. So everything which is scheduled is not working right now. There are plans to include something like this in the platform but this is not done yet. So if you really need that, we have a blocker here.

Yes. Not a problem.

communication to from inside a grain to the outside world and vice versa is very limited. Any apps for your boards will not work out of the box. There will be a lot of development required to achieve that.

Within a network it will work or only for single user? Also our functionality is like Wekan which is already available in Sandstorm.

So we need to publish our app in Sandstorm.

[ocdtrekkie]

It isn't an issue for multiple users to use a Sandstorm app. It's just that Sandstorm apps can't easily talk to outside servers. Even accessing other Sandstorm apps requires a bit of work.

If your functionality is similar to Wekan you should be fine. If there's third party integrations like allowing Twitter login or something, that probably would not work. There are some features Wekan has that don't work in Sandstorm and are disabled in the Sandstorm version, though I think a lot of those are because the Sandstorm app version of Wekan uses a single-board design.

[ssivachidambaram]

@ocdtrekkie Thanks for the explanation. We don't have any third party integrations. So it's fine for us to publish in Sandstorm app.

[JamborJan]

Hey guys, thanks for the additional input and thoughts. I’ll continue to work on the problems we encountered. If anybody else sees something please add a comment.

[ssivachidambaram]

@JamborJan Sorry to bother you. Did you got any idea about the fpm start issue?

[JamborJan]

Hey @ssivachidambaram I'm working on it, just had a meeting with a colleague discussing some options. One question: is your docker image of the RestyaPlatform board working well? I'm testing a process where we could use this as base instead of doing a lot of separate stuff in vagrant.

Thanks!

[ssivachidambaram]

@JamborJan Yes. It will work. Please use https://hub.docker.com/r/restyaplatform/restyaboard/tags/ v0.6.6.

[JamborJan]

Short update: I’ll not be able to work on this issue this week. I have some ideas and fixes but unfortunately no time this week. Sorry.

[ssivachidambaram]

@JamborJan Ok. Thank you for the update.

[ssivachidambaram]

@JamborJan Sorry for not following up on this. Did you have any chance to look up on this?

[JamborJan]

Unfortunately I was not able to find a solution yet. A lot of projects kept me busy and I was not able to work on these topics. Sorry. Right now I cannot promise when I can start working on this again. I made some progress on the overall process on packaging apps based on docker images but thetas not final yet.

[ssivachidambaram]

@JamborJan Thanks for your reply and help. Can i have that file to continue the process?

[JamborJan]

There is nothing yet for restyabord as I was experimenting with a repo where I'm more familiar with. Basically I'm using docker images, this tool:
https://github.com/zenhack/docker-spk and I try to apply a process like this:
zenhack/docker-spk#9

[ssivachidambaram]

@JamborJan Ok. I will try it and let you know.

[JamborJan]

Hey @ssivachidambaram sorry when I don't get what file you are asking for. There is no file yet. All I have done so far is a base image (here:
https://github.com/sandstormports/Sandstorm-Docker-Base-Image) which could be used for all kinds of apps but it's not a must have.

Then I have started working on the wordpress port to build the described process (see pictures here:
zenhack/docker-spk#9) but I encountered several issues. So I started low and build a Dockerfile which basically does nothing more than the steps I had formerly in the Vagrant process. You can find the Dockerfile here:
https://github.com/sandstormports/wordpress-sandstorm/blob/master/Dockerfile

It is not working yet. But should be a more or less good start.

[ssivachidambaram]

@JamborJan Thanks you very much for the reference file. Will try it.