Suggestion: alternate ports in install script

[hiro24]

Just tried the new install script, pretty nice!

I imagine I ran into a problem others might. Default ports are in use. Wasn't a big deal to work around, I just reconfigured host port in portainer. But it would be helpful to be able to specify this in the install script.

Ultimately I had to run cadence on something other than 8080 and nginx on something other than 80.

Please consider an option to configure host ports for services in install.sh

Thanks for the great installer!

[hiro24]

As an aside, is the web UI supposed to update the album art, title and song as it changes? Because for me it's never done that. Not in this version or the previous. I wasn't sure if this was a misconfiguration on my part or a glitch. (Maybe this should be its own bug report, but didn't want to flood you if it wasn't) :)

[za419]

As an aside, is the web UI supposed to update the album art, title and song as it changes? Because for me it's never done that. Not in this version or the previous. I wasn't sure if this was a misconfiguration on my part or a glitch. (Maybe this should be its own bug report, but didn't want to flood you if it wasn't) :)

It is definitely supposed to do this - It works fine on https://cadenceradio.com/, which (typically) runs master.

Sounds like there's a break in SSE on your deployments - Perhaps the firewall is blocking the websockets?

@kenellorando - What do you think of reimplementing this around Socket.io or something? I remember you mentioned seeing some finnicky behavior out of these before, perhaps moving to a protocol which provides some more guarantees and a fallback to different transports (all the way to polling and JSONP, which is ironically enough our own old implementation - Though it would poll for the change event, not for the metadata/album art) would provide some more robustness here?

I know it'll work fine in the browser and in CadenceBot, and I'm fairly certain libraries exist for Golang...

Just a thought - If you had problems before, @hiro24 is having problems, and it sounds like I'm probably going to end up having problems when CadenceBot is trying to hold a connection open for days on end to get these events, perhaps we need to consider a robustness improvement on the server/transport side of things - No point in using stuff that makes life harder for all clients and arguably the server at the same time.

[kenellorando]

Glad to hear you are still using Cadence! I didn't take into account that Cadence might be used on a host with these ports already in use so that is a valid issue. Another workaround would be to edit the config/cadence.env and the config/nginx.conf files (after running install.sh) to change the service addresses as necessary.

• cadence/config/cadence.env.example

  Lines 15 to 16 in 34db796

  	# Service Addresses
  	CSERVER_PORT=:8080

• cadence/config/nginx.conf.example

  Line 18 in 34db796

  listen 80;

Of course, that's pretty tedious so I think modifying the installation script would be a good idea. I'll just have to consider to what extent it should be customizable. As the stack is running six services on six different ports, I am not sure if I want to have to prompt the user six additional times. I also might alternatively create official documentation describing how to customize these ports. I'll think about this.

[kenellorando]

@hiro24 As to the second issue, my first instinct is it might have to do with the nginx component because you mentioned some ports were changed.

There's a specific spot in nginx.conf that allows these automatic updates to get through nginx, and my first suggestion is to confirm that your address set is correct here on your live config:

cadence/config/nginx.conf.example

Line 30 in 34db796

proxy_pass http://cadence:8080/api/radiodata/sse;

If that doesn't work, I think we should collect more information.

1. Does https://cadenceradio.com/ appear to update song status/art as songs change? It works for @za419 and myself, so it working or not could help indicate if the issue is isolated to your browser.
2. On your radio UI, are there any messages logged to your browser's developer console (F12)?

[za419]

@hiro24 Also (sorry for all the questions), what browser are you using?

SSE is fairly new, it could be your browser doesn't support it (another argument in favor of a stack with support for fallbacks..)

[hiro24]

So worries, @za419, sry been afk most of the day.

Line 30 on my nginx.conf matches what you pasted exactly. That is correct, right?

As far as https://cadenceradio.com/ I listened to it (JPOP, nice :) ) and I saw the song change. It's not on mine though. I thought maybe it was a browser so I tested it with Edge and Firefox (my normal browser).

A word on my setup. I'm using NPM to translate IPs to FQDNs. I have webhook support turned on, and icecast.xml is pointing to the FQDN I have set up for icecast2. This may be a possible cause for the problem. Maybe.

[hiro24]

Also, I noticed on my stream it says current listeners: -
Even when I had two browsers up.

Cadence logs shows listeners and updates. From the logs:

2023/03/29 01:23:27 [INFO ][ICECASTMONITOR] Listener count: <2>
2023/03/29 01:23:28 [INFO ][ICECASTMONITOR] Listener count: <1>
2023/03/29 01:26:38 [INFO ][ICECASTMONITOR] Now Playing: Fight for Your Right by Beastie Boys
2023/03/29 01:30:06 [INFO ][ICECASTMONITOR] Now Playing: Get Outta My Dreams, Get Into My Car by Billy Ocean
2023/03/29 01:34:41 [INFO ][ICECASTMONITOR] Now Playing: Back In Black by AC/DC

[za419]

Definitely sounds to me like it's SSE that's broken - That drives song metadata updates (but not the first load!) and listener casts.

Assuming you're running a recent version (5.1+), is your song history also not updating? That's also driven by SSE.

If you're running Cadence on something besides port 8080, my guess is that you'll need to change that line in your nginx.conf to have the right port number:

proxy_pass http://cadence:<PORT>/api/radiodata/sse; 

[hiro24]

Definitely sounds to me like it's SSE that's broken - That drives song metadata updates (but not the first load!) and listener casts.

Assuming you're running a recent version (5.1+), is your song history also not updating? That's also driven by SSE.

If you're running Cadence on something besides port 8080, my guess is that you'll need to change that line in your nginx.conf to have the right port number:

proxy_pass http://cadence:<PORT>/api/radiodata/sse; 

So you're correct. History does not update unless I refresh (at which point all the data is current).

As far as versions, I'm on Cadence 5.4.2. I tried changing the port as you suggested to my port, but that didn't affect anything. I also attempted changing the next proxy_pass line 3 lines down to match my port also. Neither seemed to do anything. I stopped and started all cadence related services after making the changes.

[hiro24]

As far as SSE, I can send you some logs but I'm not sure which service to check or what I'm looking for. Is there a debug mode I might be able to turn on that might reveal something for you guys?

[hiro24]

Hey-o!

2023/03/29 02:08:28 [emerg] 1#1: host not found in upstream "cadence" in /etc/nginx/nginx.conf:30
nginx: [emerg] host not found in upstream "cadence" in /etc/nginx/nginx.conf:30

[hiro24]

So I thought I was on to something there. I changed the proxy_pass lines to the IP of cadence, rather than the name. (I tried both internal IP and port and host IP and host port) This did seem to remove this error message, and I'm now seeing some search database requests in the cadence logs I wasn't seeing before.

Unfortunately the good news stops there. The UI is still not auto-updating.

[kenellorando]

OK, catching up here again. Just to pause to collect some evidence we have so far:

1. the issue is not present to anyone on other deployments (cadenceradio.com)
2. the issue is present on multiple of @hiro24's browsers, including Edge and Firefox
3. edit: the issue is isolated to SSE and apparently no other API functions

It sounds like the issue is localized to some configuration in your deployment. Now to just narrow down the issue a little more. Do you see this problem regardless of if you access cadence from your custom NPM FQDN, and if you access cadence from localhost:<cadence_port>, or is the behavior the same on both?

[hiro24]

Unfortunately I can't test this from localhost. It's running on a headless Linux server.

[kenellorando]

Hm, I'm almost out of ideas. I don't think we can isolate the issue with complete certainty with the presence of the other NPM component. Based on our evidence though, that component seems to be the most likely culprit. I'll think out loud here.

Radio data transfer normally looks like this:

Client      -> Server
[ Browser ] -> [ Nginx -> Cadence API ]

And if I understand your individual setup correctly, it looks like this:

Client      -> Server
[ Browser ] -> [ NPM Component -> Nginx -> Cadence API ]

Originally, when I created the nginx component of the Cadence stack, it needed a special configuration to be able to maintain an SSE connection with the browser. Without that configuration, SSE and all auto-update functions would cease to effectively work. Given what we know, the most likely root cause culprit is this NPM component needs to be configured in a similar way if possible to be able to pass an SSE connection.

My recommendations (it sounds tedious but the only ways I can think to narrow down any further): try to replicate your Cadence stack on a computer with a GUI so you can browse to it twice: once configured without the NPM component and once with it. While I'm not totally sure what the NPM component is doing, I will also remind that the Nginx component in Cadence is capable of handling any DNS you might provide it (provided in step 5 in the installer).

[hiro24]

So I'm trying to figure out how to step around NPM in this case. I had the IP tied to radio2. and icecast2.xml pointed to that as the hostname. This results in no playback at all.

I've tried changing this to just be the IP address instead to see if it gave better results. But I can't get that to play. I've tried setting it to the IP address of icecast2, liquidsoap and even the host's IP.

One thing I did notice was that it shows the connected link on the web interface lists https. If I go to the host IP at http on port 8000 I can get where I think it wants me to be going. But I can't for https. For whatever reason it's not listening as https. NPM fixed this for me.

Did I maybe miss something somewhere?

[kenellorando]

There seem to be multiple compounding issues which make this difficult. I'll approach this in a line by line style now that may be helpful to debug things.

So I'm trying to figure out how to step around NPM in this case. I had the IP tied to radio2. and icecast2.xml pointed to that as the hostname. This results in no playback at all.

This is because all components normally only talk to each by container name, provided by Docker's DNS. When you update the icecast.xml hostname to match, this is only telling Icecast that "people will connect in a browser at this exact hostname to be able to get Icecast". For example, if you set this to radio2, Icecast would expose this value for Cadence to read as-is and set as the address in the UI to connect to. Thus, your UI page is trying to connect to http://radio2 to fetch a stream.

Another example, I provide stream.cadenceradio.com to my Icecast.xml, and it is thus directly accessible at https://stream.cadenceradio.com. If you're on a personal computer, the address you give to Icecast.xml hostname should be the same address you intend to be able to directly connect to Icecast.

I've tried changing this to just be the IP address instead to see if it gave better results. But I can't get that to play. I've tried setting it to the IP address of icecast2, liquidsoap and even the host's IP.

Similar to above, with an IP address, it needs to be whatever address you expect to be able to reach your stream from your computer's browser. If it's an IP, you'll need to provide the port of Icecast too (default 8000).

One thing I did notice was that it shows the connected link on the web interface lists https. If I go to the host IP at http on port 8000 I can get where I think it wants me to be going. But I can't for https. For whatever reason it's not listening as https. NPM fixed this for me.

It sounds like your npm component is some kind of automagic-HTTPS reverse proxy. The vanilla Cadence stack right now doesn't support HTTPS out of the box, and my https://cadenceradio.com deployment is only HTTPS because of Cloudflare. I again suggest we purge this component from the setup completely because we can't isolate issues effectively. We should also forgo HTTPS for now until we get it working on HTTP. Since we're in a broken state right now anyway, I think it's best to start over fresh with the installation script, and provide it only default options. From there, let's see if you can just browse to your server's IP address.

[kenellorando]

As an aside, I'll also start to think about a future version of Cadence nginx with optional support for HTTPS out of the box, to automate these types of setups. (to have built-in support for the purpose of the npm component)