[SOLVED] Why both docker and .deb?

[Stooovie]

Hi, I cannot get this to run. I'm not a total noob and do run many docker containers as well as Proxmox LXCs but I don't understand how to run Clapshot. Why is there both Docker AND .deb installers?

The .deb packages have been built without errors but from there the instructions are rather unclear to me.

1. I installed the server .deb with dpkg. What is the client deb for? Does this not work as a pure web app?
   "Server should be put behind a reverse proxy in production, but can be developed and tested without one." - what does this mean?

2. tried running clapshot-server --url-base=http://192.168.0.152:8080 --data-dir=/videos as 192.168.0.152 is the server Clapshot runs on. Conecting from another guest fails (192.168.0.152 refused to connect.). SOMETIMES I get the "Server connecting" but on subsequent refresh it's "192.168.0.152 refused to connect." again.

3. I'm able to run the demo when host and guest are the same machine. When adding the URL base as instructed like this:

docker run --rm -it -p 0.0.0.0:8080:80 -v clapshot-demo:/mnt/clapshot-data/data -e CLAPSHOT_URL_BASE=192.168.0.152:8080/ elonen/clapshot:latest-demo

I get only "Connecting server" in the browser

Thanks for any tips.

[elonen]

(Several different questions here, I'll try to clarify them here a bit more than you probably need for others that may find this post.)

A. Purpose of the packages:

• Server .deb installs a local HTTP/WS server can be configured at /etc/clapshot-server.conf. It listens to HTTP and websockets connections, and the config default to 127.0.0.1 only in port 8095. The package also provides lib/systemd/system/clapshot-server.service to make it start on boot.
• Client .deb is just a convenience that installs static HTML, JS etc in /usr/share/clapshot-client/www/ and provides example configs for Nginx. You can check its contents with dpkg-deb -c clapshot-client_*.deb.
• Keeping stuff in .deb (both client and server) mainly makes upgrades simpler and cleaner in production.

B. Docker:

• Build process uses Docker to avoid contaminating your local system, as well as standardising the environment. Pretty standard.
• The demo Docker image is just that, a demo. Internally, it installs the .debs with some insecure defaults, along with Nginx, ffmpeg and so on, so you can test the system on a laptop without further configurations. I wouldn't recommend using it for hosting in production as it doesn't support HTTP or any kind of advanced authentication, logs stay inside the container etc.

C. Reverse proxy suggestion:

• Since the server .deb only listen on 127.0.0.1 by default, you cannot connect to it from remote servers directly. In production, use Nginx as a reverse proxy to handle authentication and add HTTPS/WSS encryption before it reaches port 8095 plaintext.
• In the demo Docker, notice thathttp://127.0.0.1:8080 says 8080, while server's port = 8095. This is because the container is running an Nginx on 8080, that relays to the Clapshot server on 8095, while adding toy authentication on top of it.

The reason you are getting connection refused on 2. is likely that you specified url-base as http://192.168.0.152:8080 instead of :8095. Since you are running the server manually without Nginx in front of it (like the demo Docker does), when you try to connect to :8080, nobody's answering.

When you see "connecting server", the best way to debug your setup is to open javascript console and look where exactly it's trying to connect.

(related to: #30)

[elonen]

Here's an excerpt from a production clapshot-server.conf:

[general]
url-base = https://clapshot.example.com
data-dir = /mnt/clapshot-data/data

port = 8095
host = 127.0.0.1
bitrate = 16

Notice that url-base doesn't specify a port at all. That's because there's an Nginx relaying both client files and server API calls through standard HTTPS port 443 like so:

       location / {
                root /var/www/clapshot-client;
                # ^ this is actually a symlink: /var/www/clapshot-client -> /usr/share/clapshot-client/www
                
                index index.html;
                try_files $uri $uri/ =404;

(...some authn/authz stuff here...)

                location ~* \.(html|json|conf|js|css)$ {
                        expires 4h;
                        add_header Cache-Control "public, no-transform";
                }

                location /videos {
                        alias /mnt/clapshot-data/data/videos;
                }

                location /api {
                        proxy_pass http://127.0.0.1:8095/api;

                        # Also pass along websocket
                        proxy_http_version 1.1;
                        proxy_set_header Upgrade $http_upgrade;
                        proxy_set_header Connection "Upgrade";
                        proxy_set_header Host $host;

                        # Pass authenticated username to backend
                        proxy_set_header X-Remote-User-Id    $http_x_authn_user;
                        proxy_set_header X-Remote-User-Name  $display_name;

                        proxy_set_header X-Real-IP $remote_addr;
                        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

                        # Allow large uploads
                        # (TODO: maybe limit to /api/upload ?)
                        proxy_request_buffering        off;
                        client_body_buffer_size        256K;
                        client_max_body_size           50G;
                }

[Stooovie]

Perfect. Thank you so much. This gives me enough to soldier on.

[elonen]

One more point:

Also check /etc/clapshot_client.conf.
Since the client is just HTML + JS served from Nginx directly, it doesn't know where the server is unless you tell it.

Here's an a production conf for client, for completeness:

{
    "ws_url": "wss://clapshot.example.com/api/ws",
    "upload_url": "https://clapshot.example.com/api/upload",
    "user_menu_extra_items": [
        { "label": "My Videos", "type": "url", "data": "/" },
        { "label": "Logout", "type": "url", "data": "https://vouch.example.com/logout" }
    ],
  "user_menu_show_basic_auth_logout": false
}

Notes:

• This is symlinked for Nginx as: /var/www/clapshot-client/clapshot_client.conf.json -> /etc/clapshot_client.conf
• The last two lines in this example are logout menu customizations for this particular deployment that uses Okta for authentication.

[Stooovie]

That's too much for me... I tried having both removing 8080 from the base URL in .conf as well as having 8095 as part of the base url, to no avail.

I cannot get to /htadmin at all (that is always refused), even from local IP

I probably won't be able to make this work at this time.

Thanks for your time.

[elonen]

Ok. Could you tell a little bit about your prospective use case however?

I just added a note to front page about this not being a drop-in replacement for the various cloud services, but if I get a more clear idea about what people are looking for, maybe I could make some kind of a lighter, immediately useful and less authentication/security oriented Docker image apart from the demo. 🤔

[Stooovie]

It looks good enough for general use with remote reviewing and commenting. We use Trello and manually typed timecodes which sort of works but obviously time-based comments are better. I don't have such output that would warrant monthly subscription of the paid services. Clapshot looks great and I think a lightweight docker compose could do (obviously I am not a developer but I run a lot of self-hosted stuff - Immich seems kinda similar in scope). I don't need a lot of security, simple password auth done with Nginx Proxy Server would be enough.

(to add to previous convo, I get the intricacies of reverse proxies and stuff, but why was I never able to access 192.168.0.XXX/htadmin?)

[elonen]

👍 How and where would you ideally have installed and configured it?

More specifically:

1. On some docker-enabled NAS or a Linux server with lots of RAM and a good CPU?
2. For configuring a Docker image, would you prefer some terminal installer that asks for endpoints, bitrates etc, just a combined config file in the local folder exposed to Docker (where videos go), or something else?
3. Do you need HTTPS? (entailing certificate management.)

About the /admin page, without seeing your exact environment, it’s quite hard for me to debug, but I’ll test Docker image for any regressions.

Not sure if htadmin would be the best solution for a ready-to-use Docker distribution, though. I literally spent 15 minutes to find something that could work as a demo on how to integrate custom authentication systems.
Do you have any preferences?

[Stooovie]

1. I was trying this on my homelab Proxmox LXC (container) with just 2 cores and 2GB RAM and the performance was good enough. I don't really need transcoding, I can render to low-bitrate mp4s for approvals. Trying it on my M1 MBP was obviously much faster but either was fine
2. I think the latter (combined config file) would be enough at launch :) more options can be always added later if needed at all.
3. I don't, no. If I do, Nginx Proxy Manager that I use for all my stuff can take care of that.

[elonen]

Just tried the htadmin version (with docker run --rm -it -p 0.0.0.0:8080:80 -v clapshot-demo:/mnt/clapshot-data/data elonen/clapshot:latest-demo-htadmin).

For some reason, when I pasted http://127.0.0.1:8080/htadmin to Chrome, it automatically changed it tohttp://127.0.0.1/htadmin which obviously doesn't work. Not sure what's going on there.

Anyway, when I manually edited the address line and added the :8080 back, htadmin page opened and I could login with htadmin / admin.

[Stooovie]

IIRC I did that too (I know this weird behavior of various browsers) - I have Arc, Safari and Edge open and each one behaves differently :) I'll try Firefox too

EDIT: DISREGARD, this is really the culprit with htadmin. Browsers just deciding to remove the :PORT

[elonen]

On my Macbook, Firefox even changed http:// to https:// in addition to the port!
Either something wrong with the demo container's Nginx or htadmin settings, or this is some new browser thing, pushing automatically for more secure connections. 👀

[Stooovie]

What's weird is that the Mainsail 3D printer interface for example is served over http but the browser doesn't try to redirect it to https. If I configure CS in the same way, I get this in the console and I see "https" in the URL...

Mixed Content: The page at '<URL>' was loaded over HTTPS, but requested an insecure resource '<URL>'. This request has been blocked; the content must be served over HTTPS.

When I try setting it as https in NPM, I only get 502 Bad Gateway.

I do "Empty cache and hard reset" on every attempt. The only way to access CS through the internet is to disable SSL at all (every other thing I have via NPM has a LE cert except this) and set it to http.

[elonen]

That probably means it’s trying to connect ws:// instead of wss:// from the client. Edit both the server and client configs to make sure there are https+wss instead of http+ws in the URLs. Otherwise client will think you want it NOT to use SSL when connecting to server.

The reason why you have to set (the same) url-base for the server config, too, is that the server sometimes creates URLs (eg. pointing to the videos) that it send to the client verbatim.

In NPM, it should probably say http, because your proxy is connecting to the Clapshot VM unencrypted, and then encrypting it transparently for the web.

[Stooovie]

...I did as you said (wss instead of ws and https instead of Http in both server and client config) and voila! It finally works as expected. Thank you so much! Do you have a tip jar or something?

[elonen]

Great! I don’t, but if you feel you understand how the thing works now - or at least how to configure it - a blog post (or perhaps a howto-video if you’re a tuber) could help those that want a similar setup. I’m a too familiar with the inner workings, so another perspective / way of explaining the process might benefit some people.

[Stooovie]

I'm sharing it on Mastodon right now, I don't have a blog or YT per se, but I can write something on a few subreddits for example. It's a good project!

[azukaar]

Picking on this comment:

Since the server .deb only listen on 127.0.0.1 by default, you cannot connect to it from remote servers directly. In production, use Nginx as a reverse proxy to handle authentication and add HTTPS/WSS encryption before it reaches port 8095 plaintext.

The issue is that in Docker, the local connection would not be from 127.0.0.1, because typically to run a proxy you keep the port closed. Instead, it would either be coming to

• container_name:port (in bridge setup)
• or docker_ip:port (in host mode setup)

I understand there is a CLAPSHOT_URL_BASE to change that behaviour? But then the issue is that the true URL will be the domain, not the container name/ip the reverse-proxy uses to connect.

Any help to understand how to reach the container from a reverse-proxy container would be appreciated :)

Also for the part about the Docker not being production ready, I think there's a world where users are running the docker container for ease of management, but still run a full reverse proxy on top of it, to make it production ready, so it's an unusual stance to have to call it "for test only". WHat are your thoughts?

[elonen]

@azukaar
URL base (in both client and server configs) is the address that gets passed to the browser, used for CORS etc, so this should point to your reverse proxy - wherever it's actually running - because that's what you want the Javascript and/or web browser to connect to.

If you need the server to bind to something other than localhost (e.g. a container's non-local IP address), the --host command line parameter (or with the .deb, equivalently, the host config line in /etc/clapshot-server.conf) is supposed to change it.
HOWEVER, I have apparently botched this in the codebase and --host option seems to be currently ignored!
This is a bug I'll need to fix (added as issue #55).

In principle though, a multi-container (Docker compose, Kubernetes) production deployment as you described is definitely reasonable. It's just not something I'm doing myself (using LXC instead), so the Docker image config I made is for easy building / testing / demoing.

[azukaar]

understood thanks for the detailed response!

[elonen]

Issue #55 fixed in master, --host will work as intended in the next release (or now if you want to rebuild yourself).

(Release 0.6.0 is almost ready, but I think I'll still add some unit tests for the "basic_folders" Organizer plugin, which is written in Python and more prone to undetected bugs than the Rust-based Server.)