Static Media Failure #12284
Answered
by
mtinberg
network-plus-code
asked this question in
Q&A
Static Media Failure
#12284
Replies: 3 comments 1 reply
-
|
The /static files are served by nginx which is on port 443 HTTPS but your test is against port 8001, bypassing nginx and talking to gunicorn directly. The flow here is that clients talk to the web server (nginx) and the web server proxies the requests to the netbox app server (gunicorn) with the WSGI protocol, handling static files itself. The webserver does all the web communication, connection handling, SSL/TLS, load balancing, logging, maybe auth, and the app server only accepts requests from the web server
—
Mark Tinberg ***@***.***>
Division of Information Technology-Network Services
University of Wisconsin-Madison
…________________________________
From: Network_plus_code ***@***.***>
Sent: Tuesday, April 18, 2023 10:05 AM
To: netbox-community/netbox ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [netbox-community/netbox] Static Media Failure (Discussion #12284)
[image]<https://user-images.githubusercontent.com/17090049/232579651-be894139-014d-4db8-8f72-45a4ba5aa24a.png>
I have read all of the discussions on this and none have helped to resolve my issue. (At least not that I could determine)
-Static files are all there, I ran the upgrade which runs the collections.py
-The users defined netbox.service and rq.service have access to the directories (It was already not working when most were owned by root, and still not working when I just made everything owned by netbox.)
-This is all mostly just the default install, I changed the local server IPs and proxy pass so I could use a browser from a remote device since there is no gui on the server running this, however even if that is an issue in itself, when it was on the default localhost ip and port I would get the static media failure when curling http://127.0.0.1:8001
* There is no SELinux running so that is not the issue
* Also note this is for a lab environment so I do not care if it is on https or http, I am not sure if this is the issue, but https does not work for the current configuration. The connection times out for that one.
Netbox itself will work if I run it as a development server at either ips.
python3 manage.py runserver 172.25.65.135:8000 --insecure
Performing system checks...
System check identified no issues (0 silenced).
April 18, 2023 - 14:25:54
Django version 4.1.8, using settings 'netbox.settings'
Starting development server at http://172.25.65.135:8000/
Quit the server with CONTROL-C.
[18/Apr/2023 14:26:05] "GET / HTTP/1.1" 200 86517
[18/Apr/2023 14:26:05] "GET /static/setmode.js HTTP/1.1" 200 3506
[18/Apr/2023 14:26:05] "GET /static/netbox-external.css?v=3.4.8 HTTP/1.1" 200 340587
[18/Apr/2023 14:26:06] "GET /static/netbox-light.css?v=3.4.8 HTTP/1.1" 200 232647
[18/Apr/2023 14:26:06] "GET /static/netbox-dark.css?v=3.4.8 HTTP/1.1" 200 375270
[18/Apr/2023 14:26:06] "GET /static/netbox.js?v=3.4.8 HTTP/1.1" 200 381466
[18/Apr/2023 14:26:06] "GET /static/netbox_logo.svg HTTP/1.1" 200 4719
[18/Apr/2023 14:26:06] "GET /static/netbox_icon.svg HTTP/1.1" 200 835
[18/Apr/2023 14:26:06] "GET /static/materialdesignicons-webfont-ER2MFQKM.woff2?v=7.0.96 HTTP/1.1" 200 385360
[18/Apr/2023 14:26:06] "GET /static/netbox-print.css?v=3.4.8 HTTP/1.1" 200 727156
[18/Apr/2023 14:26:06] "GET /static/netbox.ico HTTP/1.1" 200 1174
[image]<https://user-images.githubusercontent.com/17090049/232808944-c8247f90-d7ff-4614-80cf-22eac791c9b6.png>
The issue is something with gunicorn or nginx. I have the services running in systemd with the user netbox.
cat /etc/systemd/system/netbox-rq.service
[Unit]
Description=NetBox Request Queue Worker
Documentation=https://docs.netbox.dev/
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=netbox
Group=netbox
WorkingDirectory=/opt/netbox
ExecStart=/opt/netbox/venv/bin/python3 /opt/netbox/netbox/manage.py rqworker high default low
Restart=on-failure
RestartSec=30
PrivateTmp=true
[Install]
WantedBy=multi-user.target
cat /etc/systemd/system/netbox.service
[Unit]
Description=NetBox WSGI Service
Documentation=https://docs.netbox.dev/
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=netbox
Group=netbox
PIDFile=/var/tmp/netbox.pid
WorkingDirectory=/opt/netbox
ExecStart=/opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
Restart=on-failure
RestartSec=30
PrivateTmp=true
[Install]
WantedBy=multi-user.target
I can see that netbox can access
sudo -u netbox stat /opt/netbox/netbox/static
File: /opt/netbox/netbox/static
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 12719136 Links: 11
Access: (0755/drwxr-xr-x) Uid: ( 115/ netbox) Gid: ( 120/ netbox)
Access: 2023-04-17 17:46:45.422758057 +0000
Modify: 2023-04-17 14:37:28.919283607 +0000
Change: 2023-04-17 17:52:00.978254368 +0000
I can see the processes running as netbox
ps auxwww | grep netbox
netbox 2420 0.0 0.1 29728 22220 ? Ss 17:52 0:00 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbo.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
netbox 2434 0.1 0.8 254996 143180 ? Ssl 17:52 0:05 /opt/netbox/venv/bin/python3 /opt/netbox/netbox/manage.py rqworker high default low
netbox 2453 0.1 0.8 316144 145008 ? Sl 17:52 0:04 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbo.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
netbox 2455 0.1 0.8 241588 144548 ? Sl 17:52 0:04 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbo.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
netbox 2456 0.1 0.8 241648 144496 ? Sl 17:52 0:04 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbo.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
netbox 2457 0.0 0.5 267672 85956 ? Sl 17:52 0:02 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbo.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
netbox 2458 0.1 0.8 389372 145016 ? Sl 17:52 0:04 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbo.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
netbox 2463 0.0 0.8 255252 134240 ? S 17:52 0:02 /opt/netbox/venv/bin/python3 /opt/netbox/netbox/manage.py rqworker high default low
netbox 2482 0.0 0.0 51764 5208 ? S 17:52 0:00 nginx: worker process
netbox 2483 0.0 0.0 51764 5208 ? S 17:52 0:00 nginx: worker process
netbox 2484 0.0 0.0 51764 5208 ? S 17:52 0:00 nginx: worker process
netbox 2485 0.0 0.0 51764 5208 ? S 17:52 0:00 nginx: worker process
netbox 2486 0.0 0.0 51764 5208 ? S 17:52 0:00 nginx: worker process
netbox 2487 0.0 0.0 51764 5208 ? S 17:52 0:00 nginx: worker process
netbox 2488 0.0 0.0 51764 5208 ? S 17:52 0:00 nginx: worker process
netbox 2489 0.0 0.0 51764 5208 ? S 17:52 0:00 nginx: worker process
postgres 2532 0.0 0.1 217516 19036 ? Ss 17:53 0:00 postgres: 12/main: netbox netbox 127.0.0.1(54158) idle
postgres 2543 0.0 0.1 216724 16356 ? Ss 17:53 0:00 postgres: 12/main: netbox netbox 127.0.0.1(54168) idle
postgres 2544 0.0 0.0 216724 16348 ? Ss 17:53 0:00 postgres: 12/main: netbox netbox 127.0.0.1(54176) idle
postgres 2545 0.0 0.0 216724 16344 ? Ss 17:53 0:00 postgres: 12/main: netbox netbox 127.0.0.1(54178) idle
postgres 4069 0.0 0.0 216724 16352 ? Ss 18:34 0:00 postgres: 12/main: netbox netbox 127.0.0.1(37634) idle
I tried it where just static and media were netbox and everything else was root. I then just made everything netbox. I can change it back if this caused more problems but the permission is likely not the issue.
ls -l /opt/netbox/netbox
total 72
drwxr-xr-x 10 netbox netbox 4096 Apr 17 14:37 circuits
drwxr-xr-x 12 netbox netbox 4096 Apr 17 14:37 dcim
drwxr-xr-x 13 netbox netbox 4096 Apr 17 14:37 extras
-rwxr-xr-x 1 netbox netbox 278 Apr 17 14:26 generate_secret_key.py
drwxr-xr-x 11 netbox netbox 4096 Apr 17 14:37 ipam
-rwxr-xr-x 1 netbox netbox 249 Apr 17 14:26 manage.py
drwxr-xr-x 4 netbox netbox 4096 Apr 17 14:26 media
drwxr-xr-x 13 netbox netbox 4096 Apr 17 14:37 netbox
drwxr-xr-x 9 netbox netbox 4096 Apr 17 14:26 project-static
drwxr-xr-x 2 netbox netbox 4096 Apr 17 14:26 reports
drwxr-xr-x 2 netbox netbox 4096 Apr 17 14:26 scripts
drwxr-xr-x 11 netbox netbox 4096 Apr 17 14:37 static
drwxr-xr-x 17 netbox netbox 4096 Apr 17 14:26 templates
drwxr-xr-x 10 netbox netbox 4096 Apr 17 14:37 tenancy
drwxr-xr-x 8 netbox netbox 4096 Apr 17 14:37 users
drwxr-xr-x 10 netbox netbox 4096 Apr 17 14:37 utilities
drwxr-xr-x 10 netbox netbox 4096 Apr 17 14:37 virtualization
drwxr-xr-x 9 netbox netbox 4096 Apr 17 14:37 wireless
ls -l /opt/netbox/netbox/static
total 11172
drwxr-xr-x 6 netbox netbox 4096 Apr 17 14:37 admin
-rw-r--r-- 1 netbox netbox 1159 Apr 17 14:37 cable_trace.css
-rw-r--r-- 1 netbox netbox 107780 Apr 17 14:37 config.js
-rw-r--r-- 1 netbox netbox 105762 Apr 17 14:37 config.js.map
drwxr-xr-x 4 netbox netbox 4096 Apr 17 14:37 debug_toolbar
drwxr-xr-x 3 netbox netbox 4096 Apr 17 17:46 django_tables2
drwxr-xr-x 19 netbox netbox 4096 Apr 17 17:46 docs
drwxr-xr-x 5 netbox netbox 4096 Apr 17 17:46 drf-yasg
drwxr-xr-x 2 netbox netbox 4096 Apr 17 17:46 graphene_django
-rw-r--r-- 1 netbox netbox 27375 Apr 17 14:37 graphiql.css
drwxr-xr-x 3 netbox netbox 4096 Apr 17 14:37 graphiql_debug_toolbar
-rw-r--r-- 1 netbox netbox 978737 Apr 17 14:37 graphiql.js
-rw-r--r-- 1 netbox netbox 929453 Apr 17 14:37 graphiql.js.map
-rw-r--r-- 1 netbox netbox 1235 Apr 17 14:37 graphql.ico
-rw-r--r-- 1 netbox netbox 108435 Apr 17 14:37 lldp.js
-rw-r--r-- 1 netbox netbox 106474 Apr 17 14:37 lldp.js.map
-rw-r--r-- 1 netbox netbox 465188 Apr 17 14:37 materialdesignicons-webfont-2ZY4X5WS.woff
-rw-r--r-- 1 netbox netbox 561776 Apr 17 14:37 materialdesignicons-webfont-DWVXV5L5.woff
-rw-r--r-- 1 netbox netbox 1026176 Apr 17 14:37 materialdesignicons-webfont-EH3JD4GG.ttf
-rw-r--r-- 1 netbox netbox 385360 Apr 17 14:37 materialdesignicons-webfont-ER2MFQKM.woff2
-rw-r--r-- 1 netbox netbox 325244 Apr 17 14:37 materialdesignicons-webfont-KSYPMDN6.woff2
-rw-r--r-- 1 netbox netbox 1026396 Apr 17 14:37 materialdesignicons-webfont-LWDVGUDX.eot
-rw-r--r-- 1 netbox netbox 1243720 Apr 17 14:37 materialdesignicons-webfont-UHEFFMSX.eot
-rw-r--r-- 1 netbox netbox 1243500 Apr 17 14:37 materialdesignicons-webfont-WM6M6ZHQ.ttf
drwxr-xr-x 2 netbox netbox 4096 Apr 17 17:46 mptt
-rw-r--r-- 1 netbox netbox 375270 Apr 17 14:37 netbox-dark.css
-rw-r--r-- 1 netbox netbox 340587 Apr 17 14:37 netbox-external.css
-rw-r--r-- 1 netbox netbox 1174 Apr 17 14:37 netbox.ico
-rw-r--r-- 1 netbox netbox 835 Apr 17 14:37 netbox_icon.svg
-rw-r--r-- 1 netbox netbox 381466 Apr 17 14:37 netbox.js
-rw-r--r-- 1 netbox netbox 354201 Apr 17 14:37 netbox.js.map
-rw-r--r-- 1 netbox netbox 232647 Apr 17 14:37 netbox-light.css
-rw-r--r-- 1 netbox netbox 2257 Apr 17 14:37 netbox_logo.png
-rw-r--r-- 1 netbox netbox 4719 Apr 17 14:37 netbox_logo.svg
-rw-r--r-- 1 netbox netbox 727156 Apr 17 14:37 netbox-print.css
-rw-r--r-- 1 netbox netbox 4054 Apr 17 14:37 netbox_touch-icon-180.png
-rw-r--r-- 1 netbox netbox 1486 Apr 17 14:37 rack_elevation.css
-rw-r--r-- 1 netbox netbox 878 Apr 17 14:37 rest-api.ico
drwxr-xr-x 7 netbox netbox 4096 Apr 17 14:37 rest_framework
-rw-r--r-- 1 netbox netbox 3506 Apr 17 14:37 setmode.js
-rw-r--r-- 1 netbox netbox 128487 Apr 17 14:37 status.js
-rw-r--r-- 1 netbox netbox 128481 Apr 17 14:37 status.js.map
-rw-r--r-- 1 netbox netbox 109 Apr 17 14:37 tint_20.png
Gunicorn Config
cat /opt/netbox/gunicorn.py
# The IP address (typically localhost) and port that the NetBox WSGI process should listen on
bind = '172.25.65.135:8001'
# Number of gunicorn workers to spawn. This should typically be 2n+1, where
# n is the number of CPU cores present.
workers = 5
# Number of threads per worker process
threads = 3
# Timeout (in seconds) for a request to complete
timeout = 120
# The maximum number of requests a worker can handle before being respawned
max_requests = 5000
max_requests_jitter = 500
***@***.*** ~
nginx config
cat /etc/nginx/sites-enabled/netbox
server {
listen [::]:443 ssl ipv6only=off;
# CHANGE THIS TO YOUR SERVER'S NAME
server_name 172.25.65.135:8001;
ssl_certificate /etc/ssl/certs/netbox.crt;
ssl_certificate_key /etc/ssl/private/netbox.key;
client_max_body_size 25m;
location /static/ {
alias /opt/netbox/netbox/static/;
}
location / {
proxy_pass http://172.25.65.135:8001;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
server {
# Redirect HTTP traffic to HTTPS
listen [::]:80 ipv6only=off;
server_name _;
return 301 https://$host$request_uri;
}
Netbox status
sudo systemctl status netbox
● netbox.service - NetBox WSGI Service
Loaded: loaded (/etc/systemd/system/netbox.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-04-18 14:32:46 UTC; 1min 5s ago
Docs: https://docs.netbox.dev/
Main PID: 42393 (gunicorn)
Tasks: 6 (limit: 19056)
Memory: 368.5M
CGroup: /system.slice/netbox.service
├─42393 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /op>
├─42415 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /op>
├─42416 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /op>
├─42417 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /op>
├─42418 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /op>
└─42419 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /op>
Apr 18 14:32:46 Lab-Manager systemd[1]: Started NetBox WSGI Service.
Apr 18 14:32:46 Lab-Manager gunicorn[42393]: [2023-04-18 14:32:46 +0000] [42393] [INFO] Starting gunicorn 20.1.0
Apr 18 14:32:46 Lab-Manager gunicorn[42393]: [2023-04-18 14:32:46 +0000] [42393] [INFO] Listening at: http://172.25.65.135:8001 (42393)
Apr 18 14:32:46 Lab-Manager gunicorn[42393]: [2023-04-18 14:32:46 +0000] [42393] [INFO] Using worker: gthread
Apr 18 14:32:46 Lab-Manager gunicorn[42415]: [2023-04-18 14:32:46 +0000] [42415] [INFO] Booting worker with pid: 42415
Apr 18 14:32:46 Lab-Manager gunicorn[42416]: [2023-04-18 14:32:46 +0000] [42416] [INFO] Booting worker with pid: 42416
Apr 18 14:32:46 Lab-Manager gunicorn[42417]: [2023-04-18 14:32:46 +0000] [42417] [INFO] Booting worker with pid: 42417
Apr 18 14:32:47 Lab-Manager gunicorn[42418]: [2023-04-18 14:32:47 +0000] [42418] [INFO] Booting worker with pid: 42418
Apr 18 14:32:47 Lab-Manager gunicorn[42419]: [2023-04-18 14:32:47 +0000] [42419] [INFO] Booting worker with pid: 42419
nginx status
service nginx status
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-04-18 14:34:56 UTC; 16s ago
Docs: man:nginx(8)
Process: 42611 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 42624 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 42625 (nginx)
Tasks: 9 (limit: 19056)
Memory: 8.4M
CGroup: /system.slice/nginx.service
├─42625 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─42626 nginx: worker process
├─42627 nginx: worker process
├─42628 nginx: worker process
├─42629 nginx: worker process
├─42630 nginx: worker process
├─42631 nginx: worker process
├─42632 nginx: worker process
└─42633 nginx: worker process
Apr 18 14:34:56 Lab-Manager systemd[1]: Starting A high performance web server and a reverse proxy server...
Apr 18 14:34:56 Lab-Manager systemd[1]: Started A high performance web server and a reverse proxy server.
Please help and thank you in advance!
—
Reply to this email directly, view it on GitHub<#12284>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM7A5AGU5ZGF3GSOE3DXB2USXANCNFSM6AAAAAAXCYQS6M>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Wow so I did :80 instead and it turns out it has been working this whole time. Is there anything else I can change with this config? I feel like I am doing some nonsense for the server name, but I do not have any DNS set up in this private lab network. |
Beta Was this translation helpful? Give feedback.
Answer selected by
network-plus-code
-
|
We stumbled upon this issue of "Static Media Failure" and it turns out SELinux labelling was not applied to the static folder. Here is a quick way to adjust the SELinux permissions on the folder: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
A hint for people having the selinux issue, if you are following the netbox instructions, you probably created links in /opt to the current release, use the selinux commands on the real path, not the link i.e. /opt/netbox-4.1.x/netbox/static. SELinux commands does not work on linked path. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have read all of the discussions on this and none have helped to resolve my issue. (At least not that I could determine)
-Static files are all there, I ran the upgrade which runs the collections.py
-The users defined netbox.service and rq.service have access to the directories (It was already not working when most were owned by root, and still not working when I just made everything owned by netbox.)
-This is all mostly just the default install, I changed the local server IPs and proxy pass so I could use a browser from a remote device since there is no gui on the server running this, however even if that is an issue in itself, when it was on the default localhost ip and port I would get the static media failure when curling http://127.0.0.1:8001
Netbox itself will work if I run it as a development server at either ips.
The issue is something with gunicorn or nginx. I have the services running in systemd with the user netbox.
I can see that netbox can access
I can see the processes running as netbox
I tried it where just static and media were netbox and everything else was root. I then just made everything netbox. I can change it back if this caused more problems but the permission is likely not the issue.
Gunicorn Config
nginx config
Netbox status
nginx status
Please help and thank you in advance!
Beta Was this translation helpful? Give feedback.
All reactions