ptpb.pw 500 Internal Server Error on uploading png #245

winny- · 2019-02-26T19:53:31Z

This is the image https://u.teknik.io/0E2nn.png .

$ curl -vvv -F c=@- https://ptpb.pw/ < blue-vaporwave.png
*   Trying 2600:1901:0:b294::1...
* TCP_NODELAY set
* Connected to ptpb.pw (2600:1901:0:b294::1) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=ptpb.pw
*  start date: Feb 19 13:12:59 2019 GMT
*  expire date: May 20 13:12:59 2019 GMT
*  subjectAltName: host "ptpb.pw" matched cert's "ptpb.pw"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> POST / HTTP/1.1
> Host: ptpb.pw
> User-Agent: curl/7.62.0
> Accept: */*
> Content-Length: 878740
> Content-Type: multipart/form-data; boundary=------------------------27863c45c51ccf5d
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 500 INTERNAL SERVER ERROR
< Server: openresty/1.13.6.2
< Date: Tue, 26 Feb 2019 19:51:27 GMT
< Content-Type: text/html
< Content-Length: 291
< Connection: keep-alive
< X-Varnish: 9677832
< Age: 0
< Via: 1.1 varnish (Varnish/6.0)
* HTTP error before end of send, stop sending
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):

The text was updated successfully, but these errors were encountered:

buhman · 2019-02-26T21:50:16Z

Not sure what was going on here; I was not able to observe a meaningful server-side error.

I restarted the pb container and am no longer able to reproduce this.

Sorry about that.

buhman · 2019-02-26T21:51:37Z

➜  ~ curl -q https://u.teknik.io/0E2nn.png | curl -D- -F c=@- https://ptpb.pw                         
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Server: openresty/1.13.6.2
Date: Tue, 26 Feb 2019 21:43:27 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 233
Connection: keep-alive
Vary: Accept
Location: https://ptpb.pw/0aXq
X-Varnish: 295196
Age: 0
Via: 1.1 varnish (Varnish/6.0)
Accept-Ranges: bytes

date: 2019-02-26T21:43:27.899808+00:00
digest: 9d0a3340d2ff7f29d8f22fda10619a53edd1a5ea
long: AJ0KM0DS_38p2PIv2hBhmlPt0aXq
short: 0aXq
size: 878590
status: created
url: https://ptpb.pw/0aXq
uuid: 3fcf703f-7183-4905-809a-cfea0a0028eb
➜  ~ curl -X DELETE https://ptpb.pw/3fcf703f-7183-4905-809a-cfea0a0028eb
date: 2019-02-26T21:43:27.899000+00:00
digest: 9d0a3340d2ff7f29d8f22fda10619a53edd1a5ea
long: AJ0KM0DS_38p2PIv2hBhmlPt0aXq
short: 0aXq
size: 878590
status: deleted
url: https://ptpb.pw/0aXq

winny- · 2019-02-26T22:03:42Z

Thanks! I can confirm the issue is resolved.

winny- · 2019-03-07T22:49:34Z

It looks like the same fault has occurred again. It is pretty sporadic.

winston@snowcrash ~ $ date;echo;curl -sS https://u.winny.tech/lolwut1.webm | curl -D- -Fc=@- https://ptpb.pw/
Thu Mar  7 16:48:11 CST 2019

HTTP/1.1 100 Continue

HTTP/1.1 500 INTERNAL SERVER ERROR
Server: openresty/1.13.6.2
Date: Thu, 07 Mar 2019 22:48:12 GMT
Content-Type: text/html
Content-Length: 291
Connection: keep-alive
X-Varnish: 8382151
Age: 0
Via: 1.1 varnish (Varnish/6.0)

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

buhman · 2019-03-07T23:15:13Z

:(

buhman · 2019-03-08T08:09:42Z

I still have completely no idea what's happening here.

I reproduced this, and attempted a label upload:

curl -D- -F c=@- https://ptpb.pw/~lolwut1234.webm < lolwut1.webm

The string lolwut1234 does not appear in the pb application logs, indicating pb never serviced this request at all.

I then restarted openresty, and only openresty:

buhman@instance-ptpb-aio ~ $ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                      NAMES
f4d7ae94229b        ptpb/openresty      "/usr/local/openrest…"   3 seconds ago       Up 1 second         0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   openresty
5d583f3111c9        ptpb/varnish        "varnishd -F -a :608…"   5 days ago          Up 5 days                                                      varnish
37fe62b92367        ptpb/pb             "uwsgi --ini pb.ini"     5 days ago          Up 5 days                                                      pb
e36df6481124        mongo:4             "docker-entrypoint.s…"   3 weeks ago         Up 3 weeks          27017/tcp                                  mongo

I was then no longer able to reproduce the issue. This is not helpful because:

the openresty logs are currently far too noisy to be useful (and probably don't contain relevant information anyway)
varnish clearly handled the request, as indicated in the headers, though it doesn't generate persistent logs

buhman · 2019-03-08T08:34:06Z

I vaguely suspect this is some "resource not completely closed" issue between openresty/varnish. The good/bad news is this appears to be triggered over a large number of days. Reproducing this in a development environment might be tricky; some fun stats about the ptpb.pw deployment:

varnish 98.96% cache hit rate (last 5 days)
varnish response body egress: 2.95Tbytes (last 5 days) (also confirmed by my GCP bill 🤒 )
varnish request body ingress: 106.19Gbytes (last 5 days) (garbage requests?)

Over the most recent ~25,000 requests, at least 90% were related to wownero/monero command-and-control/executable distribution (specifically the minexmr.com operators appear to be abusing ptpb.pw for free network transit).

buhman · 2019-03-08T08:35:41Z

Top 5 pastes on ptpb.pw right now:

/Nqo-
/DXpq
/KAlo
/D8r9
/hgZI

buhman · 2019-03-08T08:48:16Z

Contributing factors:

request volume is currently ~biblical, compared to historical volume
server-side errors aren't being aggregated/collected in a way that is useful for this volume

To properly catch this, I'll need to:

configure persistent error logging for openresty/varnish
deploy some form of log aggregation
add a ~daily smoke test for this symptom

It would also be convenient to generate/log/return request identifiers at all levels in the stack.

buhman · 2019-03-08T09:01:53Z

Tangentially I might also need to shut down ptpb.pw, until/unless I implement long-needed abuse mitigation features.

ptpb.pw has shutdown ptpb/pb#245 ptpb/pb#240

mia-0 · 2019-03-09T17:44:41Z

I run 0x0.st and I have also observed increasingly crafty attempts at uploading miner payloads, configuration files and so on. However I am actively filtering them, as well as deploying more aggressive firewall rules and malware scanning.

This and some idiot scanning all possible file URLs via anonymizing networks have caused well over 15 TB of monthly traffic since November 2018 (which doesn’t cost me a penny because I don’t run on shitty cloud hosting). It seems I have mitigated that for now.

Edit: After inspecting logs, I can assume that a majority of requests has been blocked by my dynamic firewall rules before it could hit the web server. Props to https://github.com/firehol/firehol for making this easy.

ptpb.pw has shutdown ptpb/pb#245 ptpb/pb#240

ptpb.pw has been our default provider since 2017-03-23T21:10:25+01:00, this was introduced in the commit 993db7e. But now ptpb.pw has been shut down due to extensive abuse: ptpb/pb#245 ptpb/pb#240 ptpb.pw, you were the best paste service out there and have served webpaste.el and many others very well. You will be missed. So Long, and Thanks for All the Fish.

ptpb.pw has been our default provider since 2017-03-23T21:10:25+01:00, this was introduced in the commit 993db7e. But now ptpb.pw has been shut down due to extensive abuse: ptpb/pb#245 ptpb/pb#240 ptpb.pw, you were the best paste service out there and have served webpaste.el and many others very well. You will be missed. So Long, and Thanks for All the Fish. This fixes #36.

ptpb.pw has shutdown ptpb/pb#245 ptpb/pb#240

buhman added the deployment label Feb 26, 2019

buhman closed this as completed Feb 26, 2019

buhman reopened this Mar 7, 2019

buhman mentioned this issue Mar 8, 2019

ptpb.pw permanent shutdown #246

Closed

wandersoncferreira mentioned this issue Mar 8, 2019

ptpb was discontinued etu/webpaste.el#36

Closed

ghost mentioned this issue Mar 9, 2019

ptpb.pw is out of disk space #240

Closed

hekel added a commit to hekel/sopel that referenced this issue Mar 9, 2019

Replace pastebin service for help

bf0b0d4

ptpb.pw has shutdown ptpb/pb#245 ptpb/pb#240

hekel mentioned this issue Mar 9, 2019

Replace pastebin service for help sopel-irc/sopel#1495

Merged

hekel added a commit to hekel/sopel that referenced this issue Mar 9, 2019

Replace pastebin service for help

73d8b4e

ptpb.pw has shutdown ptpb/pb#245 ptpb/pb#240

kwaaak pushed a commit to kwaaak/sopel that referenced this issue Mar 25, 2019

Replace pastebin service for help

5c50cb6

ptpb.pw has shutdown ptpb/pb#245 ptpb/pb#240

kotajacob mentioned this issue Mar 31, 2019

Screenshots are down kotajacob/wal_steam#73

Open

buhman closed this as completed May 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ptpb.pw 500 Internal Server Error on uploading png #245

ptpb.pw 500 Internal Server Error on uploading png #245

winny- commented Feb 26, 2019

buhman commented Feb 26, 2019

buhman commented Feb 26, 2019 •

edited

Loading

winny- commented Feb 26, 2019

winny- commented Mar 7, 2019

buhman commented Mar 7, 2019

buhman commented Mar 8, 2019

buhman commented Mar 8, 2019 •

edited

Loading

buhman commented Mar 8, 2019

buhman commented Mar 8, 2019

buhman commented Mar 8, 2019 •

edited

Loading

mia-0 commented Mar 9, 2019 •

edited

Loading

ptpb.pw 500 Internal Server Error on uploading png #245

ptpb.pw 500 Internal Server Error on uploading png #245

Comments

winny- commented Feb 26, 2019

buhman commented Feb 26, 2019

buhman commented Feb 26, 2019 • edited Loading

winny- commented Feb 26, 2019

winny- commented Mar 7, 2019

buhman commented Mar 7, 2019

buhman commented Mar 8, 2019

buhman commented Mar 8, 2019 • edited Loading

buhman commented Mar 8, 2019

buhman commented Mar 8, 2019

buhman commented Mar 8, 2019 • edited Loading

mia-0 commented Mar 9, 2019 • edited Loading

buhman commented Feb 26, 2019 •

edited

Loading

buhman commented Mar 8, 2019 •

edited

Loading

buhman commented Mar 8, 2019 •

edited

Loading

mia-0 commented Mar 9, 2019 •

edited

Loading