显示:服务器内部错误

[ooabcoo]

一直抓取不到代理
/api/status/
{"code": 0, "msg": "success", "data": {"spiders": [{"status": "stopped", "name": "coderbusy", "last_start_time": "1533775630"}, {"status": "stopped", "name": "kuaidaili", "last_start_time": "1533775630"}, {"status": "stopped", "name": "mix", "last_start_time": "1533775630"}, {"status": "stopped", "name": "data5u", "last_start_time": "1533775630"}, {"status": "stopped", "name": "xicidaili", "last_start_time": "1533775079"}, {"status": "stopped", "name": "checker", "last_start_time": "1533776171"}, {"status": "stopped", "name": "coolproxy", "last_start_time": "1533775079"}, {"status": "stopped", "name": "3464", "last_start_time": "1533775630"}, {"status": "stopped", "name": "yundaili", "last_start_time": "1533775630"}, {"status": "stopped", "name": "ip66", "last_start_time": "1533775630"}], "proxies": {"total": 0, "detail": {"http": 0, "https": 0, "transparent": 0, "anonymous": 0}}}}
/api/spider/run_all/
{"code": 500, "msg": "\u670d\u52a1\u5668\u5185\u90e8\u9519\u8bef", "data": {}}
显示:服务器内部错误
环境:Debian 9 x64 (stretch),python3.6.5

[ooabcoo]

老大,您留下QQ吧,我的QQ:929144

[ooabcoo]

I [2018-08-09 01:39:16,636] [HeartBeat.start] [-] Server heartbeat count: 3720 I [2018-08-09 01:39:16,645] [.deploy_spider] [-] Started <class 'proxy_spider.spiders.checker.CheckerSpider'> at 1533778756. Key: fp_server:spider:checker:checker. I [2018-08-09 01:39:16,659] Overridden settings: {'AUTOTHROTTLE_ENABLED': True, 'AUTOTHROTTLE_MAX_DELAY': 30, 'BOT_NAME': 'proxy_spider', 'CONCURRENT_REQUESTS': 32, 'DOWNLOAD_DELAY': 3, 'EDITOR': 'vim', 'HTTPCACHE_ENABLED': True, 'HTTPCACHE_EXPIRATION_SECS': 900, 'LOG_STDOUT': True, 'NEWSPIDER_MODULE': 'proxy_spider.spiders', 'RETRY_HTTP_CODES': [400, 500, 502, 503, 504, 408], 'RETRY_TIMES': 10, 'SPIDER_MODULES': ['proxy_spider.spiders']} I [2018-08-09 01:39:16,663] Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.throttle.AutoThrottle'] I [2018-08-09 01:39:16,665] [.run_spider] [-] Got crawler: <scrapy.crawler.Crawler object at 0x7fe6eaaefa20> I [2018-08-09 01:39:16,671] Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'proxy_spider.middlewares.RandomUserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'proxy_spider.middlewares.PureRedisMiddleware', 'proxy_spider.middlewares.TimerMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats', 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware'] I [2018-08-09 01:39:16,674] Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] I [2018-08-09 01:39:16,676] Enabled item pipelines: ['proxy_spider.pipelines.PersistencePipeline'] I [2018-08-09 01:39:16,677] Spider opened I [2018-08-09 01:39:16,683] [.callback_unregister_status] [-] One spider finished working. key: fp_server:spider:checker:checker. Total time: 0

[Karmenzind]

加你了
spider run all是我自己测试用的接口

我自己电脑上一直在长期运行，没发现你这个问题，我的是archlinux

[Karmenzind]

似乎和#4 是同一个问题

[ooabcoo]

fp-server_12345.log

[ooabcoo]

fp-server_5000.log

[ooabcoo]

/api/status/
运行正常：
{"code": 0, "msg": "success", "data": {"spiders": [{"status": "running", "name": "xicidaili", "last_start_time": "1533810584", "total_time": 46277}, {"status": "running", "name": "coolproxy", "last_start_time": "1533856358", "total_time": 503}, {"status": "running", "name": "coderbusy", "last_start_time": "1533810584", "total_time": 46277}, {"status": "stopped", "name": "3464", "last_start_time": "1533855754"}, {"status": "running", "name": "kuaidaili", "last_start_time": "1533838267", "total_time": 18594}, {"status": "running", "name": "data5u", "last_start_time": "1533856358", "total_time": 503}, {"status": "stopped", "name": "yundaili", "last_start_time": "1533856358"}, {"status": "stopped", "name": "ip66", "last_start_time": "1533832244"}, {"status": "stopped", "name": "mix", "last_start_time": "1533855754"}, {"status": "running", "name": "checker", "last_start_time": "1533856780", "total_time": 81}], "proxies": {"total": 6785, "detail": {"http": 4259, "https": 2526, "transparent": 1431, "anonymous": 5354}}}}

[ooabcoo]

/api/proxy/
运行正常：
{"code": 0, "msg": "success", "data": {"count": 1, "detail": [{"ip": "142.44.247.60", "scheme": "http", "port": "80", "need_auth": "False", "url": "http://142.44.247.60:80", "anonymity": "anonymous"}]}}

[ooabcoo]

/api/proxy/?count=10&scheme=HTTP&anonymity=anonymous
运行正常：
{"code": 0, "msg": "success", "data": {"count": 10, "detail": [{"ip": "139.255.110.3", "scheme": "http", "port": "53281", "need_auth": "False", "url": "http://139.255.110.3:53281", "anonymity": "anonymous"}, {"ip": "195.201.186.107", "scheme": "http", "port": "3128", "need_auth": "False", "url": "http://195.201.186.107:3128", "anonymity": "anonymous"}, {"ip": "154.119.50.246", "scheme": "http", "port": "53281", "need_auth": "False", "url": "http://154.119.50.246:53281", "anonymity": "anonymous"}, {"ip": "36.89.194.195", "scheme": "http", "port": "53281", "need_auth": "False", "url": "http://36.89.194.195:53281", "anonymity": "anonymous"}, {"ip": "163.125.250.172", "scheme": "http", "port": "8118", "need_auth": "False", "url": "http://163.125.250.172:8118", "anonymity": "anonymous"}, {"ip": "85.187.246.109", "scheme": "http", "port": "8080", "need_auth": "False", "url": "http://85.187.246.109:8080", "anonymity": "anonymous"}, {"ip": "114.235.22.90", "scheme": "http", "port": "9000", "need_auth": "False", "url": "http://114.235.22.90:9000", "anonymity": "anonymous"}, {"ip": "37.123.222.124", "scheme": "http", "port": "8080", "need_auth": "False", "url": "http://37.123.222.124:8080", "anonymity": "anonymous"}, {"ip": "150.129.54.103", "scheme": "http", "port": "6666", "need_auth": "False", "url": "http://150.129.54.103:6666", "anonymity": "anonymous"}, {"ip": "183.147.220.110", "scheme": "http", "port": "9000", "need_auth": "False", "url": "http://183.147.220.110:9000", "anonymity": "anonymous"}]}}

[ooabcoo]

/api/spider/run_all/
内部错误：
{"code": 500, "msg": "\u670d\u52a1\u5668\u5185\u90e8\u9519\u8bef", "data": {}}

[ooabcoo]

/api/proxy/report/
404
{"code": 404, "msg": "Not found", "data": null}