Skip to content
This repository has been archived by the owner on Apr 3, 2019. It is now read-only.

Blocking on connect to redis blocks the whole process #84

Open
gward opened this issue Jan 12, 2015 · 2 comments
Open

Blocking on connect to redis blocks the whole process #84

gward opened this issue Jan 12, 2015 · 2 comments

Comments

@gward
Copy link

gward commented Jan 12, 2015

If I make a query to redis that needs to open a connection, and that connection blocks (eg. slow network, network partition), then the whole tornado process blocks. No requests can be handled, even requests that do not depend on redis.

Example: here is a simple tornado HTTP service that handles two URLs, /r1 and /r2. /r1 does not depend on redis at all: it immediately returns a hardcoded string. /r2 fetches a value from redis and returns it.

import tornadoredis
import tornado.httpserver
import tornado.web
import tornado.ioloop
import tornado.gen


class R1Handler(tornado.web.RequestHandler):
    '''handle a request which does not depend on redis'''
    @tornado.web.asynchronous
    @tornado.gen.engine
    def get(self):
        self.write('all is well\n')
        self.finish()


class R2Handler(tornado.web.RequestHandler):
    '''handle a request which does depend on redis'''
    @tornado.web.asynchronous
    @tornado.gen.engine
    def get(self):
        c = tornadoredis.Client()
        foo = yield tornado.gen.Task(c.get, 'foo')
        self.set_header('Content-Type', 'text/plain')
        self.write('foo = %s\n' % (foo,))
        self.finish()


application = tornado.web.Application([
    (r'/r1', R1Handler),
    (r'/r2', R2Handler),
])


if __name__ == '__main__':
    # Start the data initialization routine
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    print 'Demo is runing at 0.0.0.0:8888\nQuit the demo with CONTROL-C'
    tornado.ioloop.IOLoop.instance().start()

(I put that script in demos/mixed/app.py in the tornado-redis source tree.)

Expected outcome: if my tornado process is unable to connect to redis, then I expect queries to /r2 to fail or block, but queries to /r1 should continue to work fine.

Actual outcome: as soon as a request for /r2 blocks, requests for /r1 also block. It appears that the entire process is blocked connecting to redis.

Here's how I reproduced it. First, in one terminal run the tornado server:

$ PYTHONPATH=. python demos/mixed/app.py
Demo is runing at 0.0.0.0:8888
Quit the demo with CONTROL-C

In another window, start polling /r1:

$ while true ; do echo -n `date`": "  ; curl http://localhost:8888/r1 ; sleep 1 ; done
Mon Jan 12 13:25:01 EST 2015: all is well
Mon Jan 12 13:25:02 EST 2015: all is well
Mon Jan 12 13:25:03 EST 2015: all is well
[...]

In a third window, ensure that /r2 works:

$ redis-cli 
127.0.0.1:6379> set foo "hello world"
OK
$ curl http://localhost:8888/r2
foo = hello world

Now simulate a network partition: make all packets to redis disappear (this assumes Linux):

$ sudo iptables -F              # wipe all existing firewall rules
$ sudo iptables -A INPUT --proto tcp --dport 6379 -j DROP    # drop packets to redis

Hop over to the window that is polling /r1; it should still be working fine:

[...]
Mon Jan 12 13:27:50 EST 2015: all is well
Mon Jan 12 13:27:51 EST 2015: all is well
Mon Jan 12 13:27:52 EST 2015: all is well
[...]

Now request /r2, which blocks because redis is on the other side of a network partition:

$ curl http://localhost:8888/r2

This request blocks, which is entirely OK. (I'd like it to fail with a timeout eventually, but whatever. Not important.)

However, the /r1 poll is now blocked:

[...]
Mon Jan 12 13:28:28 EST 2015: all is well
Mon Jan 12 13:28:29 EST 2015: all is well
Mon Jan 12 13:28:30 EST 2015: [blocked here]

We can see where it's blocked by hitting Ctrl-C on the tornado process:

^CTraceback (most recent call last):
  File "demos/mixed/app.py", line 40, in <module>
    tornado.ioloop.IOLoop.instance().start()
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 607, in start
    self._run_callback(callback)
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 458, in _run_callback
    callback()
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 331, in wrapped
    raise_exc_info(exc)
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 302, in wrapped
    ret = fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/tornado/iostream.py", line 341, in wrapper
    callback(*args)
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 331, in wrapped
    raise_exc_info(exc)
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 302, in wrapped
    ret = fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/tornado/httpserver.py", line 327, in _on_headers
    self.request_callback(self._request)
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1600, in __call__
    handler._execute(transforms, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1134, in _execute
    self._when_complete(self.prepare(), self._execute_method)
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1141, in _when_complete
    callback()
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1162, in _execute_method
    self._when_complete(method(*self.path_args, **self.path_kwargs),
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1311, in wrapper
    return result
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 198, in __exit__
    return self.exception_handler(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1115, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1298, in wrapper
    result = method(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 159, in wrapper
    deactivate()
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 198, in __exit__
    return self.exception_handler(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 136, in handle_exception
    return runner.handle_exception(typ, value, tb)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 556, in handle_exception
    self.run()
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 505, in run
    yielded = self.gen.throw(*exc_info)
  File "demos/mixed/app.py", line 23, in get
    foo = yield tornado.gen.Task(c.get, 'foo')
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 153, in wrapper
    runner.run()
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 533, in run
    self.yield_point.start(self)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 371, in start
    self.func(*self.args, **self.kwargs)
  File "/data/src/tornado-redis/tornadoredis/client.py", line 690, in get
    self.execute_command('GET', key, callback=callback)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 159, in wrapper
    deactivate()
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 198, in __exit__
    return self.exception_handler(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 136, in handle_exception
    return runner.handle_exception(typ, value, tb)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 556, in handle_exception
    self.run()
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 505, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 153, in wrapper
    runner.run()
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 507, in run
    yielded = self.gen.send(next)
  File "/data/src/tornado-redis/tornadoredis/client.py", line 407, in execute_command
    self.connection.connect()
  File "/data/src/tornado-redis/tornadoredis/connection.py", line 72, in connect
    timeout=self.timeout
  File "/usr/lib/python2.7/socket.py", line 562, in create_connection
    sock.connect(sa)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
KeyboardInterrupt

The problem appears to be that tornado-redis is using the blocking socket calls to connect to redis. I suspect it should probably use tornado.tcpclient instead. I am not a tornado expert though! I just spotted that module in the docs.
@leporo
Copy link
Owner

leporo commented Jan 13, 2015

Sorry for the late reply.

There is a pull request to solve this issue: #62
It's a shame I haven't checked and integrated it yet.

Could you please check it?
There is a chance it can help in your case.

gward referenced this issue in neg3ntropy/tornado-redis Jan 16, 2015
Do not block when connecting socket
cdcec17
@gward
Copy link
Author

gward commented Jan 16, 2015

Thanks for pointing out that PR. I've reviewed it and tested it: see my comment there.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gward @leporo