Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Backlog read keeps failing with "simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)" #400

Open
AdamWill opened this issue Mar 8, 2017 · 1 comment
Open

Backlog read keeps failing with "simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)" #400

AdamWill opened this issue Mar 8, 2017 · 1 comment
Labels
bug

Comments

@AdamWill
Copy link
Contributor

AdamWill commented Mar 8, 2017

When restarting fedmsg-hub on one of our infra systems (openqa01), I keep getting this error:

[2017-03-08 16:55:16][    fedmsg    INFO] Retrieving datagrepper page 55 of 518
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 171, in _backlog
    for message in self.get_datagrepper_results(then, now):
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 197, in get_datagrepper_results
    data = _make_query(page=page)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 183, in _make_query
    rows_per_page=100, page=page, start=then, end=now, order='asc'
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 812, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 171, in _backlog
    for message in self.get_datagrepper_results(then, now):
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 197, in get_datagrepper_results
    data = _make_query(page=page)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 183, in _make_query
    rows_per_page=100, page=page, start=then, end=now, order='asc'
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 812, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 171, in _backlog
    for message in self.get_datagrepper_results(then, now):
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 197, in get_datagrepper_results
    data = _make_query(page=page)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 183, in _make_query
    rows_per_page=100, page=page, start=then, end=now, order='asc'
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 812, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 171, in _backlog
    for message in self.get_datagrepper_results(then, now):
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 197, in get_datagrepper_results
    data = _make_query(page=page)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 183, in _make_query
    rows_per_page=100, page=page, start=then, end=now, order='asc'
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 812, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 171, in _backlog
    for message in self.get_datagrepper_results(then, now):
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 187, in get_datagrepper_results
    data = _make_query()
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 183, in _make_query
    rows_per_page=100, page=page, start=then, end=now, order='asc'
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 812, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib64/python2.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 171, in _backlog
    for message in self.get_datagrepper_results(then, now):
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 187, in get_datagrepper_results
    data = _make_query()
  File "/usr/lib/python2.7/site-packages/fedmsg/consumers/__init__.py", line 183, in _make_query
    rows_per_page=100, page=page, start=then, end=now, order='asc'
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 812, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This has happened twice now - @ralphbean advised wiping the history files to get around it the first time, I did, now it's happening again.
@jeremycline jeremycline self-assigned this Sep 11, 2017
@jeremycline
Copy link
Member

There was no error handling when querying datagrepper for messages which is what caused this. This leads me to wonder what we should do when there are problems. Halt, or go on without message playback after logging the issue? Going on without message playback is easy enough, but leads to a lot of missed messages. That has been the case 100% of the time for months now, though...

Halting and letting the user wait for datagrepper to come back (or explicitly disabling it) feels like the better choice, but looks like it will require serious refactoring since what currently happens is playback is done concurrently with regular message subscription and when a message arrives via ZeroMQ fedmsg just overwrites the old history file.

@jeremycline jeremycline removed their assignment Sep 11, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AdamWill @jeremycline