Setting verbose=False issues with threads

[jeberger]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Observed Behavior

Setting verbose = False when creating a Llama instance causes all program output to disappear while the model is running. This includes in particular logging output from other threads. In addition, if I create several Llama instances and use them in parallel from multiple threads, I sometimes receive some (but not all) output from one of the instances (I assume that's because another instance finished running first and restored the output for everyone). And sometimes output is not restored when all instances have finished (probably because the last instance to finish restored output to a blocked state).

Expected Behaviour

Setting verbose = False when creating a Llama instance should cause output from that instance to disappear. Other program output should still be visible (this includes logging output and output from other Llama instances).

[jeberger]

As a workaround for logging on Linux, you can duplicate the stderr file descriptor and log to that instead of the true stderr. E.g.:

logging.basicConfig (
    level = LOG_LEVEL, format = LOG_FORMAT,
    handlers = [
        logging.StreamHandler (os.fdopen (
            os.dup (sys.stderr.fileno()), 'w'))
    ])

This allows logging to proceed even while output has been blocked by llama.cpp.