Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

node: Improve live progress output (and other small tweaks) #315

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

refi64
Copy link
Collaborator

@refi64 refi64 commented Aug 26, 2022

Second commit message has the basic summary:

This upgrades the progress output quite a bit:

  • There's a pretty animation at the start of the line!
  • Instead of printing the last package interacted with (which made no
    sense), show as many of the currently-processing packages as can fit
    on a single line.
  • Show live download progress for large downloads, so any that are
    holding up generation don't result in appears to be a hang.

In addition, this adds a hidden CLI flag, --traceback-on-interrupt, that
prints a full traceback from every active package coroutine on Ctrl-C,
which was useful while debugging this (and probably will be in the
future!)

Live demo (with a weirdly low framerate, but it actually looks smooth live, not sure why it was being weird here):

Screencast.from.08-26-2022.05.59.29.PM.webm

This may or may not have taken like, 6 hours to figure out. (Terminals are hard.)

@gasinvein
Copy link
Member

There is a plenty or ready-made "progressbar" libs, honestly I'd prefer to use one of those instead of implementing (or extending) our own.

@refi64
Copy link
Collaborator Author

refi64 commented Sep 23, 2022

Right, so I actually started out using tqdm but ended up removing it, because as it turns out, I didn't actually want a progress bar, per se, and it definitely wasn't a single line.

The main things are that, if you see in the video:

  • The terminal space is used to show the packages being processed. I felt like this was a lot more useful, esp since it was clearer what packages might actually be holding things up. We can't fit as many on one line with a progress bar there
  • Multiple progress lines might be spawned when packages need big files. Unfortunately, although tqdm handles nested progress bars well, it does not like it when a progress bar in the middle disappears before the ones around it:
    from tqdm import tqdm
    import time
    
    p1 = tqdm(total=10)
    p2 = tqdm(total=10)
    p3 = tqdm(total=10)
    
    for bar in p2, p3, p1:
        for i in range(5):
            time.sleep(0.2)
            bar.update(i)
    
        bar.close()
    Here, after the middle bar closes, four lines appear on the screen, with one 100% line being repeated twice.
  • There's no particular guarantee we'd have a length when downloading a file, and if we don't...tqdm's support for some display of infinite progress isn't that great.
  • ...tqdm doesn't let me easily do the pretty animation.

The main loss is honestly time estimates, but given that some packages require extra downloads and such anyway, those weren't super useful.

Additionally, although there are a bunch of other libraries...they all seem to be missing something relatively tricky that we'd have to add anyway, most particularly the proper handling of multiple progress bars. Hence, DIY progress bars.

(Plus, the code isn't that untested, I ripped the core logic from an old tool in C++ that had multiple lines of progress that would finish at inconsistent intervals. Feel free to try to break it though!)

This upgrades the progress output quite a bit, largely thanks to the
rich CLI library:

- There's a pretty animation at the start of the line!
- Instead of printing the last package interacted with (which made no
  sense), show as many of the currently-processing packages as can fit
  on a single line.
- Show live download progress for large downloads, so any that are
  holding up generation don't appears to be hanging.

In addition, this adds a hidden CLI flag, --traceback-on-interrupt, that
prints a full traceback from every active package coroutine on Ctrl-C,
which was useful while debugging this (and probably will be in the
future!)

Signed-off-by: Ryan Gonzalez <[email protected]>
@refi64
Copy link
Collaborator Author

refi64 commented Dec 16, 2022

So as it turns out, there is a nice library that does what I need, it's just not marketing as a progress library: Rich. It also gives some very fancy and nice progress bars! So, the current output is:

Screencast.from.2022-12-16.16-44-59.webm

(ignore the random frame drops at the 0:07 and 0:14 mark, no idea what those are from)

Ideally I'd like to add some colors which Rich makes relatively easy, but this is already a big improvement and that can be added later.

@hfiguiere hfiguiere added the node label Apr 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants