let's talk about encodings #21
Comments
|
Hey, I actually had something similar in mind, in the spirit of HTTP's
I thought Alacritty does not support images? The stats from my computer and with my TE (Sixel) added to it ( Kitty:contour:Alacritty:I have really no clue why Kitty was so infinitely slow compared to the rest. But then I noticed it was also transporting 1.7 GB data for that one test. I don't know why the timing and size numbers differ that much though.
Some parser do still parse C1. But C1 in UTF-8 mode is a 2-byte sequence that does not look like an 8-bit C1 anymore. However, I doubt we change tell most of the TE devs to stop accepting C1's. Iff at all, then via mode switches. I doubt that will be realistic either. :)
Agreed on tweaks, but with regards to size. I hope it will not get much larger but instead improve on wording, of course some semantic tweaks, and surely adding TE does make sense to me, too. Adding Z-axis on top would make it even larger then (I think the latter will be a research topic for quite some more days/weeks before I can write something down based on the discussions we will have and had).
the latter is a quote from your link. I think that will be even impossible. What about broken or suddenly disconnected clients. How does the TE recover conveniently. While I agree that terminating (and embedded) sequences are the reason why we're talking here at all, I doubt this will be ever accepted.
Oh, I am not sure I do understand correctly. If I do, I think the client must know the cell pixel dimensions, so the client can (literally) pixel-perfect update single stitched cells. That goes directly against the design requirements (notion of pixels). Again, I'm not sure I did understand right. But if you intent do render rerender a single cell of an image you have previously put into
I am not sure I get you on the first part (cutting out an image from the graphic), but how can layered images help here, and why the above explained rendering of tiles of an uploaded image not help? :)
It is exactly 33% overhead in space. Other transport encodings could be added (or even as extension added later) to lower the encoding overhead. Egmont gives nice examples in the his good-image-protocol TWG post on how that could be lowered without doing too much rocket science.
I think one cannot compare pixel-image transfer vs unicode-block rendered images. But just looking at Kitty-transmitted-bytes vs Sixel-transmitted-bytes I start to feel that the Sixel way to transport pixels is quite efficient (in other words: why the hex is the former protocol so slow/inefficient?). p.s.: We don't have to come up with a perfect encoding in day-1 anyways. The size problem should only most relevant when transferring raw RGB/RGBA data as PNG/WebP do have its own compression, which in turn must be transfer-encoded. So I think it is safe to add that option to the |
|
when i refer to alacritty, i refer to the most recent @ayosec/graphics fork+branch: |
|
good feedback as always.
well, it was sending almost exactly 1000x as much data as the
well, we needn't change "every terminal" -- this would only show up in terminals supporting this protocol. not vomiting up upon seeing such a C1 would be part of supporting the protocol. if they don't advertise support, these graphics aren't being generated. right?
hah, in kitty it doesn't: dankamongmen/notcurses#1416 an interrupted graphic hangs the terminal, requiring a reset (shift+ctrl+delete). sol i agree that ESC is pretty important, starting as it does CSI and DCS and probably others. hence why i included that wart in my encoding. but the others? do we really need their capabilities, as well?
oh, absolutely. this proves necessary for just about anything. try writing some real text+bitmap code if you don't think so. you'll run into the problems very quickly. termios(4) ioctl(2)s are absolutely critical -- if your ioctl(2) for getting screen geometry doesn't fill in the cell pixel fields, i don't use bitmaps with your terminal.
i'll answer this in another comment, as it's orthogonal to encoding.
i'm aware it's 33% of space. =] when i am told i must pay 33% taxes, that gets me pretty unhappy. given the (postulated, not proven) near-linear relation of data and time in the large, that's 33% more latency. 33% more bandwidth. when you're over a 100ms ssh, that will be very, very serious.
because sixel is:
if i sent a PNG to kitty, it would be much, much less data, but i'd have to encode it up etc.
yep, as said, the best compression is going to come from real image containers. but as otherwise noted, i sometimes need work with the raw pixel data, and converting that back to PNG/WebP is something i'd like to avoid if reasonable. |
What I see of the payload description here worries me.
I have found in the year's Notcurses testing that performance in the large is absolutely dominated by the number of bytes transmitted. This is magnified when the application generating the data is remote. I collect a good amount of data on how many bytes are transmitted. Here's
notcurses-demo'sxraydemo, which plays a video scaled to the terminal size:Kitty
Alacritty
Note that the Alacritty
notcurses-demowas idle 98% of the time, while the Kitty one was idle 41% of the time. This is directly due to the 874MB transmitted to Kitty that wasn't sent to Alacritty; we sent Kitty almost 50x(!) the total amount of data, due to chunked, Base64-encoded, RGBA data as opposed to raw palette-indexed data with elided transparent pixels. Note that the Notcurses sixel implementation derives an entirely new palette for each frame of multiframe data.The source video is about 1MB:
so we have a 1920x1080 video at 1MB, becoming 18MB on a 800x1417 Alacritty, becoming 892MB on a 809x1417 Kitty.
I find this kinda disgusting, and would very much like to see it come down.
The most effective way to transmit visual data is of course using the advanced compression schemes of image-specific formats. So it's great that we're supporting containerized data, to be decoded by the terminal. I'd like to see a focus on WebP over PNG, but whatever. With WebP, we get a format that works well for both PNG-style and JPEG-style data, with transparency and other useful features. But that's neither here nor there.
Sometimes raw data is important, especially when modifying input data on the fly, and it would be desirable to transmit this in as little space as possible. This was probably the major focus of my STEGAP proposal.
Is avoiding C1 truly necessary, especially when this is a payload, especially when the size is transmitted, especially when we're in UTF8 mode anyway? It seems to be that this protocol is a sufficiently large endeavor that one's state machine can take a small tweak. If we can stomp on everything but 0x1B, we can get close to no overhead (see link). If we can stomp on 0x1B, that's even better.
Another issue I'd like to bring up is a bit unorthodox, and maybe this isn't important if we have a sufficient z-indexing/layering story: rather than using pure row-major order across the entirety of the graphic, it would be convenient for me (and maybe for terminal authors) to transmit row-major order within a cell's area. I.e. if the cell-pixel geometry is 20 tall, 10 wide, and i have a graphic that is 50 tall and 20 wide, i'd like to transmit:
why? because here's my methodology:
ncvisual_from_*())ncvisual_render())now, maybe in the next frame, i have a cell i need cut out from the graphic (again, z-indexing/layering can let me work around the need to do so). if i have a row-major byte stream (or even worse, a row-major sixel stream), this is something of an ungodly mess. i mean, i can do it--i have code to do it--but it's a ton of in-place editing and mangling:
if i was transmitting cell-major, and then row-major within that, i could blast or rebuild one region at a time. given that we've already encoded to an arbitrary byte stream at this point, we needn't worry about what would otherwise be cache-suboptimal behavior (if we encode to cell-major, we work cell-major).
Thoughts?
tl;dr: base64 encoding our RGBA is not free, not one bit
Just to see how graphics tend to dominate TUI bandwidth, even when lightly used, here's full sample output from
notcurses-demo. The only demos making extensive use of bitmaps arexray,view, andyield:Alacritty
these three demos were responsible for ~48% of the bytes, but only 14.7% of the frames.
Kitty
here, these three demos accounted for 89%(!) of transmitted bytes, but only 14.7% of the frames.
i hope that this demonstrates the value of cutting down image bytes.
The text was updated successfully, but these errors were encountered: