Inline images

[jubalh]

PR for #1301

So far this is just a test. Displaying an image ~/test.png when Profanitys "about" is printed.
So far in ASCII art.

libchafa can also return colourful images using escape codes. But not sure how to use them inside ncurses yet.

I assume later we want a setting that either displays all images (URLs which we assume to be images based on ending) or we want it only explicitly with somethine like /image / /imagepreview / /inlineimage where one can tab through the latest URLs with image endings.

[jubalh]

chafa has a test branch with added functionality for us:
https://github.com/hpjansson/chafa/tree/wip/cell-access
example: https://github.com/hpjansson/chafa/blob/wip/cell-access/tests/ncurses.c

Need to build this chafa version and see whether this helps progressing on this task.

[ghost]

i'm pretty sure using viu would work too, no? It displays stuff with colored blocks, or uses Kitty's image protocol if available, which i guess would be similar to what you're looking for, plus Kitty users could get stuff in high quality

[jubalh]

viu is an cli image viewer. There are plenty of those. It outputs things on the terminal we can't use that to display a picture inside our chat windows. viuer is the library that viu uses. It is written in Rust. Profanity is written in C. So we we settled on libchafa.

[hpjansson]

I merged the wip/cell-access branch into master some time ago. I don't expect it to change very much between now and release, so it should be safe to start using.

I'm also planning Kitty and iTerm2 support, but I'm not sure how you'd get ncurses to cooperate...

[jubalh]

@binex-dsk a little similar to the QR code task. Do you think that would be something for you? ;)

[ghost]

@binex-dsk a little similar to the QR code task. Do you think that would be something for you? ;)

I could possibly do basic block art in the same vain as the QR code. Not sure if colorful images would be my forte, but I'll look into it.

Also: are shell color codes agnostic to operating systems/shell standards?

Also, I'm currently on a band trip without access to a computer. I'll see if I want to tackle this after next Sunday/Monday.

[jubalh]

I could possibly do basic block art in the same vain as the QR code. Not sure if colorful images would be my forte, but I'll look into it.

In this PR I tried to use the library "chafa" actually it should do most of the work. We just need set the correct settings (what it will return) and then print this properly in the window.

Like chafa_symbol_map_add_by_tags (symbol_map, CHAFA_SYMBOL_TAG_ASCII); and there are various thing to set there. I also think @hpjansson added new options since I tried this last time.

Also: are shell color codes agnostic to operating systems/shell standards?

I'm not entirely sure. But I don't think so.

Also, I'm currently on a band trip without access to a computer. I'll see if I want to tackle this after next Sunday/Monday.

Enjoy!

[hpjansson]

I recommend reading the ncurses example @jubalh mentioned before. I've tried to comment it liberally, but if anything is unclear I'll gladly answer questions and review code.

As an aside, I think it's possible to support Kitty and sixel graphics in ncurses too, but you'd have to print raw character codes, so it probably wouldn't work inline (ncurses repaints would likely mess it up). But e.g. vifm uses a frame in a fixed location, and that seems to work (using fileviewer *.png,*.jpg chafa -f kitty -s %pwx%ph %c:p %pd %N 2>/dev/null).

[ghost]

In this PR I tried to use the library "chafa" actually it should do most of the work. We just need set the correct settings (what it will return) and then print this properly in the window.

Like chafa_symbol_map_add_by_tags (symbol_map, CHAFA_SYMBOL_TAG_ASCII); and there are various thing to set there. I also think @hpjansson added new options since I tried this last time.

Okay, I see. In that case I'll mess around with the settings and see what I can do. What is it you're looking for that this PR does not yet do (other than deciding on what to do for displaying them, of course)

And should I test this on the cell-access thing mentioned? Or on master?

EDIT: cell-access is old and merged, so I'll test this on master. And also I won't have a huge amount of time again (Robotics, summer classes, sports, etc...) but depending on what's needed I'm confident I'll be able to get it done quickly.

[hpjansson]

If your package manager has Chafa 1.8.0 or later, the cell access API will be there, so you can build against the devel package instead of master. In case it saves time.

[ghost]

If no one else takes this up within a few weeks I'll go ahead and do it... right now I've got a boatload of stuff to do: summer classes, driving practice, and soccer. I might experiment with the ncurses examples at some point though :)

[mrusme]

Having inline images rendered via Chafa would really be awesome! I have only scratched the surface here, but this also requires integration with OMEMO in case of aesgcm:// URLs, correct?

[jubalh]

@binex-dsk how are your driving classes? Do you have time for this feature now maybe?

[ghost]

@binex-dsk how are your driving classes? Do you have time for this feature now maybe?

Finished them up. I have my driver's test this week and soccer (should) end this week as well, so I will begin work for real (like, actually for real this time I swear) next week.

[jubalh]

That's great news!

[ghost]

Quick update, I have been doing some basic libchafa testing (on its own, not with profanity) and although I haven't made that much progress, I AM getting there. I would be farther but, I had to go to urgent care, which (unfortunately) is more important than programming...

[jubalh]

Thanks for the update!
Take your time and take care!

[ghost]

I've been messing with this for a while and so far I've not been able to figure out several things.

Here's the point I've reached:

I have yet to figure out:

• True color; term info IS properly detected but truecolor mode is not yet working. In my previous tests I never managed to get this to work either...
• Kitty pixel support; Also never managed to figure this out in previous tests.
• Wrong aspect ratio; Never a problem in my previous tests.

You can also notice that the pixels don't look right. It's a QR code so it should be all squares.

Commit is here:

https://github.com/binex-dsk/profanity/commit/8e5aa52c72fbb99b8ca4b50b8a7278521ad56b83

[ghost]

Looked into the ncurses example a bit more and I'm confused.

@hpjansson Would you mind explaining the canvas_to_ncurses function in the ncurses example? I believe this is the function we need for truecolor but I haven't managed to decipher it.

Also, how would we go about actual Kitty support? It seems like it would be really hard to make it work with kitty, though I don't know the internals of the Kitty protocol so maybe I'm missing something.

[hpjansson]

@hpjansson Would you mind explaining the canvas_to_ncurses function in the ncurses example? I believe this is the function we need for truecolor but I haven't managed to decipher it.

Sure! It iterates over the ChafaCanvas character cells, row by row and column by column. For each cell, it:

• Extracts the Unicode character with chafa_canvas_get_char_at() and turns it into a wchar_t, which is what ncurses wants.
• Extracts the cell's BG and FG colors using chafa_canvas_get_colors_at(), which yields signed ints that are either -1 for transparency or else an RGB triplet in the lower three bytes (truecolor mode -- other modes use the _raw equivalent, which gets -1 or indexes from a fixed palette).
• Initializes a corresponding "color pair" for ncurses using init_extended_pair().
• Initializes an ncurses cchar_t structure combining the character, color pair and default attributes using setcchar().
• Puts the resulting cchar_t to the ncurses window using mvadd_wch().

The ncurses functions init_extended_pair, setcchar and mvadd_wch can be looked up with man in a shell. The Chafa functions have documentation pages on the web.

Note that you need to link with the correct build of ncurses (e.g. ending in w6 for wide characters), and your TERM variable must be set to a terminal type that supports directcolor for this to work (e.g. TERM=xterm-direct).

Also, how would we go about actual Kitty support? It seems like it would be really hard to make it work with kitty, though I don't know the internals of the Kitty protocol so maybe I'm missing something.

I think this would be impossible, or at least not worth the effort, to support in an ncurses program unless there's a non-scrolling preview rectangle where you can just bypass ncurses and overwrite the contents using your own escape codes. I think vifm does this (see: hpjansson/chafa#77, although it's calling chafa externally).

[ghost]

Note that you need to link with the correct build of ncurses (e.g. ending in w6 for wide characters)

Aha, that's why the function didn't end up working after I plugged it in!

An interesting thing I'll need to look into is getting it to work with the standard win_println-style of printing--that is, with the time and indented as other messages are. Although it might be better to end up displaying it without the indent/time, and instead, image messages have text content "Image" or "Media Message" or similar, and then in the next line the image is displayed.

I'm unfamiliar with the internal workings of the profanity window thingy so I have no clue if it'll properly scroll if printed with mvadd_wch. Although a brief look at the print functions does suggest it will.

I suppose I'll look into all of this, perhaps more, tomorrow. Depends how much free time I've got. Which might not be a lot, because God knows if I'll end up needing to go to the ER again.

I think this would be impossible, or at least not worth the effort, to support in an ncurses program unless there's a non-scrolling preview rectangle where you can just bypass ncurses and overwrite the contents using your own escape codes.

This is what I suspected. I have an idea on how that could work, but it would be clunky, awkward, perhaps user-unfriendly, and not to mention a HUGE pain to program.

Thanks for the assistance @hpjansson; this makes a lot more sense now!

[ghost]

After a test using canvas_to_ncurses, mvadd_ch does not properly print the image to the window... I think what we should do is add a way to print a wchar array, and in the canvas_to_ncurses function, we construct a wchar array and use that to print. But the former is beyond my area of expertise...

Also, if ncursesw isn't found we probably have to do basic ASCII/block art

[ghost]

@jubalh perhaps you could help with the wchar printing? I have no idea how to implement this, as I'm not familiar with the internal print functions.

[jubalh]

@binex-dsk ok I will try to look at it! :)

[ghost]

@jubalh how has the wchar printing progress coming? Perfectly okay if you haven't done anything yet

[jubalh]

@binex-dsk sorry I haven't yet. I'm having some health issues. I will try to look at it when all is okay again :)

[hpjansson]

Just from a quick look, it looks like buffer.c would be a good place to start. You could add code there to queue messages with images in them. Then window.c has a redraw function and print functions -- functionality could be added there to paint the contents of images from the buffer object.

I'd store the original raster image (after decoding from the network) in the buffer entry, and call Chafa every time it needs to redraw. That lets you deal with window resizes etc. gracefully after the message is received.

[hpjansson]

Also -- @jubalh, take care and get well soon!

[ghost]

I'm having some health issues

Lmao same

Hope you get well! :)