Make deps of several C++ header libraries to improve terminal

[WSLUser]

Windows Terminal uses these C++ libraries: https://github.com/microsoft/terminal/tree/main/oss for a few things that include type-safety, thread-safety, safer math, performance, and easier implementation of some functionalities. Since this is a C++ terminal project, I thought I'd recommend these to improve things here. These libraries work on all platforms (Linux, Windows, MacOS).

[christianparpart]

Many thanks @WSLUser . I'll definitely have a look at it.

p.s.: fmtlib (and range-v3) I am already using. boost I'll definitely not use, but I'll look at the rest tomorrow. thanks :)

[WSLUser]

Yeah they really didn't want to import anything of boost but there were certain things within it that proved too helpful for some things in the project. They have kept the amount of boost code down to as safe a minimum as they could possibly do. Undoubtely it could be reduced further, but it would take someone with more experience with boost and intimately familiar with the project to further reduce unnecessary boost code.

[christianparpart]

The list of C++-level helper dependencies I am currently having are:

• yamlcpp (for config file parsing)
• fmtlib (best C++ formatting, and going to be part of C++20, anyways)
• range-v3 (finally a useful STL)
• Catch2 (a really convenient testing api)

all but yamlcpp are even header-only. Iff i see fit, I am not against adding a other libs, since it's all auto-fetched).

Checklist of WT OSS deps

OSS dep	WT uses	my comments
boost	general-stuff	I don't want to use boost (if at all, to a minimum). (I use Catch2 for testing)
chromium	for base/numerics from this project	not sure yet I need this
dynamic_bitset	why?	But how's that better than vector<bool> though?
libpopcnt	library for counting the number of 1-bits	that could be used for optimization techniques maybe later.
interval_tree	Used for URL detection (and other stuff?)	could I use that for this too?
xorg_apps_rgb	?	?
fmt	string formatting	I'm using that already

[WSLUser]

the xorgs_apps_rgb is just a reference showing they've adopted the rgb table from Xorg to better support colors for OSC. They try to provide all color support to ensure even legacy apps can benefit.

They use interval_tree for supporting the OSC 8 hyperlink sequence, mostly to ensure Windows Terminal can automatically detect and allow clickable URLs in the terminal window.

Dynamic_bitset has helped improve things in noticeable ways. I'd check the code for that project out and if there's room for improvement, I'm sure they'd make the necessary updates. On top of this, libpopcnt is designed to work automatically with it if defined.

Chromium /base/numerics provides the safe math needed for doing various calculations. There might be other math projects that provide just as safe or safer math than chromium but since it's MIT, that's what WT went with.

Overall, you might best look at https://github.com/microsoft/terminal/tree/main/src/til/ut_til, which are libraries created by the WT devs to make things easier for implementing or improving code health in various ways including especially performance. I believe most of these libraries utilize some if not all of these OSS projects so all the hard to read for users code can be hidden and not need to be reviewed once understood what it's doing. In essense, those libraries make code easier to read throughout the rest of the project. Not sure if you'd be able to adopt them as-is or if you even need to, but should help with understanding the project a bit more.

[christianparpart]

the xorgs_apps_rgb is just a reference showing they've adopted the rgb table from Xorg to better support colors for OSC. They try to provide all color support to ensure even legacy apps can benefit.

That is nice. I was actually looking into the xterm source code manually for that.

They use interval_tree for supporting the OSC 8 hyperlink sequence, mostly to ensure Windows Terminal can automatically detect and allow clickable URLs in the terminal window.

OSC-8 does not need that, but detecting URLs that are not embedded by OSC-8 do.

Dynamic_bitset has helped improve things in noticeable ways. I'd check the code for that project out and if there's room for improvement, I'm sure they'd make the necessary updates. On top of this, libpopcnt is designed to work automatically with it if defined.

Chromium /base/numerics provides the safe math needed for doing various calculations. There might be other math projects that provide just as safe or safer math than chromium but since it's MIT, that's what WT went with.

That's why i created the above table, so I can get an overview of what might be beneficial for Contour, too. I think I will have a look over the above deps and will consider each of them individually, given some time and space (of course) :-)

Overall, you might best look at https://github.com/microsoft/terminal/tree/main/src/til/ut_til, which are libraries created by the WT devs to make things easier for implementing or improving code health in various ways including especially performance.

Not sure if you'd be able to adopt them as-is or if you even need to, but should help with understanding the project a bit more.

Being short before my next release, I am currently especially focusing on code health (bug fixes / API cleanups for better maintainability / modularity). That will surely be helpful. Many thanks.

[data-man]

Replace Catch2 to https://github.com/boost-ext/ut (works with std 17)?
Use https://github.com/CLIUtils/CLI11.git?
Why YAML?
Maybe migrate to TOML? E.g. https://github.com/marzer/tomlplusplus.git.
Also https://github.com/robhz786/strf seems faster (I tested develop branch) than fmt.

[WSLUser]

Yeah I'm not so sure about CLI11 here. It's a good argument parser that's used for the terminal application (WT) to do things like create profiles, tabs, panes, etc. but maybe not necessary for this terminal. I had suggested they use TOML but currently everything is configured using json. TOML would be a better configuration manager. I hadn't been aware of that particular project. That UT project looks pretty awesome. I do wonder, MS Terminal supports some stuff from C++20 as they use gsl (which switched to C++20) (required updating use of span) and FMT (which will be part of the native C++ 20 library). But that project is still mostly C++17. Unfortunately looks like UT doesn't work for MSVC krzysztof-jusiak/ut@bc87813 so I hope the Windows builds for this project use gcc still.

[christianparpart]

Well, I need to be able to compile on:

• somewhat latest OS/X
• Windows 10 x64
• Linux Ubuntu 20.04 but also especially 18.04 (which is using quite some oldish toolchain, I am happy to at least have C++17 support)
• Arch Linux

At least that's what we're currently having in the CI (considering ARM64 since today, by accident)
So using some library that's not backwards compatible down to the old Ubuntu 18.04 is really hard for me to support then. :)

@WSLUser I know you're most likely more active on Windows (not sure how often you touch linux natively), bare in mind, my Windows port is like a second step brother currently because I am still lagging a proper DirectWrite implementation of the text shaping / font rasterization API (see here). I'm having a WIP, but that one is a pure research battlefield for getting to know Windows APIs. Freetype/Fontconfig/Harfbuzz does indeed compile and somehow on Windows, But I cannot do font fallback without talking to DirectWrite, so I'm stuck until the earlier mentioned interface is properly implemented for windows. (just FYI) ^^

[WSLUser]

In that case, I'd highly suggest you unashamedly steal the DirectWrite implementation Windows Terminal uses. It's gone through a number of optimizations and support for various things (There's even now a PR to get Bold finally supported there). You can even import shader files for your background if you wanted (you'll need to link to your own settings manager of course) but font fallback is included among other things. If you look through some of the PRs they've done, they reference various MS docs that describe using the APIs for DirectWrite.

[WSLUser]

I touch Linux natively but in other areas. I'm by no means a C++ or terminal expert. But I do try to pick up on things to increase my knowledge.

[data-man]

@WSLUser

Yeah I'm not so sure about CLI11 here.
but maybe not necessary for this terminal.

MS Terminal also uses it. 😃

[WSLUser]

Yes I know. If you started using it (haven't checked), then my statement can be ignored.

[christianparpart]

Actually I am using my own rolled CLI API, as the requirements I have set on how the CLI syntax should look like are not classical POSIX, but more like how iproute2 package binaries are working:

While I usually definitely prefer the POSIX style syntax here in that case I prefer the human phrase style one. (I think I've added some POSIX compatibility to it though, which is simply not printed in the help output as I didn't encourage that style, I'm open though).

[WSLUser]

I'd honestly prefer you keep POSIX style syntax. Truth is everybody is used to it, especially in non-Windows world. But even Windows has some POSIXisms to it now (purposely so due to WSL). That said, you can always offer the human phrase style as an alternative in addition to the POSIX style. Maybe people will adopt that instead and request other terminals do the same, you never know.

[data-man]

I am using my own rolled CLI API

Yeah, it's cool!
Please put it into its own repository!

[christianparpart]

I am using my own rolled CLI API

Yeah, it's cool!
Please put it into its own repository!

Will do. And then link it here

[data-man]

Use https://github.com/gulrak/filesystem instead of boost::filesystem?

[christianparpart]

Use https://github.com/gulrak/filesystem instead of boost::filesystem?

I couldn't decide yet. I'd prefer to stick with std::filesystem, which I do, and I only use boost::filesystem when not supported by the platform (older OS/X). Not sure yet.

As @uspasojevic96 posted in his "EDIT", Maybe we can now remove boost::filesystem and just use std::filesystem? We could try in a PR.

p.s.: I think it makes sense to convert this ticket into a Discussion. I hope you guys don't mind. :)

[uspasojevic96]

Keep in mind that >=10.12 ^ <11.x.x don't have std::filesystem yet so a polyfill is still needed, but I agree that boost should be completely dropped

[uspasojevic96]

Use https://github.com/gulrak/filesystem instead of boost::filesystem?

macOS 10.12/10.14/10.15 11.x.x needs to be tested also unfortunately

EDIT: 11.x.x supports std::filesystem so all should be good

[data-man]

macOS 10.12/10.14/10.15 11.x.x needs to be tested also unfortunately

It is currently tested on macOS 10.12/10.14/10.15, Windows 10, Ubuntu 18.04, CentOS 7, CentOS 8, FreeBSD 12 and Alpine ARM/ARM64 Linux but should work on other systems too, as long as you have at least a C++11 compatible compiler.

?

[uspasojevic96]

I was talking about 11.x.x (Big Sur)

[data-man]

I'm not familiar with Apple products. What has changed significantly in Big Sur API?

[uspasojevic96]

it's 64bit ARM and non ARM, a lot of things don't work on it, no OpenGL debug symbols, increased "security" by becoming even more Vista, apparently updated LLVM

[data-man]

https://github.com/BowenFu/matchit.cpp is awesome!

[christianparpart]

Matchit certainly looks amazing grill ah architectural point of view.

My concerns are:

• how easy is interactive debugging, stepping through the code
• stack trace readability
• performance implications on largeratch counts (i think complexity is O(n) i think?).

I however care a lot in code maintainability, readability and health, where i think that it might maybe help.

[data-man]

https://github.com/p-ranav/awesome-hpp/ 👍

[data-man]

https://github.com/DuffsDevice/tiny-utf8 is a nice drop-in replacement for std::string.

[christianparpart]

it certainly has some interesting member functions. conversion is handled via libunicode already though, and random access we don't need.

p.s.: Now I think we have so many library/header proposals, it'll take me some time to carefully investigate on each :-D