New LUT calibration based on mp4 test videos (part I)

[awawa-dev]

I don't know what the final effect will be, but since AI is currently very popular ;) Let's use Deep Learning Function Approximation to calibrate SDR/HDR/whatever video source. We will use Python and its related libraries here but it is an optional functionality and you do not have to use it nor will we install it: it must be present in the system.
But first, let's sort out the calibrator code because it needs cleaning before further work. We will use a new library for linear algebra that will help us.

---

Main changes (non-AI):

• Old method using Windows 10 and a web browser is no longer mandatory, you can use the old method using browser or simply play the video test file in your favorite video player.

• New algos that support even video source trans-coded to full BT2020 range and provide better quality

• Captured colors are saved in the HyperHDR directory after calibration as calibration_captured_yuv.txt and can be provided to me for analysis if I have some spare time.

• Support only for YUV/NV12/MJPEG modes. RGB mode is not supported anymore.

• Confirmed support for webos calibration with backend that supports NV12 flatbuffers communication. Each webos mode (SDR/HDR/DV) may require its own calibration.

• Grabber benchmark supports flatbuffers source

• Video source noise detection

• If calibrating using Windows (using the web browser or the video player as a video source) please disable features like 'Night mode' or 'Auto-HDR'. Do not change the color balance in your graphic driver.

---

Video test files:
https://github.com/awawa-dev/awawa-dev.github.io/tree/master/calibration
Use SDR video only if your OS or video player will automatically apply SDR to HDR tone mapping. Or if you are calibrating eg. webos.

The web browser method and playing yuv444 file usually provides the best calibration quality but yuv444 is supported probably only by the PC players. yuv420 has better compatibility but worse quality (the risk the player is starting using its own filters like dithering).

---

Presenting capabilities of new tone mapping algos:

Source desktop (Windows 11 HDR enabled):

Captured by MS2130 (F3 firmware !!!) without tone mapping:

Tone mapping enabled:

[s1mptom]

By the way Yolofant said the right thing - once you add in image blending from the ui layer things become a lot slower
and since I have this part disabled when sending NV12 it is probably due to this that performance increases =) so you should check in the same conditions - with UI capture disabled

check === compare with old method i mean =)

[awawa-dev]

Always tested here with GUI disabled including comparing NV12 mode to RGB mode performance. I agree that mixing RGB with YUV converted to RGB when GUI is enabled incurs additional costs and BTW disables NV12 mode so the calibration wouldn't work anyway so it could be easily detected.

EDIT: If I understand correctly when GUI capturing is enabled, the correct RGB from the GUI is mixed with material that requires LUT correction (YUV from HDMI input that must be converted internally to RGB before mixing)? But it will end badly when it comes to quality...The proper way: libdile/vtcapture provide capturing => ignore GUI, in the other case switch to GUI mode, if GUI mode colors are correct then disable tone mapping. Do not mixup both modes.

[satgit62]

The proper way: libdile/vtcapture provide capturing => ignore GUI, in the other case switch to GUI mode, if GUI mode colors are correct then disable tone mapping. Do not mixup both modes.

In my opinion, this makes sense and is the more elegant solution.

[s1mptom]

for me it's around 62-64ms with linux socket on 640x360 and 70-80ms on 720p
but for some reason i'm getting an error on the end =) Cannot read properties of undefined (reading 'device')

[awawa-dev]

Probably it's caused by the missing USB grabber module in the build. It seems even the grabber is not used the report tries to read the module properties searching for selected device and capturing resolution. Will take a look later.

[awawa-dev]

1 @s1mptom I think I fixed it in the latest commit
2 @satgit62 it turned out that the HTML color named green is 128 not 255 so you should have a perfect calibrated LUT because the room for error is small. I fixed it and assigned to fixed 255 not using names. Try now.
3 Added new 'NV12 quarter of frame mode' option to flatbuffers settings. So now you can safely reduce the excess brightness information in NV12 without losing color by reducing the size in HyperHDR which should relieve some CPU load and reduce latency especially with larger frames.

[satgit62]

New webOS org.webosbrew.hyperhdr.loader
Add Quarter of frame mode for NV12 flatbuffers from @awawa-dev

https://github.com/satgit62/hyperhdr-webos-loader/releases/download/latest/org.webosbrew.hyperhdr.loader_0.1.9_all.ipk

[awawa-dev]

Added YUV limited test files to test the impact of this setting (usually used in most of video files) on the calibration:
https://github.com/awawa-dev/awawa-dev.github.io/tree/master/calibration

[s1mptom]

interesting thing - with a limited range test file i'm getting BT601 and with full range i'm getting FCC. And result still more contrasty than original picture =( looks like there's no difference at all so maybe i'm doing something wrong. and checked few times that i'm not using full range

…

On Wed, Sep 25, 2024 at 10:07 PM Awawa ***@***.***> wrote: Added YUV limited test files to test the impact of this setting (usually used in video files) on the calibration: https://github.com/awawa-dev/awawa-dev.github.io/tree/master/calibration — Reply to this email directly, view it on GitHub <#896 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOKOHMIHR7M3UQY4JA23FLZYMQ2ZAVCNFSM6AAAAABKPQAGCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZVGI3DCMRZGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[awawa-dev]

As I told you coef are just starting reference, while searching kR/kB delta can change them significantly. And the most important: BT601 and FCC are almost identical. Link: http://avisynth.nl/index.php/Color_conversions

[awawa-dev]

Added few improvements, rather more significantly for USB grabbers:

• signal noise detection, also capture few samples (4) per test board
• removed FCC coef testing (BT601 is almost the same)
• fixed result presentation error caused by rounding
• minor improvements in the fine-tune algo
• more detailed result saved in calibration_captured_yuv.txt file (new format)
• an option to turn off post-processing (you probably don't want to do this) and rely solely on ITU equations

Since there are a lot of changes here that block other new HyperHDR functionalities that I would like to add, I plan to move the AI part ​​to another pull-request. Here I want to work on a better scoring function, because RGB distance is a fast algorithm, but it is not very well suited to the situation.

[awawa-dev]

The last commit brings an important fix (corrected saving results) and a new denoising algorithm that significantly improves calibration results using the YUV420 format, especially YUV420 LIMITED. Based on these tests, I am considering completely removing the YUV420 FULL RANGE test files since LIMITED works just as well and is somewhat standard. I don't plan on adding any more new features in this PR unless I find some bugs.
Using different players or text files may slightly affect the calibration result and color reproduction, so it is better to test different combinations if something does not suit us.

---

WIndows 11 HDR 1080p 12-bits RGB, MS2130 (F3 firmware, 1080p YUV)

• Firefox browser = 322

• MPC-HC (clsid2) + madvr

  • yuv420_mpc_madvr = 651
  • yuv420_limited_mpc_madvr = 339
  • yuv444_mpc_madvr = 320

• VLC player

  • yuv420_vlc = 629
  • yuv420_limited_vlc = 614
  • yuv444_vlc = 443

---

WEBOS 1080p5, internal player

• SDR yuv420 = 257
• SDR yuv420 limited = 190
• HDR ~ 2000 💩

---

WEBOS 1080p5, MPC-HC/MADVR + Win10 1080p

• HDR yuv444 = 218
• HDR yuv420 = 227
• HDR yuv420 limited = 238

• SDR yuv444 = 724
• SDR yuv420 = 697
• SDR yuv420 limited = 717

---

WEBOS 1080p5, VLC player + Win10 1080p

• HDR yuv444 ~ 5000 💩
• HDR yuv420 ~ 3700 💩
• HDR yuv420 limited = 349

• SDR yuv444 = 484
• SDR yuv420 = 839
• SDR yuv420 limited = 383

---

results.zip

[satgit62]

@awawa-dev
Thanks for the changes and for fixed bugs.
Made the changes and created new org.webosbrew.hyperhdr.loader for LG webOS.
Here is the link:https://github.com/satgit62/hyperhdr-webos-loader/releases/download/latest/org.webosbrew.hyperhdr.loader_0.1.9_all.ipk

[awawa-dev]

mpc-hc + madvr SDR results look weird so after replacing the madvr with built-in enhanced video renderer custom presenter and the results for SDR 420 limited improved from 717 to 262... guess madvr is really heavy processing SDR even without any up/downscaling and it's not the best solution for me - lesson learnt.

calibration_captured_yuv_sdr_webos_420_limited_mpc_enhanced_video_renderer_cs.txt

[awawa-dev]

Comparison of the new calibration capabilities from this PR compared to the previous one .. so of course also to the old LUTs which can still be downloaded from the application. On Windows with HDR enabled, calibration using mpc-hc + madvr and Elgato HD60X and HDR_420_limited test file gave a perfect result of 195 (using SDR_420_limited in such a configuration with Elgato failed).

Test Subject: my favorite Fairy Tail ;-)

---

Org:

New:

Old:

---

Org:

New:

Old:

---

Org:

New:

Old:

---

Org:

New:

Old:

---

calibration_captured_yuv_hdr.txt

[satgit62]

@awawa-dev
Hello,
when I try to copy the three LUT's (lut_lin_tables.3d, lut_lin_tables_dv.3d and lut_lin_tables_hdr.3d) on my old OLED65B7D LG to /home/root/.hyperhdr/, it does not work. Only one LUT with a full 150 MB is copied. The second one is also copied, but it stops halfway through.
Only the LEDs where the corresponding LUT is found light up.

Is it the same with your C9?

I then copied lut_lin_tables_dv.3d to /home/root/.hyperhdr/ and the lut_lin_tables.3d, lut_lin_tables_hdr.3d to /media/developer/apps/usr/palm/services/org.webosbrew.hyperhdr.loader.service/hyperhdr/
The LUT's are found, and The LEDs then light up for each video material.

What is the best way to do this?
By swapping the .hyperhdr folder to a USB stick works, but then I can no longer save any settings. It's because of the hyperhdr.db, which is located in /home/root/.hyperhdr/db/.

[awawa-dev]

@satgit62
I think @s1mptom had a similar problem in the past... Are you sure you have that much free space on the node that is mapped to home? Use df -h. BTW with tone mapping enabled (it always has to be enabled even for SDR but on webos it's not a problem) and NV12 source these LUTs are redundant and take up a lot of space... only the middle 50MB of each will be used and another 100MB will be wasted. Maybe something will be done about it someday.

[satgit62]

@satgit62 I think @s1mptom had a similar problem in the past... Are you sure you have that much free space on the node that is mapped to home? Use df -h. BTW with tone mapping enabled (it always has to be enabled even for SDR but on webos it's not a problem) and NV12 source these LUTs are redundant and take up a lot of space... only the middle 50MB of each will be used and another 100MB will be wasted. Maybe something will be done about it someday.

Yes, indeed, with this device the free space on /home/root is very limited. I will try to create free space for one more LUT.

[satgit62]

@awawa-dev
By cleaning up the /home/root/ directory, I managed to place the required LUTs in /home/root/.hyperhdr/.
As a result, Ambilight works as it should on the old B7.
Thanks again for all the changes to HyperHDR that made this possible.
@s1mptom, thank you for your modified hyperion-webos.

[s1mptom]

@satgit62 I think @s1mptom had a similar problem in the past...

actually not =) looks like i have a lot of free space on G3...) but previously i'm already thinking about it (3 parts in LUT and 3 separate files with only 1 usefull part) but... it's not my 1st priority right now..)) but in near future we could handle this too ofc (i mean there will definitely need to implement support on the HyperHDR side and on the WebOS backend side as well)

[satgit62]

@satgit62 I think @s1mptom had a similar problem in the past...

actually not =) looks like i have a lot of free space on G3...) but previously i'm already thinking about it (3 parts in LUT and 3 separate files with only 1 usefull part) but... it's not my 1st priority right now..)) but in near future we could handle this too ofc (i mean there will definitely need to implement support on the HyperHDR side and on the WebOS backend side as well)

Hello,
I find the three different LUTs useful in this case, and I am very satisfied with the results so far, and it works as it should.
For this reason, I am in the process of writing a guide for LG webOS users so that even non-experienced users can run it.
If you find anything in this guide that is incorrect or have any suggestions for improvement, please let me know, so I can change them.
Thank you.
https://github.com/satgit62/Ultimate-HyperHDR-Ambilight-fine-tuning-experience-for-LG-webOS-with-new-LUT-calibration-

[awawa-dev]

@satgit62 good work 👍 I will have more time during the weekend, I am preparing another cherry on top of the cake for the current calibrator... but that will be in a few days.

[satgit62]

@satgit62 good work 👍 I will have more time during the weekend, I am preparing another cherry on top of the cake for the current calibrator... but that will be in a few days.

Hello,
Thank you, I'm already looking forward to what's to come. I won't miss out on the cherry. 😍