Improve performance of VideoStreamFrame.on_frame

[Baharis]

Context

In the light of our discussion in #123 , before attempting to introduce some convenience features that would slow the GUI down, I sat down to look for some ways to make the on_frame method faster (to have more "room" to slow things down later). I discovered several issues that impacted the improvement:

• The intensity of input image was scaled down to 256 rather than 255 – the actual upper limit for ImageTk;
• After scaling, the image was not clipped to (0, 255), so values over 255 were being passed to PIL.Image.fromarray;
• The array provided to PIL.Image.fromarray was of dtype=np.float32 and required conversion by PIL.
• The linear brightness correction performed by PIL was very expensive and could be sped up by doing it manually in numpy.

I produced several versions of the on_frame function with alternative algorithms. Then, using a custom benchmark class, I timed all the methods. I did not use the built-in fps nor interval displays because IMO they are more influenced by inaccuracies in the system clock i.e. self.after(self.frame_delay, self.on_frame) can wait longer than self.frame_delay on threading / system whims. The tables below document selected timings collected on two PCs for various scenarios on the main and this branch. The mean on_frame times come from the frames # 100–600 displayed in the GUI and have an uncertainty of ~150ms.

Branch "main" algorithm timings:

Scenario	Mean on_frame time
Default settings (auto-contrast, brightness=1)	5406 µs
No auto-contrast	5343 µs
No scaling (dynamic_range=11800)	2718 µs
Brightness = 0.5	5968 µs

Then I probed some alternative algorithms on the "main" branch:

Scenario	Mean on_frame time
Decide contrast using np.partition rather than np.percentile	5343 µs
Multiplying by brightness = 0.5 in numpy rather than PIL	5531 µs
Convert data to np.int16, then clip to (0,255) before PIL	4968 µs
Convert data to np.int16, then clip to (0,255), then convert to np.uint8 before PIL	3375 µs

This way I found two major improvements. First and foremost, converting and clipping data manually in numpy instead of leaving it to PIL cuts the on_frame time on average by ~2 ms i.e. ~40%. Interestingly converting to int16, then clipping integers, then converting to uint8 seems the most performant (though it does use the most memory which could be an issue if not the 50 ms to clear RAM in-between frames). Secondly, PIL Image Enhance introduced brightness by linearly scaling the intensity, which we can just as well do in numpy at basically no cost. By implementing these two changes, we get:

Scenario	Mean on_frame time
Default settings (auto-contrast, brightness=1)	3343 µs
No auto-contrast	3000 µs
No scaling (dynamic_range=11800)	2468 µs
Brightness = 0.5	3406 µs

I also confirmed this works on my slower microscope PC:

Branch	Scenario	Mean on_frame time
main	Default settings (auto-contrast, brightness=1)	9125 µs
main	No auto-contrast	9531 µs
main	Brightness = 0.5	9625 µs
this	Default settings (auto-contrast, brightness=1)	5750 µs
this	No auto-contrast	5625 µs
this	Brightness = 0.5	5937 µs

Some final notes: replacing PIL's enhancement with numpy scaling has some advantages and drawbacks. It does behave a bit differently: previously at brightness = 0.5 no pixel could be lighter than #808080. Now, the lightest 0.5% of pixels might be still white, which in my opinion is an improvement: brightness can be now used to easily find these lightest spots even if auto-contrast is ON while still keeping the image "dark". Moreover, in this PR I fixed a bug where by setting the dynamic range to the default value, the image would be scaled to fixed value of 256 instead of the dynamic range setting itself. This was the reason why the simulated image was displayed fine at default of 11800 (now changed to 255), but was completely dark at 11799.

Minor changes

• Improved the performance of on_frame responsible by image display by ~40%

Bugfixes

• Fixed an issue where the VideoStreamFrame would scale the display range strictly to 256 whenever it was set to camera.dynamic_range (by default 11800, this is also set as the max value).