More Challenging Alignments?

[ttbek]

I'm trying to align some sets of images here that were taken freehand and the results I'm getting are quite bad. On the one hand, obviously using a tripod would improve things. On the other hand, I thought the results would be better than what I'm seeing. I poked around just a tiny bit in the code and see that you use only the Affine transformation. How many other things need to be changed in order to use a Homography tranformation instead? I tried briefly just changing that call and now have the output:

focus-stack stack_test_2/*.JPG --output=./stack2.jpg --threads=12 --no-whitebalance --no-contrast --denoise=0 --verbose

OpenCL device: NVIDIA Corporation GeForce GTX 1080 Ti OpenCL 1.2 CUDA
0.332 [ 1/ 1] T3 Starting task: Load stack_test_2/9M6A0488.JPG
0.332 [ 2/ 9] T6 Starting task: Load stack_test_2/9M6A0487.JPG
0.332 [ 3/ 21] T2 Starting task: Load stack_test_2/9M6A0489.JPG
0.332 [ 4/ 21] T11 Starting task: Load stack_test_2/9M6A0490.JPG
0.332 [ 5/ 25] T0 Starting task: Load stack_test_2/9M6A0486.JPG
0.332 [ 6/ 28] T5 Starting task: Load stack_test_2/9M6A0485.JPG
0.333 [ 7/ 69] T9 Starting task: Load stack_test_2/9M6A0491.JPG
0.333 [ 8/ 69] T10 Starting task: Load stack_test_2/9M6A0492.JPG
0.333 [ 9/ 69] T7 Starting task: Load stack_test_2/9M6A0483.JPG
0.334 [ 10/ 69] T8 Starting task: Load stack_test_2/9M6A0484.JPG
0.334 [ 11/ 76] T1 Starting task: Load stack_test_2/9M6A0493.JPG
0.334 [ 12/ 80] T4 Starting task: Load stack_test_2/9M6A0482.JPG
0.938 T7 Finished task 9 in 0.605 s.
0.956 [ 13/110] T7 Starting task: Load stack_test_2/9M6A0494.JPG
1.019 T5 Finished task 6 in 0.687 s.
1.019 [ 14/110] T5 Starting task: Load stack_test_2/9M6A0481.JPG
1.087 T4 Finished task 12 in 0.753 s.
1.088 [ 15/110] T4 Starting task: Load stack_test_2/9M6A0495.JPG
1.128 T11 Finished task 4 in 0.796 s.
1.128 [ 16/110] T11 Starting task: Load stack_test_2/9M6A0480.JPG
1.137 T9 Finished task 7 in 0.804 s.
1.137 [ 17/110] T9 Starting task: Load stack_test_2/9M6A0496.JPG
1.151 T1 Finished task 11 in 0.817 s.
1.152 [ 18/110] T1 Starting task: Load stack_test_2/9M6A0479.JPG
1.161 T8 Finished task 10 in 0.828 s.
1.161 [ 19/110] T8 Starting task: Load stack_test_2/9M6A0497.JPG
1.227 T6 Finished task 2 in 0.895 s.
1.227 [ 20/110] T6 Starting task: Load stack_test_2/9M6A0478.JPG
1.246 T3 Finished task 1 in 0.914 s.
1.246 [ 21/110] T3 Starting task: Grayscale stack_test_2/9M6A0488.JPG
Using grayscale weights R:0.304, G:0.335, B:0.361
1.256 T2 Finished task 3 in 0.924 s.
1.271 T10 Finished task 8 in 0.938 s.
1.287 T0 Finished task 5 in 0.955 s.
1.632 T3 Finished task 21 in 0.386 s.
1.632 [ 22/110] T3 Starting task: Align 9M6A0488.JPG to 9M6A0488.JPG
1.632 T3 Finished task 22 in 0.000 s.
1.632 [ 23/110] T10 Starting task: Grayscale aligned_9M6A0488.JPG
1.634 [ 24/110] T2 Starting task: Grayscale stack_test_2/9M6A0487.JPG
1.634 [ 25/110] T3 Starting task: Grayscale stack_test_2/9M6A0489.JPG
1.634 [ 26/110] T0 Starting task: Grayscale stack_test_2/9M6A0486.JPG
1.948 T7 Finished task 13 in 0.992 s.
1.949 [ 27/110] T7 Starting task: Grayscale stack_test_2/9M6A0490.JPG
1.997 T11 Finished task 16 in 0.869 s.
1.997 [ 28/110] T11 Starting task: Grayscale stack_test_2/9M6A0485.JPG
2.030 T8 Finished task 19 in 0.869 s.
2.030 [ 29/110] T8 Starting task: Grayscale stack_test_2/9M6A0491.JPG
2.130 T1 Finished task 18 in 0.978 s.
2.130 [ 30/110] T1 Starting task: Grayscale stack_test_2/9M6A0484.JPG
2.192 T9 Finished task 17 in 1.055 s.
2.192 [ 31/110] T9 Starting task: Grayscale stack_test_2/9M6A0492.JPG
2.201 T4 Finished task 15 in 1.113 s.
2.201 [ 32/110] T4 Starting task: Grayscale stack_test_2/9M6A0483.JPG
2.244 T5 Finished task 14 in 1.225 s.
2.244 [ 33/110] T5 Starting task: Grayscale stack_test_2/9M6A0493.JPG
2.248 T3 Finished task 25 in 0.614 s.
2.248 [ 34/110] T3 Starting task: Align 9M6A0489.JPG to 9M6A0488.JPG
2.315 T6 Finished task 20 in 1.088 s.
2.315 [ 35/110] T6 Starting task: Grayscale stack_test_2/9M6A0482.JPG
2.374 T10 Finished task 23 in 0.742 s.
2.374 [ 36/110] T10 Starting task: Forward-wavelet aligned_9M6A0488.JPG
2.395 T2 Finished task 24 in 0.761 s.
2.395 [ 37/110] T2 Starting task: Align 9M6A0487.JPG to 9M6A0488.JPG
2.409 T0 Finished task 26 in 0.775 s.
2.409 [ 38/110] T0 Starting task: Align 9M6A0486.JPG to 9M6A0487.JPG
Task Align 9M6A0489.JPG to 9M6A0488.JPG on thread 3 failed with exception:
OpenCV(4.2.0) ../modules/video/src/ecc.cpp:388: error: (-215:Assertion failed) map.rows ==3 in function 'findTransformECC'

Task Align 9M6A0486.JPG to 9M6A0487.JPG on thread 0 failed with exception:
OpenCV(4.2.0) ../modules/video/src/ecc.cpp:388: error: (-215:Assertion failed) map.rows ==3 in function 'findTransformECC'

Task Align 9M6A0487.JPG to 9M6A0488.JPG on thread 2 failed with exception:
OpenCV(4.2.0) ../modules/video/src/ecc.cpp:388: error: (-215:Assertion failed) map.rows ==3 in function 'findTransformECC'

3.041 T8 Finished task 29 in 1.011 s.
3.060 T1 Finished task 30 in 0.930 s.
3.080 T7 Finished task 27 in 1.131 s.
3.096 T11 Finished task 28 in 1.099 s.
3.252 T9 Finished task 31 in 1.060 s.
3.261 T6 Finished task 35 in 0.946 s.
3.304 T4 Finished task 32 in 1.103 s.
3.320 T5 Finished task 33 in 1.076 s.
3.781 T10 Finished task 36 in 1.407 s.

Error exit due to failed steps

I don't have experience with OpenCV, are they just so poorly aligned that it is failing to find a Homography transform, or is that change breaking something else, maybe finding Homography will return a larger matrix?

Well, I am attaching two result examples. The first had the images taken freehand. For the 2nd, the camera was rested, but that one might be having some issues due to banding from the lighting (it was really bad even though you don't really see it in the output, like a quarter of each image was amber). Had to reduce the size of the second image to make it small enough to upload... but I promise what you're seeing is not jpeg artifacting even though it is at only the 80 quality setting.

I'll shoot with a tripod if that is really what it takes to get decent stacking results. I understand that alignment and focus stacking are hard problems, but I know I've seen a lot of better results than what I'm getting.

[PetteriAimonen]

Regarding the errors you get with homography transform:

OpenCV(4.2.0) ../modules/video/src/ecc.cpp:388: error: (-215:Assertion failed) map.rows ==3 in function 'findTransformECC'

This is because on line 43 in task_align.cc the m_transformation is initialized to 2x3 matrix, while homography transform uses 3x3 matrix.

By a quick look, this would be the list of things to change to try it out, all in task_align.cc:

1. Change line 43 to 3x3 matrix
2. Initialize entry 2,2 to 1.0f also on line 46
3. On line 119 the resize, init of 2,2 and subregion is no longer needed because it is already 3x3.
4. Prints on line 148 should be updated.
5. On lines 267 and 284 the scaling of the transform found with downscaled image may need updating, but I haven't checked the math.
6. On line 406 the warpAffine has to be changed.

If you try it out and it works well, I would be happy to accept a pull request. The initial code used homography transform simply because I was working on microscope images.

It may also be that in your case some of the images may be failing to align while others do. Using --save-steps and browsing through the aligned images is easy way to check how well they are aligned.