Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement][Core] New approaches for Page/Template Alignment #196

Open
Udayraj123 opened this issue Jul 12, 2024 · 2 comments
Open

[Enhancement][Core] New approaches for Page/Template Alignment #196

Udayraj123 opened this issue Jul 12, 2024 · 2 comments
Labels
advanced enhancement New feature or request Hard

Comments

@Udayraj123
Copy link
Owner

Udayraj123 commented Jul 12, 2024

The template alignment problem is one of the central problems which currently remains unsolved for a general OMR sheet. This issue tracks various issues that we have noticed over a period of time in various samples.

Shifts during scanning

Despite fixing page curvature, we may still face the template alignment problem. Sometimes a scanner may malfunction at minute levels, we have seen uneven stretching of the page when using a basic scanner. The boxes need to be close to the bubbles otherwise the output is not reliable for any OMR checking software.
Example:
image

Curvature during bulk scanning

Sometimes the scan itself seems to be curved. we can see that the instructions text is not in a straight horizontal line.
Example:
image

The scans need to be straightened before applying template detection.

PDF-printed vs Photocopied misalignment

We have also seen shifts upto 10 pixels between scans of pdf-printed vs photocopied sheets. Link to an explainer: Google Drive

Describe the solution you'd like
Rough notes (ref):

- **Get all control-destination points **
    - From SIFT: 
        - Filter list: Slide a 10px(windowSize) window on the image and pick the point with max score
        - Filter list: pick the points with reasonable shifts(maxDisplacement)
    - This way we can scale to any set of feature detectors like ORB or custom detectors from the ScanZones. Just need a set of control-destination pairs.

- **Align the points **
    - Approach 1: 
        - Loop over each field, find 4 nearest anchor points and
        - apply the transform on the field box using findHomography
    - Approach 2: 
        - Triangulate the whole image based on the anchor points and 
        - perform affine transform on each triangle(all pixels get transformed)
image image

Describe alternatives you've considered
There are many other approaches that can be tried:

Additional context
NA
Link to Discord Thread: New Approaches for Page/Template Alignment

@Udayraj123 Udayraj123 added enhancement New feature or request advanced Hard labels Jul 12, 2024
@Udayraj123 Udayraj123 changed the title [Enhancement] New approaches for Page/Template Alignment [Enhancement][Core] New approaches for Page/Template Alignment Jul 12, 2024
@akashrajk54
Copy link

Hi @Udayraj123,

I’m interested in contributing to this issue and exploring solutions for improving Page/Template alignment. Based on my understanding, document scanning apps like CamScanner tackle similar alignment problems using a combination of Edge Detection, Perspective Transformation, and Image Warping techniques.

Here’s my proposed approach:

Detect Key Points & Borders: Use Canny Edge Detection + Contour Detection to find the sheet’s boundaries. Hough Line Transform can help ensure straight edges.
Correct Perspective Distortion: Apply cv2.getPerspectiveTransform to warp the scanned sheet to match the template. If curvature is present, Thin Plate Spline Warping could help.
Align the Template: Use Feature Matching (SIFT/ORB) to find control points, then apply Affine Transformation or Homography to adjust shifts.
Alternative Approaches: Testing cv2.findTransformECC or Optical Flow-based corrections for robustness.
Would love to get your feedback on this approach! Let me know if there’s a preferred direction or any specific constraints I should consider.

Looking forward to contributing! 🚀

@Udayraj123
Copy link
Owner Author

Hi @akashrajk54, glad to see someone showing an interest in having a go at this challenging problem.

You can start with applying Feature matching to get a basic alignment solution, for interpolating data you can explore different popular methods.

I have been following and attempting some document de-warping techniques to solve this, especially methods that make use of the layout of the page elements which is fixed per OMR Template. I have some starter code in the dev branch for this.
You can join the Discord server and discuss more with me over there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
advanced enhancement New feature or request Hard
Projects
None yet
Development

No branches or pull requests

2 participants