What is the best way to get this library to work with perspective image cropper

[rosentoshev]

Hi,

I am trying to do perspective image cropping once an image is captured. The rectangle coordinates would be used as the overlay. I have tried with the react-native-perspective-image-cropper but cannot make the crop actually work. Is there another option for perspective image cropping that can be recommended or a working example of react-native-rectangle-scanner and ``react-native-perspective-image-cropper`? Thank you.

[chunkydonut21]

Did you find any solution @rosenexpend . I also want to know this.

[humphreyja]

I'm not 100% familiar with react-native-perspective-image-cropper but assuming you can pass it an image and a set of coordinates, here's how I'd do it.

Use the latest detected coordinates from your state (in the complete example for this project, this is this.state.detectedRectangle). Then use the original image from onPictureProcessed({ initialImage }) and pass that to the image cropper.

You will most likely need to convert the detectedRectangle coordinates to match the output image. This is because the output image will most likely be a higher resolution (and in the case of androids, can often be a different aspect ratio). The detectedRectangle object also contains the dimensions of the camera preview it is scanning which is what you can use for this.

If the aspect ratios are different, you will need to map them to a new aspect ratio. Basically you need to push out either the width or the height to match while also adjusting the coordinates to match. (I think you can simply just add the change in width to the X coordinates for example).

Note: I've considered extracting my cropping/transform code in this package into a callable function from JS so that you could run the crop from javascript yourself. This would allow devs to build a perspective crop tool themselves, or use a package like the one mentioned to manually adjust the coordinates before cropping. It will take a bit of work to get that working, so I haven't given it the time to build it.

[chunkydonut21]

@humphreyja Thanks! Got it to work with react-native-perspective-image-cropper by passing coordinates and dimension to that library and applying it to initialImage. But their cropper doesn't work correctly. Can you tell what function do you call to crop the image with given coordinates in react-native-rectangle-scanner?

[humphreyja]

When you say it doesn't work, does it not crop the image correctly? Is the image rotated, or is the crop not the right size, etc.

For this package:
iOS: https://github.com/HarvestProfit/react-native-rectangle-scanner/blob/master/ios/RectangleDetectionController.m#L161
Android: https://github.com/HarvestProfit/react-native-rectangle-scanner/blob/master/android/src/main/java/com/rectanglescanner/helpers/ImageProcessor.java#L236-L268

For the perspective image cropper:
iOS: https://github.com/Michaelvilleneuve/react-native-perspective-image-cropper/blob/master/ios/CustomCropManager.mm#L33
Android: https://github.com/Michaelvilleneuve/react-native-perspective-image-cropper/blob/master/android/src/main/java/fr/michaelvilleneuve/customcrop/RNCustomCropModule.java#L59-L108

Just from a quick scan of what they are doing, their iOS code is using an older API (actually one of the reasons why I rebuilt this package). It is missing a parameter that I believe helps scale the coordinates and the image to the same scale. And for their Android stuff, it looks similar to mine (they have this weird scaling thing they do which makes absolutely no sense 🤦‍♂️, but the math works out the the same). I suspect what could be happening is that the coordinates came from an image of a different aspect ratio than the final image being passed in.

I use this function first to match the aspect ratios, and then this function to scale the coordinates to match the image being cropped. If I were to expose this logic, I would also include those functions as the output photo is almost always much larger than the preview photo.

I hope this helps. I might spend 5 mins today to create an experimental pull request that exposes this code as javascript functions so you can give that a try.

[humphreyja]

Sorry as I'm digging into this, I've changed my perspective on that code. So actually this package's iOS code and the perspective package's iOS code are essentially the same for cropping the image. Where I could see some big problems with that package is in the returning of the image. After cropping the image it returns that image in a callback as the base64 encoded version of it. That's terrible. Honestly. Basically that means that it could be sending a 200mb image over the javascript bridge. I never bothered with adding a return image type of base64 to this package because I noticed that it makes my phone freeze and crash. It probably only works with the lowest quality jpeg image.

I'm throwing together a PR that instead stores the image in the cache folder and instead returns the directory of that changed image.

[chunkydonut21]

@humphreyja On iOS, the cropper of this react-native-perspective-image-cropper library sometimes give corrupt base64 or otherwise low resolution image with incorrect crop. Although its detecting coordinates correctly. Can you tell how do I manually call crop function with given coordinates and an image using react-native-rectangle-scanner from JS code? Much appreciated.

[humphreyja]

@chunkydonut21 sorry, I spent a few minutes on this the other day and didn't come up with something usable for you unfortunately. I think personally what I would do is actually just fork that project and simply replace the base64 return portion of that package with the file saving approach of this package.

For Android, here is the function that is called with the final image(s) and the steps I take to save them to storage before returning the URI to that saved image. https://github.com/HarvestProfit/react-native-rectangle-scanner/blob/master/android/src/main/java/com/rectanglescanner/views/RNRectangleScannerView.java#L119-L163

Here is the same for iOS: https://github.com/HarvestProfit/react-native-rectangle-scanner/blob/master/ios/RNRectangleScannerView.m#L95-L157

Like I said, everything should work the same, it's just replacing how you interact with the image. If you get something solid working, be sure to share it as this is definitely something that people would like!

[rosentoshev]

@chunkydonut21 Yes, I made a working prototype version though it is with a local version of the react-native-perspective-image-cropper that I have, which is based on several people fixing issues with Android compatibility and the cropper cropping inaccurately. The functioning prototype, without my local version react-native-perspective-image-cropper can be found here: https://github.com/rosenexpend/rectangle-scanner-example