Pdf to image memory leak

[kbrajwani]

Hi i am using the library to convert the pdf pages into images. Where i am getting too much ram consumption problem.
You can see below image where in start it's using 514 mb ram where after pdf to image conversion where it will use 2.8 gb. i can't able to share document but it contains 69 pages.

Code example

import fitz      
from PIL import Image
pdf_document = fitz.open(filename)
images = []      
for page_index in range(pdf_document.pageCount):  
    page_pixmap = pdf_document.getPagePixmap(page_index,matrix=fitz.Matrix(300/72,300/72),alpha=False)
    images.append(Image.frombytes("RGB", [page_pixmap.width, page_pixmap.height], page_pixmap.samples))    
pdf_document.close()

output of print(sys.version, "\n", sys.platform, "\n", fitz.__doc__)
3.7.10 (default, May 3 2021, 02:48:31)
[GCC 7.5.0]
linux

PyMuPDF 1.18.14: Python bindings for the MuPDF 1.18.0 library.
Version date: 2021-06-01 08:11:38.
Built for Python 3.7 on linux (64-bit).

[JorjMcKie]

Have you ever computed the size of one pixmap?
Let's say for one A4 page, 595 x 842, in RGB (3 bytes per pixel) you will get 595 * 842 * 3 * (300 / 72) * (300 / 72) = about 25 MB.
For 70 pages, this will amount to 1.7 GB.
So where is the bug?

[kbrajwani]

i was not aware of that my bad but let's say if i have to remove cache how can i remove that. i have tried to delete images object but it's doesn't make any changes.

del images
import gc
gc.collect()

[JorjMcKie]

• Why do you store images in raw PIL image format?
• Why not as PNG or JPG bytes objects?
• What are those enormous zooming factors for? They have a quadratic effect on sizes! Using 300/72 as zoom factor makes a pixmap more than 16 times larger than standard resolution.
• Memory will never be freed if it is still referenced by something else - which is the case with the PIL images: they point to those pix.samples presumably.
• Why store so many, so large images in memory at all? Why not temporarily save them on disk for later use?

[kbrajwani]

I am not aware about the raw pil format.
I don't mind png or jpg bytes object can you tell me how to do in python?
The zooming effects are for the ocr process. i have need of this resolutions to process image further.
i am storing image directly into memory because if i will save that image and load again it will take 10 seconds for 70 images to store and retrive.

You can see here. I want to process image as quick as possible.

If i am using any wrong way please make me correct so i can reduce the ram usage.

[JorjMcKie]

Converted the issue to a Discussion, because it is no issue in PyMuPDF at all.

I am not aware about the raw pil format.

But you are using it: Image.frombytes("RGB", [page_pixmap.width, page_pixmap.height], page_pixmap.samples)!

I want to process image as quick as possible.

What is your OCR software? Can it read PIL images directly from a Python list? Hard to believe.

png = pix.tobytes() is a bytes object containing the image in PNG format. Similar method: jpg = pix.pil_tobytes("JPEG") which produces a JPG format bytes string.
Both should be about 0.7 MB size - even for your resolultion. Quite a reduction from 25 MB pixmap size, isn't it?

Why don't you use ocrmypdf? It does all that for you. Give it a PDF plus some parameters e.g. for improving text recognition, and it will output a PDF with a new layer of OCR-ed text.

[kbrajwani]

page_pixmap = pdf_document.getPagePixmap(page_index,matrix=fitz.Matrix(300/72,300/72),alpha=False,colorspace = "gray")
data = page_pixmap.pil_tobytes("JPEG")
_img = Image.open(io.BytesIO(data))     

You are telling me to use this right?
I have already tried also seen this issue #322 .
Here main issue is it taking 3-4X times in compare to page_pixmap.samples .

Why i am doing this process?
The pipeline is like take pdf convert to images.
Identify the images which we have to ocr. So i am not doing ocr of all 70 pages.
So in between ocr and pdf there is one classifier.

[JorjMcKie]

Well then it looks like you have to bargain processing speed against memory consumption.
You have to swallow one of the toads at hand.