First version to support MuPDF v1.19.*

[JorjMcKie]

Introduces major new features like PDF journalling and OCR support by directly invoking Tesseract-OCR.
In addition, it is possible to detect whether object are covered (hidden) by other objects.

As part of the new version, the following issues have resolved:
#1313, #1311, #1290, #1286, #1287, #1284.

---

This discussion was created from the release First version to support MuPDF v1.19.*.

[victor-ab]

Hi! Thanks for the release!

Can you please add a bit more detail on how the Tesseract works with PyMuPDF?
Are you also storing the char locations yielded from Tesseract?
Tesseract will detect text only on images? What if there are two overlapping images?

[JorjMcKie]

Hi,
sure. This is documented in a "Reference Manual" format for the new methods Pixmap.ocr_save() and Pixmap.ocr_tobytes(). There also is the a new method Page.get_textpage_ocr().

OCR-ing a pixmap is probably simpler to start with:
Both methods create a 1-page PDF (either in a file or in memory), which shows the pixmap's image and contains a hidden OCR text layer. Reading and text extraction works as usual with this page.

The other way round, if you have a a PDF page, you can do the following:

# first OCR the page by creating a special textpage
tp = page.get_textpage_ocr(flags=3, language="eng")  # use "eng,spa" for multiple languages
# then execute any get_text, / get_textbox / search_for, etc. ... REUSING that textpage:
print(page.get_text("text", textpage=tp)
rects = page.search_for("needle", textpage=tp, ...)
text = page.get_textbox(rect, textpage=tp)

The message is: yes, text locations and other output of e.g. get_text("dict") is as it was before.
How Mupdf works here is:

• If normal text is detected on the page, it is available as before
• Then the page parts covered by an image are OCR-ed. Any text detected in this way is appended to the TextPage object (which, as said, may already contain standard text).
• All text (and, depending on the flags used, also images) is available as you are accustomed to experience it.
• MuPDF does not do the OCR-ing by itself of course, but invokes Tesseract. So MuPDF will simply forward what it gets from Tesseract. For overlapping images, I fear we have to expect garble, but I will check that out.

[JorjMcKie]

@victor-ab - just made a little test with overlapping images:
I could have answered your question better in the first place: MuPDF internally makes a PNG of the page in question, which it hands over to Tesseract. It does not hand over separate images one by one or so. So Tesseract's response will contain what it "sees" on that PNG and knows nothing about overlapping images ...

[victor-ab]

Then the page parts covered by an image are OCR-ed. Any text detected in this way is appended to the TextPage object (which, as said, may already contain standard text).

If there's a mix between embedded text and images with text, does it makes sure it is not OCR'ing the embbeded text twice?

[JorjMcKie]

But you may undergo the effort and extract the page's images yourself one by one, OCR each of them, extract the text from the produced intermediate 1-page PDFs, adjust text coordinates based on the image's rectangle on the original page and continue with that.

This may be even a better way in some cases, because the image can be OCR-ed based on its original size - thus giving a better recognition rate.

[victor-ab]

Got this error:

>>> import os
>>> os.environ['TESSDATA_PREFIX'] = "C:\\Program Files\\Tesseract-OCR"
>>> print(os.path.exists(os.environ['TESSDATA_PREFIX']+'/tessdata'))
True

>>> page = doc[0]
>>> tp = page.get_textpage_ocr(flags=3, language="eng")

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_8528/1466852534.py in <module>
      1 page = doc[0]
----> 2 tp = page.get_textpage_ocr(flags=3, language="eng")

~\mambaforge\envs\layoutlmv2\lib\site-packages\fitz\fitz.py in get_textpage_ocr(self, clip, flags, language)
   5639             if not clip:
   5640                 clip = self.rect
-> 5641             textpage = self._get_text_page_ocr(clip, flags=flags, language=language)
   5642         finally:
   5643             if old_rotation != 0:

~\mambaforge\envs\layoutlmv2\lib\site-packages\fitz\fitz.py in _get_text_page_ocr(self, clip, flags, language)
   5617         language: OptStr = None,
   5618     ) -> "TextPage":
-> 5619         val = _fitz.Page__get_text_page_ocr(self, clip, flags, language)
   5620         val.thisown = True
   5621 

RuntimeError: Tesseract initialisation failed

There's also a typo at initialisation

[victor-ab]

Still got the same error using tessdata

>>> import os
>>> os.environ['TESSDATA_PREFIX'] = "C:\\Program Files\\Tesseract-OCR\\tessdata"
>>> print(os.path.exists(os.environ['TESSDATA_PREFIX']))
True

I can help to debug it, just let me know how.

If it should point to tessdata, then I think the wording in README.md is misleading:

... the system environment variable "TESSDATA_PREFIX" must be defined and contain the tessdata folder name of the Tesseract installation location

[JorjMcKie]

It does not help to set the environment variable inside Python using the os module.
You must use the Windows system for doing this.

[JorjMcKie]

There is documentation how this should look like:

[JorjMcKie]

In Linux respectively, command export TESSDATA_PREFIX=... mus be used ... before starting Python.

[victor-ab]

Using Windows to set it helped. Thanks!

[victor-ab]

Just tested it.
using page.get_textpage_ocr(flags=3, language="eng") generated a file ass.png lol
Wasn't that supposed to be temporary?

It generated font details for the OCR'd data, like ascender, descender, size and even font name. How?

Besides that, what's the DPI? Seems too low, 300 would be better. It'd be nice to have the ability to set it.

[JorjMcKie]

Just tested it. using page.get_textpage_ocr(flags=3, language="eng") generated a file ass.png lol Wasn't that supposed to be temporary?

😂 Yes - a bug if you will inside MuPDF. I had to overwrite their respective C program. But forgot to do that for the final Windows wheels ... damn. Will be postponed to next version.

It generated font details for the OCR'd data, like ascender, descender, size and even font name. How?

What do you mean "how"?

Besides that, what's the DPI? Seems too low, 300 would be better. It'd be nice to have the ability to set it.

Yes, I haven't that one out yet. Still working on it.

[JorjMcKie]

BTW you can always make your high-dpi pixmap of the page then use pix.pdfocr_tobytes().

[victor-ab]

How it inferred the font name, size, etc from just an image? I don't think Is tesseract capable of detecting which font is on the image, or is it?

[JorjMcKie]

How it inferred the font name, size, etc from just an image? I don't think Is tesseract capable of detecting which font is on the image, or is it?

If MuPDF can match OCR-ed text with "normal" text it seems to take over those properties.
True OCR text is deliberately given the font Courier. The reasoning can be found in their source code in "ocr_device.c". The basic point is that Tesseract seems unable to provide a bbox for single characters. It only does that for words. So MuPDF just assumes a monospaced font, the bbox of a word is accepted (from which font size and all the rest of info can be deducted), and hopes for the best as to what a bbox for single characters come out.

If that same page is OCR-ed completely from its pixmap, then the font "GlyphlessFont" provided by Tesseract appears for all text on the page without exception.

[victor-ab]

The basic point is that Tesseract seems unable to provide a bbox for single characters.
It is able.
This is how I do it using pytesseract:

def tesseract_reader(img_section: bytes) -> str:
    return pytesseract.run_and_get_output(
        Image.open(BytesIO(img_section)),
        lang="eng",
        config="--oem 1 -c hocr_char_boxes=1 batch.nochop makebox",
        extension="hocr",
        nice=0,
        timeout=10,
    )

def gen_hocr_bbox(word):
    bbox, conf = word["title"].split(";")
    x, y, x2, y2 = bbox.split(" ")[1:]
    confidence = conf.split(" ")[-1]
    text = word.text
    return {
        "bbox": (
            int(x),
            int(y),
            int(x2),
            int(y2),
        ),
        "text": text.replace("\n", ""),
        "accuracy": float(confidence),
    }

def generate_tesseract_ocr_blocks(ocr_output):
    parsed_html = BeautifulSoup(ocr_output, "html.parser")
    word_spans = parsed_html.body.find_all("span", attrs={"class": "ocrx_word"})

    all_bboxes = []
    for word in word_spans:
        bbox = gen_hocr_bbox(word)
        chars_spans = word.find_all("span", attrs={"class": "ocrx_cinfo"})
        char_bboxes = []
        for char in chars_spans:
            char_bbox = gen_hocr_bbox(char, x_factor, y_factor, img_bbox)
            char_bbox["c"] = char_bbox.pop("text")
            char_bboxes.append(char_bbox)
        bbox["bboxes"] = char_bboxes
        bbox["line_num"] = int(word.parent.attrs["id"].split("_")[-1])
        bbox["par_num"] = int(word.parent.parent.attrs["id"].split("_")[-1])
        bbox["block_num"] = int(word.parent.parent.parent.attrs["id"].split("_")[-1])

        all_bboxes.append(bbox)

Then: generate_tesseract_ocr_blocks(tesseract_reader(img_section))

This tries to mimic the PyMuPDF raw_dict output

[JorjMcKie]

It is able.

It is MuPDF's argument - not mine. Here a quote from said C file:

... Unfortunately, it turns out that Tesseract (with the LSTM engine (the most modern one)) is really bad at giving bounding boxes for characters. It seems that the neural network can say "hey, there is an 'X'", but it can't actually say where the X occurred within the word. So tesseract knows where the words are, and knows the order of the letters within the word, but basically guesses at bboxes for the letters.

[victor-ab]

Doesn't seems like guessing to me. But ok, I understand the design choice

[JorjMcKie]

@victor-ab - I spent a few hours to deepdive into MuPDF code for finding a way to improve resolution for page OCRs.

Nothing seems to work. Their mutool draw does not do any better, although there exists a resolution parameter 🙄.
So I decided to take a different technical approach - that will also remove the ass.png 😎

There will be a dpi parameter, too:

tp = page.get_textpage_ocr(clip=clip, language="eng", flags=flags, dpi=72)
# with this textpage all text-oriented methods work with a speed like usual:
rects = page.search_for("needle", textpage=tp)
xxx = page.get_text(option, textpage=tp)
# etc.

My new approach will OCR the full document page. So the font will be "GlyphlessFont" for everything.

[JorjMcKie]

Here is a script performs "mixed" text extractions:

• for each image on a page, OCR and extract any contained text and return a list of text spans
• extract the spans of "normal" text on the page and extend the previous list

It has the following advantages:

• (probably) faster than a full page OCR: only the image parts of the page are OCR-ed
• text is properly extracted also if there are overlapping images
• do not lose information for normal extractable text
• images are OCR-ed with their native resolution - they do not go through another rendering via page pixmap
• no image resolution concerns

Here is the script:
mixspans.zip

I will also publish this as a Jupyter notebook soon.

[JorjMcKie]

There is progress in supporting OCR directly for document pages. This will be available in v1.19.1.
As mentioned here, a major rewrite of page.get_textpage_ocr() will be available:

textpage = page.get_textpage_ocr(
                clip=None,
                flags=...,
                dpi=72,
                language="eng",
                full=False,
            )

All subsequent text searches and text extractions must use this textpage to access the OCR results.

• full=True: OCR the full page with desired quality / resolution using the dpi argument. This will internally generate a Pixmap. Applicable text extractions will see "GlyphlessFont" as the fontname for all text.
• full=False: Only OCR the images shown - not the full page. No page pixmap will be created. Every image is being processed with its native quality. The dpi parameter will be ignored. The returned textpage will contain a mixture of original and OCR-ed text. If images overlap, their text will nonetheless be entirely available.

[JorjMcKie]

@victor-ab - the new v1.19.1 is out. It contains the announced improvements for document page OCR - with one minor change:
I do no longer support a clip parameter for creating the OCR textpage.
Otherwise, setting desired dpi is possible.
What I find even more interesting is the feature to do a "partial" OCR for pages:
As indicated in the previous post, this means that normal page text goes through as it is, and only text inside displayed (!) images (if any) will be OCRed.

[victor-ab]

Seems like a great improvement! Will it OCR the images at original resolution?

[JorjMcKie]

Will it OCR the images at original resolution?

Yes, that's why I decided to ignore the dpi argument in that case.
Although technically it would be possible to also use it then. I just believe that OCR detection quality wouldn't profit.

[JorjMcKie]

You can try the resp. Jupyter notebook which walks through a simple such comparison full vs. partial OCR.

[JorjMcKie]

This one.

[victor-ab]

@JorjMcKie
Any ideas on what went wrong here?

>>> partial_tp = page.get_textpage_ocr(flags=0, full=False)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_5268/725528090.py in <module>
      1 page = doc[0]
----> 2 partial_tp = page.get_textpage_ocr(flags=0, full=False)

~\mambaforge\envs\env\lib\site-packages\fitz\utils.py in get_textpage_ocr(page, flags, language, dpi, full)
    581             continue
    582         pix = Pixmap(block["image"])  # get image pixmap
--> 583         imgdoc = Document("pdf", pix.pdfocr_tobytes())  # pdf with OCRed page
    584         imgpage = imgdoc.load_page(0)  # read image as a page
    585         pix = None

~\mambaforge\envs\env\lib\site-packages\fitz\fitz.py in pdfocr_tobytes(self, compress, language)
   6855 
   6856         bio = BytesIO()
-> 6857         self.pdfocr_save(bio, compress=compress, language=language)
   6858         return bio.getvalue()
   6859 

~\mambaforge\envs\env\lib\site-packages\fitz\fitz.py in pdfocr_save(self, filename, compress, language)
   6838         self, filename: AnyType, compress: int = 1, language: OptStr = None
   6839     ) -> AnyType:
-> 6840         return _fitz.Pixmap_pdfocr_save(self, filename, compress, language)
   6841 
   6842     def pdfocr_tobytes(self, compress=True, language="eng"):

RuntimeError: PDFOCR expected to be Grayscale or RGB

[JorjMcKie]

Ah, interesting case!
One of the images shown on the page is neither grayscale nor RGB (e.g. instead CMYK).
Which OCR Tesseract does not support apparently. New to me.
So, for this version, you will have to go with full page OCR in this case.
In the next version, I will capture this condition and internally convert such images to RGB.

[JorjMcKie]

Are you ok sharing that file / page to test a fix?

[victor-ab]

Yeah, just confirmed, it is a CMYK
Here is the original image in bytes. Not a real zip, just change the extension.
bytes.zip

[JorjMcKie]

@victor-ab : Solution was easy - thanks for the file.

I am modifying the OCR textpage method page.get_textpage_ocr() somewhat as a consequence of your report, and also clarify details on what will happen when using full vs partial OCR. The following will find its way into the next version's code and documentation:

Full page OCR

Stays as it is. All text will be in GlyphlessFont format.
The flags bits will have whatever effect on the OCRed text itself.
If requesting images (TEXT_PRESERVE_IMAGES is set), then only the PNG image of the full page will be included (resolution as requested by dpi). No other images can be found in the produced OCR textpage.

Partial page OCR

Functionality todate stays the same. Following changes will be implemented:
* The image extraction bit TEXT_PRESERVE_IMAGES will be enforced. For obvious reasons: Otherwise, nothing could be found to OCR and using this method would make no sense at all.

• When stepping through the images, I will only consider those that are large enough: (original) width and height must > 3 pixels. This serves as a performance improvement only.
• If encountering a Non-RGB, an internal (and temporary) conversion will take place to ensure OCR is possible. E.g. CMYK images are not supported and must be converted. Same applies to images with alpha channel! When the user extracts stuff from the TextPage he will see all images still in their original format.

The question I have for you:
That new pixmap conversion to RGB might fail ... for whatever reason. What would be an adequate reaction?

1. Do nothing: Let the exception happen and leave appropriate reaction to the programmer.
2. Automatically fall back to full page OCR (with a warning message?). In this case, the dpi parameter would remain important and not be ignored.

[RomaKoks]

Hi! Thanks for your work!
On the documents I tested, OCR works very well, the block representation is especially pleasing.
But, I could not find in the documentation how to put the whole resulting OCR into a document's page and save document with invisible text layer. How to do it right?

[JorjMcKie]

@RomaKoks - thanks for the feedback 👍.

Please look at the documentation for method Pixmap.pdfocr_tobytes(). It shows how to join and convert images in a folder to a PDF made from the OCRed images.

If you have a scanned PDF and don't want to use the specialized OCRmyPDF, try this:

for page in doc:
    tp = page.get_textpage_ocr(...)
    words = page.get_text("words", textpage=tp)
    for word in words:
        bbox = fitz.Rect(word[:4])
        page.insert_text(bbox.bl, word[4], fontname="cour", render_mode=3)
doc.ez_save("scanned-ocr.pdf")

Tesseract OCRs with its own font "GlyphLessFont" which is mono-spaced. So best output it as Courier - or some other mono font of your liking.
Instead of render mode you could of course also use overlay=False or fill_opacity=0.

[RomaKoks]

Thanks a lot for the tips! If I save the text in this way, will the block structure be preserved when I extract the text from the document again?

for page in doc:
    blocks = page.get_text("blocks")

[JorjMcKie]

BTW: I forgot to say how to compute the fontsize above ...!

Should at least be similar. However, OCR always is a bit shaky.
Please remember that the OCR tool recognizes on the word level, and each word is being given the minimum possible bbox. This means that a word like "area" will have a different height than "Area".
OCR also does not understand that words using descender characters (like "g", "q", etc.) or characters using both, the descender and the full ascender like "|", "(" etc. may still have been originally written in the same fontsize.
This is something you must provide as additional logic: the word bbox for "Gag" must be inserted with an insertion point a little higher than bbox.bl - because of the "g", whereas "GAG" should be okay with it.
Similaly, if you use bbox.height as the fontsize for the bbox of "area", the word will come out larger smaller than expected, because it contains no uppercase character ... etc., etc.

[JorjMcKie]

You probably want to evade all that hassle and choose a different approach:

doc=fitz.open("scanned.pdf")  # input
out=fitz.open()  # output containing OCR text layer
for page in doc:
    pix = page.get_pixmap(dpi=300)  # make a high resolution page image
    pdfbytes = pix.pdfocr_tobytes()  # convert to 1-page temp PDF with OCR
    src = fitz.open("pdf", pdfbytes)  # open it
    # make output page with same dimension
    outpage = out.new_page(width=page.rect.width, height=page.rect.height)
    # include OCRed page in output page
    outpage.show_pdf_page(outpage.rect, src, 0)
    src.close()
out.save("scanned-with-ocr.pdf")

Your output will contain the input images with the text layer.
When extracting text from it, you will see the original Tesseract font "GlyphLessFont".

[RomaKoks]

Thank you very much, it works super, but for quite a long time. It feels like OCR in this mode works in single thread.
Also I don't know why, results are a bit better than from OCRmyPDF (same DPI), but resulting doc size is x10 from original. Do you know easy way to reduce size?

[JorjMcKie]

better than from OCRmyPDF

Really? After all, this package is specialized for this task and has features not (yet) present in PyMuPDF: automatic de-skewing, dynmic image improvement, output of PDF-1A standard.

works in single thread

True, PyMuPDF supports no Python threading. But you can use the multiprocessing module: make groups of page ranges of the input (adjusted to the number of available prcessors). Then you need logic to collect each processes output into one single output PDF.

size is x10 from original

Simply save the resulting file with appropriate options: garbage=4, deflate=True. Also play with the DPI value: its influence on image size is quadratic! A more serious problem is that the standard Pixmap image format is PNG, whereas your input scanned PDF presumably has JPG page images.
This is not easy to resolve, but possible. Works along the lines of this demo script. In essence it would mean to go over each page, extract the image, make a JPG version of it (using e.g. Pillow), then replace the page image with its JPG version. Could save 80% of size.

Another hickup: my recipe makes pictures of the page - i.e. creates additional images in the end.
But the input file already has those images, because the input page simply shows it.
So a more correct approach would be to not make another pixmap, but directly take the page's image.
This also has its own set of problems, because - depending on the scanner product - multiple images may make up the displayed image, and special techniques would be required to get this resolved.