Inaccurate bbox information obtained for 1.18.17 while correct information for 1.18.5

[Yichen-fqyd]

Hi

I have a file while the the bbox information outputs from 1.18.17 are pretty small in terms of height dimension. while the 1.18.5 version seems correct.

I will send the file via email for the privacy issue
Here is the output
1.18.17 output

(221.4538116455078, 110.48623657226562, 372.69598388671875, 113.1869888305664) (41.40359878540039, 156.53033447265625, 317.1805114746094, 158.03076171875) (41.40359878540039, 180.53701782226562, 177.94168090820312, 182.03744506835938) (41.40359878540039, 204.54371643066406, 153.93499755859375, 206.0441436767578) (41.40359878540039, 228.5504150390625, 254.16293334960938, 230.05084228515625) (42.30384826660156, 274.4210510253906, 437.8141784667969, 275.6213684082031) (42.30384826660156, 295.4269104003906, 428.21148681640625, 296.6272277832031) (42.30384826660156, 316.4327697753906, 447.4168395996094, 317.6330871582031) (42.30384826660156, 337.4386291503906, 437.8141784667969, 338.6389465332031) (42.30384826660156, 358.4444580078125, 456.4193420410156, 362.04547119140625) (42.30384826660156, 384.8518371582031, 466.6221923828125, 386.0521545410156) (42.30384826660156, 405.8576965332031, 466.02203369140625, 407.0580139160156) (42.30384826660156, 405.8576965332031, 466.02203369140625, 407.0580139160156) (41.40359878540039, 626.5928955078125, 100.52008056640625, 627.9432373046875) (41.70368194580078, 657.201416015625, 549.4453125, 658.5517578125) (53.70703125, 674.006103515625, 171.94000244140625, 675.3564453125) (53.70703125, 674.006103515625, 171.94000244140625, 675.3564453125) (290.1729736328125, 816.3746948242188, 304.5769958496094, 817.5750732421875) (290.1729736328125, 816.3746948242188, 304.5769958496094, 817.5750732421875)

1.18.5 bbox
(221.4538116455078, 93.73312377929688, 372.69598388671875, 115.33915710449219) (41.40359878540039, 147.22305297851562, 317.1805114746094, 159.22640991210938) (41.40359878540039, 171.229736328125, 177.94168090820312, 183.23309326171875) (41.40359878540039, 195.23643493652344, 153.93499755859375, 207.2397918701172) (41.40359878540039, 219.24313354492188, 254.16293334960938, 231.24649047851562) (42.30384826660156, 266.9752197265625, 437.8141784667969, 276.577880859375) (42.30384826660156, 287.9810791015625, 428.21148681640625, 297.583740234375) (42.30384826660156, 308.9869384765625, 447.4168395996094, 318.589599609375) (42.30384826660156, 329.9927978515625, 437.8141784667969, 339.595458984375) (42.30384826660156, 350.9986267089844, 456.4193420410156, 363.0019836425781) (42.30384826660156, 377.406005859375, 466.6221923828125, 387.0086669921875) (42.30384826660156, 398.411865234375, 466.02203369140625, 408.0145263671875) (42.30384826660156, 398.411865234375, 466.02203369140625, 408.0145263671875) (41.40359878540039, 618.21630859375, 100.52008056640625, 629.0193481445312) (41.70368194580078, 648.8248291015625, 549.4453125, 659.6278686523438) (53.70703125, 665.6295166015625, 171.94000244140625, 676.4325561523438) (53.70703125, 665.6295166015625, 171.94000244140625, 676.4325561523438) (290.1729736328125, 808.9288940429688, 304.5769958496094, 818.5315551757812) (290.1729736328125, 808.9288940429688, 304.5769958496094, 818.5315551757812)

[JorjMcKie]

Uff - you do have a talent to dig out problem-PDFs 😉.

This is a MuPDF problem with that specific font, SimSun. You can reproduce it using the PDF viewer SumatraPDF (works on Windows only - or with Wine on Linux).
Adobe Acrobat works ok, as well as Foxit reader. Nitro PDF and PDF XChange have the same problems as MuPDF / SumatraPDF.

All you can do is adding a plausibility check to your text extraction routine:

1. Execute fitz.TOOLS.set_small_glyph_heights(True), this ensures that character bbox height equals fontsize - in normal situations.
2. In your text extraction, check whether bbox height is smaller than fontsize. If it is, then you have that problem. Set y0 = y1 - fontsize and go with the resulting bbox.

Example:

>>> fitz.TOOLS.set_small_glyph_heights("True")
True
>>> page=doc[0]
>>> page.clean_contents()
>>> blocks = page.get_text("dict")["blocks"]
>>> for b in blocks:
	if b["type"] !=0: continue
	for l in b["lines"]:
		for s in l["spans"]:
			bbox = fitz.Rect(s["bbox"])
			if bbox.height < s["size"]:
				bbox.y0 = bbox.y1 - s["size"]
			page.draw_rect(bbox, width=0.3, color=(1,0,0))

Gives you nice results.

[Yichen-fqyd]

More for this setting fitz.TOOLS.set_small_glyph_heights(True), what if the case when the bbox height is greater than the font size, will the setting force the character bbox height equals fontsize

[JorjMcKie]

Thank you for the reply!! It works, but just wondering why the 1.18.5 version bbox output looks right?

I have change my bbox height correction a few times to adapt to special cases brought up by users. So this may well be - although I do not remember the last time of change.

[JorjMcKie]

More for this setting fitz.TOOLS.set_small_glyph_heights(True), what if the case when the bbox height is greater than the font size, will the setting force the character bbox height equals fontsize

No. The normal case is that the font values ascender and descender are such that ascender - descender >= 1 (descender is negative).
The line height standard is this value * fontsize. Setting small glyph height ensures that the line height equals fontsize. This is often more practical when text searching and redaction definition: no unwanted deletion of text in lines above / below the intended one.

[JorjMcKie]

my suggested code snippet ensures that only cases with ill-defined fonts like SimSun are considered.

[Yichen-fqyd]

So the second method is more recommended? I will try the second method then!

[JorjMcKie]

I have looked at the embedded font SimSun. Its total height (ascender - descender) is far below 1, even below 0.2.
I do have my own correction routine for such cases, but it currently only jumps in if font.ascender < 1e-3. If I would put the limit higher, e.g. font.ascender < 0.2, then my text extraction would look normal like the correction in previous post.

[Yichen-fqyd]

Always hard to find a perfect threshold/config = =

[JorjMcKie]

page.clean_contents() eliminates some problems that a page may have.
Exec immediately after loading a page if you have suspicions about the health of the PDF / the page.