Object precision

[ramcdona]

Is there any rounding or formatting of floating point values that occurs when pdfplumber extracts line, rect, curve?

The floating point numbers seem to typically have three digits after the decimal point. Is that the full extent of the information contained in the PDF file, or has some rounding or formatting been done by pdfplumber along the way? If there has been some control on precision, is there an option to increase the precision?

I'm presently using the command line pdfplumber program and writing out *.csv files. It certainly may be that precision is truncated when writing the *.csv that is preserved in the json or if I were to access the library directly from Python.

[jsvine]

Is there any rounding or formatting of floating point values that occurs when pdfplumber extracts line, rect, curve?

pdfplumber converts floating point values to Decimal objects here, using Decimal.quantize:

pdfplumber/pdfplumber/utils.py

Lines 149 to 162 in de767c3

	def _decimalize(v, q=None):
	# Convert int-like
	if isinstance(v, numbers.Integral):
	return Decimal(int(v))
	# Convert float-like
	elif isinstance(v, numbers.Real):
	if q is not None:
	return Decimal(repr(v)).quantize(Decimal(repr(q)), rounding=ROUND_HALF_UP)
	else:
	return Decimal(repr(v))
	else:
	raise ValueError(f"Cannot convert {v} to Decimal.")

The precision is, by default, set to 0.001 but can be adjusted while reading in a PDF:

pdfplumber/pdfplumber/pdf.py

Lines 18 to 29 in de767c3

	class PDF(Container):
	cached_properties = Container.cached_properties + ["_pages"]
	def __init__(
	self,
	stream,
	pages=None,
	laparams=None,
	precision=0.001,
	password="",
	strict_metadata=False,
	):

In my experience this default setting has not posed a problem but, of course, your mileage may vary.

I'm presently using the command line pdfplumber program and writing out *.csv files. It certainly may be that precision is truncated when writing the *.csv that is preserved in the json or if I were to access the library directly from Python.

There is (or should be) no number-precision difference between the command-line output and what you would get from using the Python library directly.

[ramcdona]

Actually, it looks like pdfminer.six converts the points data to a string before it ever gets to pdfplumber. And the truncation to three digits after the decimal happens there.

That seems very unfortunate...

[jsvine]

Lots to unpack here! Let's start with this:

Actually, it looks like pdfminer.six converts the points data to a string before it ever gets to pdfplumber. And the truncation to three digits after the decimal happens there.

I believe that's incorrect. The method you point to appears only to be used when pdfminer.six is rendering an XML representation of a PDF — see here. Instead, the rest of pdfminer.six processing works with the original .pts attribute. Indeed, directly running pdfminer.six's highlevel.extract_pages(...) method produces objects with .__dict__ properties like these, full of arbitrary-precision floats:

{
 'bbox': (33.12, 550.3, 975.36, 588.0039999999999),
 'evenodd': True,
 'fill': True,
 'height': 37.70399999999995,
 'linewidth': 0,
 'non_stroking_color': [0.8, 1, 1],
 'pts': [(33.12, 550.3),
         (975.36, 550.3),
         (975.36, 588.0039999999999),
         (33.12, 588.0039999999999)],
 'stroke': False,
 'stroking_color': None,
 'width': 942.24,
 'x0': 33.12,
 'x1': 975.36,
 'y0': 550.3,
 'y1': 588.0039999999999
}

Instead, the rounding takes place in this library, pdfplumber, during the conversion of these floating point values to Decimal representations.

The main reason is that, as you can see above, Python's floats are often unwieldy ... and often imply false precision. (588.0039999999999 is not a number precise to more than a dozen decimal places; instead, it's likely one precise to just a few places but which cannot be represented exactly as a binary fraction. For more on Python's handling of floating point arithmetic, see here.)

Another reason: Python's floats are especially tricky when doing a combination of arithmetic and equality-testing, as is often common when using pdfplumber. For instance, let's say I want to find lines with an x0 position exactly 0.1 points to the right from a given line. And let's say that given line is at 0.35, so we should be catching all lines where x0 is 0.45. With Decimal-converted values, this works as expected. But with raw floating point values: 0.35 + 0.1 == 0.45 evaluates to False ... because 0.35 + 0.1 evaluates to 0.44999999999999996.

-- not as formatted strings

Per above, they're not formatted into strings, but rather Decimal representations.

If the PDF file contains a perfectly good binary number, why convert it?

Per the above, the PDF files don't always contain perfectly good binary numbers. Text characters are, I think, the prime example: When creating a PDF, you rarely say "this letter goes at this point, and this one at that point, etc."; instead, you say, "this string of text starts here and then continues" ... and then pdfminer.six (and other parsers and renderers) have to use the font definition to calculate each character's position.

From that perspective, I would like to see no rounding [...]

Hopefully the explanation above has convinced you why this is not the case in pdfplumber. Instead, if you'd like a particular level of precision, I'd recommend setting the precision keyword argument (noted above) to the level you desire [edit: or to None if you want no rounding at all]. That said, I'm certainly open to hearing more perspectives on this, and will absolutely consider any counterpoints.

[ramcdona]

I'm very well versed in the ins and outs of representing numbers with floating point values.

This is not a situation where the exact representation characteristics of Decimal are needed vs. normal floating point values.

When you're dealing with floating point numbers and computers, you basically have to do away with ==. It doesn't have real meaning except in very particular cases. Using Decimal may 'fix' this in an algorithmic sense (not really), but it certainly doesn't fix it in a practical sense.

Instead, if you want to find objects in a box, you can use < or <= and > or >=. If you want to perform a hit-test, you'll need to use a tolerance and find the nearest point to the hit -- not one with the distance == 0.

Any floating point binary number is a perfectly good binary number. even if it is a 0.44999999999996 -- I'm OK with that.

If you really want to do some rounding / truncation in order to do your own tests internally (strongly not recommended), then OK -- but I still strongly feel that users will expect data to pass through and be converted in format -- but not changed in value.

All that said -- It is very possible that the PDF specification says three digits after the decimal point. That would explain why pdfminer.six does what it does when it writes the XML representation (thanks for clearing that up). If all PDF files already truncate / round to three digits after the decimal, then this conversation is moot and we can truck on forward.

I just spent some time with the PDF spec.... Section 3.2.2 discusses numeric object types. It says that real numbers are treated as fixed point. It points to Appendix C for specific limits. App C has a table that says that real values in PDF files have five digits of precision in the fractional part.

So, with the PDF spec setting the precision, most of this is moot. It probably makes sense for you to use .00001 for your tolerance by default. It also probably makes sense for pdfminer to change their format statement when writing XML PDF files.

That said, I would still encourage you to avoid using ==. It really doesn't have great behavior in the real (number) world. Avoiding it will avoid a lot of problems.

Even if you take the limits of computer representation out of the picture and just think about real valued numbers. Imagine a random number generator that returns real values between 0 and 1 with equal probability. Run two of these RNG's and think about the utility of comparing their results with ==. The problem is, there are infinite real numbers between 0 and 1. The probability of ever actually achieving == is actually zero. So, from a pure math standpoint, the concept of == becomes dubious and impractical as soon as you are working with the real number line.

[jsvine]

I'm very well versed in the ins and outs of representing numbers with floating point values.

Ah, of course. I didn't mean to imply that you weren't, but rather to spell out my thinking in as concrete terms as possible. And I'm glad I did! Seeing those arguments laid out, and your counter-arguments, I think there's a good case for changing this behavior in pdfplumber. Thank you for that.

I've pushed a commit (to a new un-decimalize branch) that would do this: e7dcc10

Here's the commit message, which summarizes the motivations and results:

Per discussion at https://github.com/jsvine/pdfplumber/discussions/346
and input from @ramcdona, this commit changes pdfplumber's behavior
regarding floating point numbers. Specifically, it removes all
conversion of floats to Decimal objects. This brings several advantages:

- Increased precision (where applicable)
- Decreased code complexity
- Increased performance (~10% speedup on test suite)
- Increased fidelity to `pdfminer.six` output

These seem to outweigh the disadvantages:

- Some tests break (but have been easily fixed) due to increased
  precision and/or floating point arithmetic artifacts
- Some users' scripts may also break, if they depend on strict equality
  testing, though these *should* also be easily fixable

Because some form of automatic rounding may still be desirable for the
pdfplumber CLI utility, the conversion methods (.to_csv, .to_json) have
been adjusted to accept a `precision` argument.

Given that this is a non-trivial architectural change, I'd be curious for @samkit-jain's thoughts, as well as those of anyone else closely watching this repository/thread. (And if the change is made, does it warrant bumping the library's version up to 0.6.0?

[jsvine]

Thank you again for this suggestion, @ramcdona. And an update: As of version 0.6.0, pdfplumber no longer uses decimalization. It also allows a --precision argument to be passed to the pdfplumber CLI.

[ramcdona]

Thanks for the update -- glad this made its way in. I've been pulled away from the project that was going to have me use pdfplumber. Hopefully I'll be able to find my way back sometime this year. Rob

…

On Thu, Dec 23, 2021 at 7:11 PM Jeremy Singer-Vine ***@***.***> wrote: Thank you again for this suggestion, @ramcdona <https://github.com/ramcdona>. And an update: As of version 0.6.0, pdfplumber no longer uses decimalization. It also allows a --precision argument to be passed to the pdfplumber CLI. — Reply to this email directly, view it on GitHub <#346 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKETQDJHFLTXHOJG6AGQCTUSPQE5ANCNFSM4XEG5ZYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>