bug/Two Column PDF partition result in incorrect text.

[pfcharles]

Describe the bug
When running partition on a two column pdf, text extraction puts characters is the wrong position
To Reproduce
two_col.pdf

Provide a code snippet that reproduces the issue.
elements = partition("two_col.pdf", strategy="fast")

text attribute of elements[2] = '1. Exchange of Information. The parties agree to exchange Confidential Information for the purpose of (the evaluating a potential business "Purpose") in accordance with this Agreement.'
text attribute of elements[3] = 'relationship'

Actually text from the pdf = '1.Exchange of Information. The parties agree to exchange Confidential Information for the purpose of evaluating a potential business relationship (the "Purpose") in accordance with this Agreement.'

two_col.json

Expected behavior
Extracted text matches the actual text

Screenshots

Environment Info
Please run python scripts/collect_env.py and paste the output here.
OS version: macOS-14.5-arm64-arm-64bit
Python version: 3.9.6
unstructured version: 0.14.9
unstructured-inference version: 0.7.36
pytesseract version: 0.3.10
Torch version: 2.3.1
Detectron2 is not installed
PaddleOCR is not installed
Libmagic version: file-5.41
magic file from /usr/share/file/magic
LibreOffice version: ==> libreoffice: 24.2.4

Additional context
Add any other context about the problem here.

[pfcharles]

Looks like this is a problem with the underling use of the pdfminer library. the data returned by the pdfminer.layout.LTTextBoxHorizontal object get_text() method in pdf.py is wrong.

[pfcharles]

two_col_not_justified.pdf

This appears to be related the document being text justified and there being larger spaces between words. The issue appears to be related to the implementation of find_neighbors in the pdfminer layout. To some extent this can be controlled by the LAParams initialized in init_pdfminer. Other libs like PyPDF and (java)PDFBox handle with no issue or special configuration.

[pfcharles]

Running this with pdfminer's pdf2txt.py, does not scramble this paragraph, so it must be something in unstructured use of pdfminer.