Not detecting some vertical lines

[RyanAquino]

Hello everyone,

I am not sure if I have the proper table settings, but I am trying to retrieve all the data and format it in a way that I can determine in which table the extracted data came.

Current issue is that, some vertical lines are not being detected properly, although my table setting is already set as vertical_strategy: lines.

page = pdf.pages[1] 
start = min([x['x0'] for x in page.horizontal_edges])
end = max([x['x1'] for x in page.horizontal_edges])

table_settings = {
    "vertical_strategy": "lines",
    "horizontal_strategy": "text",
    "intersection_x_tolerance": 10,
    "explicit_vertical_lines": [start, end]
}

PDF file
05122000.pdf

Lines that was not detected

[samkit-jain]

Hi @RyanAquino Appreciate your interest in the library. To get the missing lines, you can use page.curves+page.edges. The updated code will be

page = pdf.pages[1]
start = min([x['x0'] for x in page.horizontal_edges])
end = max([x['x1'] for x in page.horizontal_edges])

table_settings = {
    "vertical_strategy": "explicit",
    "horizontal_strategy": "text",
    "intersection_x_tolerance": 10,
    "explicit_vertical_lines": page.curves+page.edges + [start, end]
}

and the output will be

[RyanAquino]

Hello @jsvine,

Thanks for your reply. Unfortunately, none of the options of repairing the PDF works.

I tried using extract_text to extract all data, but my next issue comes when I want to know which data are from which table. Considering tables like this that has 2 columns on the same row. Do you think this is still possible to identify when using extract text?

[jsvine]

Thanks for your reply. Unfortunately, none of the options of repairing the PDF works.

How about this?:

# This throws out the tiny edges of the tiny rectangles,
# which were causing problems
vlines = [ e for e in page.vertical_edges if e["height"] > 10 ]

# This will help get you vertical lines at the far left and right 
# of the table
extreme_xs = [
    min(e["x0"] for e in page.horizontal_edges), 
    max(e["x1"] for e in page.horizontal_edges) 
]

im.reset().debug_tablefinder({
    "horizontal_strategy": "text",
    "vertical_strategy": "explicit",
    "explicit_vertical_lines": vlines + extreme_xs,
})

... which gets you this:

I tried using extract_text to extract all data, but my next issue comes when I want to know which data are from which table. Considering tables like this that has 2 columns on the same row. Do you think this is still possible to identify when using extract text?

I don't think I completely understand this. Could you explain a bit more and/or provide some code and outputs that demonstrate the question?

[RyanAquino]

Thanks for your reply. The table settings you have provided fixes the issue on vertical lines not being detected correctly.

I don't think I completely understand this. Could you explain a bit more and/or provide some code and outputs that demonstrate the question?

• Considering the table below. Table III-B and Table III-C is on the same row. since pdfplumber extracts the data basing it on vertical lines/text. My issue comes when I want to know which table data(in this case the table label with its values e.g Debt held by the public : [ '$ 4,730,099', '$ 4,729,588', etc...]) is on which table it belongs to?

Here is a sample data extracted by pdfplumber using extract_table.

# Somewhere in the response data

['TABLE III-B—Adjustment of Public Debt',
  None,
  None,
  None,
  None,
  'TABLE III-C—Debt Subject to Limit',
  None,
  None,
  None,
  None],
 ['Transactions to Cash Basis',
  None,
  None,
  None,
  None,
  '',
  None,
  None,
  None,
  None],
 ['This Fiscal',
  None,
  None,
  None,
  None,
  'Balance Transactions',
  'Closing\nbalance\ntoday',
  'Opening balance\nThis Fiscal\nToday',
  None,
  None],
 ['Transactions',
  'Today',
  'month\nto date',
  None,
  'year\nto date',
  'Debt Held by the Public',     <-- e.g I have no way of telling which table this data came from
  '$ 4,730,099',
  '$ 4,729,588',
  'month\n$ 4,709,403',
  'year\n$ 4,601,239'],

I am not sure if this is something that pdfplumber can support?

[jsvine]

The issue here is that, to a simple algorithm (which pdfplumber's is), the tables on that page look like they're all part of one table, since they have lines joining them. My approach for something like that would be to use the line/rect attributes to determine the positioning of those tables, and then use page.crop(...) to isolate them ... and then do the table extraction.

[RyanAquino]

Got it. thanks!