[Feature Request] Table structure extraction at the API

[troplin]

There is already some table detection mechanism in tesseract but unfortunately, there is seems to be no possibility to access the table structure at the API.

This could be done only minimal changes to the API, just by expanding the PageIteratorLevel enum by two additional members RIL_TABLEROW and RIL_TABLECELL or similar.
Those would only be relevant inside PT_TABLE blocks, just like PT_PARAGRAPH is only meaningful for text blocks.

[zdenop]

Are you able to send PR for this including simple test case (similar to #1614)?

[troplin]

@zdenop I didn't mean to imply that it was easy to implement, just that the interface changes are small. I have honestly no idea what it takes but if I find the time, I'll give it a try.

[amitdo]

I assume tesseract handle tables in one of these two ways:

1. Tables columns are held in tesseract blocks and cells are held as lines within blocks.
2. Tables rows are held in tesseract blocks and cells are held as lines within blocks.

I bet on option (1).

[troplin]

@amitdo I'm pretty sure that this is not the case. I fear that the information is lost completely.
IMO it's also not a very good representation. Cells can be multi-line, so they should be comparable to paragraphs, not lines.

The table I'm testing with seems to be recognized as a single block (which makes sense IMO).
But then the table is split into two paragraphs (one for the first row and one for the rest), which does not make much sense.
The lines span the whole table. For multiline cells, the lines of each cell are combined into nonsensical long lines.

If this reflects the internal table structure, that would mean that the table detection is really bad and I can just disregard it.
If not, the results could be presented much better.
The fact that the table separators are actually recognized as horizontal and vertical lines makes me think that the information might be there.

I'm going to investigate a bit more, once I've successfully set up the debug viewer.

[amitdo]

Tesseract considers any table it can recognize as block, so it's neither of the cases.

[amitdo]

The table detection code is here:
https://github.com/tesseract-ocr/tesseract/blob/master/src/textord/tablefind.cpp

[amitdo]

Play with the variables:

tesseract/src/textord/tablefind.cpp

Line 143 in 509a6f0

BOOL_VAR(textord_show_tables, false, "Show table regions");

[amitdo]

They published a paper about the table detection module.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.638.7400&rep=rep1&type=pdf

[Shreeshrii]

https://github.com/tabulapdf/tabula/issues/409#issuecomment-327050906 from someone who has scripted the process of splitting a PDF into cells and OCR'ing them separately: chain pdf-table-extract, ghostscript and tesseract

[Shreeshrii]

Related issue - How to detect table region after the update in Tablefind.cpp? #825

[Shreeshrii]

Also see https://github.com/DanBloomberg/leptonica/blob/5dca24f9674c7fd057ab55bbfc71efa87a83a520/version-notes.html#L180

 Improved table detection on scanned page images (tests: pageseg_reg.c)

DanBloomberg/leptonica@18342b4

[troplin]

Thanks for all the pointers. I don't want to change the table detection though, or even implement it myself. I just want the results accessible at the public API, if available.

Is there a high-level description of the internal processing pipeline of tesseract somewhere?

[troplin]

Ok, I've got the debug viewer running.

It seems, that the table detection works perfectly:

But then, the contents of the table are just processed as any other text, which doesn't make sense to me:

So, this means that the data is actually there, but it's not actually used.
Is this, because the whole table is a simple block? Would it be better to treat every cell as single block and represent the table structure on a higher level?

[Sintun]

Hi, I'm thinking about writing the API / structure part necessary to hold and access the otherwise lost information (as described by @troplin ). Is the approach described in @troplin s first comment feasible? Would a commit from a tesseract-team outsider be acceptable? Is there a Guideline for your c++ code-styling? And: Is this the right place to ask these questions?

[amitdo]

Hi @Sintun!

I'm thinking about writing the API / structure part necessary to hold and access the otherwise lost information (as described by @troplin ). Is the approach described in @troplin s first comment feasible?

I don't know.

Would a commit from a tesseract-team outsider be acceptable?

Yes, of course.
https://github.com/tesseract-ocr/tesseract/graphs/contributors
AFAIK, only 2 people in this list are from Google.

Whether a specific PR will be accepted or rejected will depend on it's code quality.

Don't break an existing API.

Is there a Guideline for your c++ code-styling?

Not officially, but since most of the code comes from Google, it's a good idea to use Google C++ Style Guide.

Is this the right place to ask these questions?

I think so.
.

[Sintun]

Nice, I will start this and a traceback and possible fix of #1712 as a weekend-project.

[amitdo]

I suggest that you provide an example that demonstrates the use of the new API.

Good luck!

[troplin]

@Sintun It might actually be better to invert my API suggestion and do something like:

• RIL_TABLE
  • RIL_TABLEROW
    • RIL_BLOCK (== cell)
      • RIL_PARA

instead of my earlier suggestion, which was:

• RIL_BLOCK (== table)
  • RIL_TABLEROW
    • RIL_TABLECELL
      • RIL_PARA

I don't know which one is better, it depends on what a block means to the engine. I already tried to figure out how the recognition process works on a high level, but I'm a bit lost.

Maybe someone with a deeper understanding of the internals could give a hint?

[amitdo]

Maybe someone with a deeper understanding of the internals could give a hint?

I think only @theraysmith can help here.

[Sintun]

@troplin At a first sight your new suggestion looks more logical. I hope that the existing structures will give me a clear path to hold the table information, otherwise i will stick with the most logical (at least for me) non-API breaking approach. Thanks for the hint, I will take it into account when looking at the code.

@amitdo I think i have to invest some time and dive through the surrounding code before i can understand helping hints of people who are familiar with the code base.

[Shreeshrii]

@zdenop Please label as Feature Request.

[zdenop]

@Sintun : any progress on this issue? Is API needs to be changes I would like to get it for 4.0 release...

[Sintun]

@Sintun : any progress on this issue? Is API needs to be changes I would like to get it for 4.0 release...

Unfortunately not yet, i'm still working on #1192 / #1712 , because it seems to be a more pressing matter.

[krishna11888]

what happened to table extractor feature api

[Sintun]

Hi, I wanted to make the information accessible through the api.
Unfortunately i wasn't able to find enough free time to do it.
Fortunately my employer needs this feature, and I'm going to propose (next week) using this tesseract code and updating the tesseract api. I'm pretty confident that this will go through, so i could focus on it, which should enable me to finish it in the near future.

[zdenop]

Just side note: there is python project for extracting table data from pdf: Camelot and there is also web interface for it: excalibur

[saiprasadjnv]

Is this feature request still open? I am interested in working on it. If someone is already working on it, we can collaborate and speed up the process. Please feel free to contact me at [email protected].

[zdenop]

@saiprasadjnv you can work on this. @Sintun started to work on this, but never send PR here, so IMO this is unfinished task.

[amitdo]

I suggest to start with testing @Sintun's patches in his fork.

[amitdo]

do we have any sample codes on table detection using tesseract?

#1714 (comment)

[balachandarsv]

I tried with some sample tables, @Sintun's solution works well. Any idea when this would be merged into master?

[Sintun]

I could update it and create a pull request around the next weekend, if someone gives me an absolution for the usage of a singleton approach (an object that shares properties with global variables :( ).

[balachandarsv]

@Sintun any update on the pull request?

[Sintun]

Hi, sorry for the delay, and thanks for the reminder. Since no one objected my approach i will create a pull request and a more in-depth example tomorrow.

[balachandarsv]

Thank you @Sintun. Appreciate your response.

[Sintun]

I just created a PR #3330 , my approach circumvents the tesseract iterator infrastructure. This enables the feature, but it is uncertain if it will be merged into master.

The current master with my updated changes can be found at
https://github.com/Sintun/tesseract
There is a demonstation example and example commands on how to compile the demo and how to compile tesseract. But you need to insert the paths that correspond to your setup.

[Shreeshrii]

@Sintun Thanks.

I also suggest to make the demo example as part of tesseract repo.

@zdenop @stweil @egorpugin @amitdo
What is the recommended location for keeping the examples? Should we add a directory in this repo to keep this and other samples of usage of api?

[egorpugin]

/example dir is fine for any examples.
But since we do OCR, possible image/training files can slow down git clones. So, I think separate repo would be better for this.

[Sintun]

I adjusted the demo file License to Apache 2. It can be added to any repo. I was not able to find an /example directory.

[Shreeshrii]

Yes, /example directory does not exist. Currently api examples are there as part of documentation, tessdoc repo (having been transferred from Wiki).

I had added one apiexample to 'test' repo for use in ci.

It would be good to move them all to one place and have option to build them.

[stweil]

I wonder whether it would be even better to add a test case in unittest (which could also serve as an example).

[Shreeshrii]

I wonder whether it would be even better to add a test case in unittest (which could also serve as an example).

That's a good idea!

For starters, though, I suggest that we add /example directory to tesseract repo itself (similar to unittest) and the images needed for running can be from the test repo.

@stweil Does the directory need to be created before @Sintun can add the demonstration example to it?

[egorpugin]

No, I think we should put examples in separate repo (and make it a submodule if you want example dir in main repo).

[stweil]

Then I suggest a new repository tesscontrib (not to be confused with tessdata_contrib) and provide API examples there in a directory examples.

That new repository could include tesseract as a submodule and provide continuous integration tests which check whether all examples work with the latest tagged Tesseract release.

It could also add tesserocr and maybe other third party Tesseract APIs as submodules and test those as well.

[stweil]

Does the directory need to be created before @Sintun can add the demonstration example to it?

No. If new files are added to new directories, git will handle that automatically.

[Shreeshrii]

Then I suggest a new repository tesscontrib (not to be confused with tessdata_contrib) and provide API examples there in a directory examples.

Sorry, I missed this message earlier.

@stweil Can you set this up?

I had extracted the api examples that were in the wiki at https://github.com/tesseract-ocr/tessdoc/tree/master/examples
Those can also be added.

Thanks!

[stweil]

Pull request #3330 has to be reverted because of two severe regressions, so we still need another implementation.

[amitdo]

See PR #3505.

[amitdo]

What code triggered the issues? Does it come from tablerecog_test.cc? Why it was not detected before?

[stweil]

The Python wrapper tesserocr (which is used by OCR-D) has a global TessBaseAPI object which is destroyed at program exit. That interferred with the singleton introduced by PR #3330. The resulting segmentation fault was detected before by people working with OCR-D but not analyzed. Our CI tests don't include a global TessBaseAPI (that should be added, maybe to baseapi_test).

The other regression was an assertion which occurred with some images with tables, see #3330 (comment). tablerecog_test did not detect that issue.

[stweil]

See pull request #3509 which detects one of the two regressions.

[ZivFisher]

Did you solved the issue with tables?