Restructure general workflow, cli, services, processors, docs, tests #40

kba · 2018-03-26T13:05:58Z

This is a rewrite based on our discussions.

All XML access is now in ocrd.model, pythonic API to METS and PAGE.

ocrd.resolver handles interactions with external web resources (such as downloading files), caching and with the data/model repository (the latter NIH yet).

ocrd.workspace represents the "working directory", a folder for the processors to work on, created by the resolver. Provides hooks to the METS file and images/pages contained therein.

ocrd.processor contains Processors, i.e. the actual functionality bundled in a class that is instantiated with a workspace and has a process method to execute that functionality on the workspace. In addition to accessing the files in the METS, parameters can be passed as an object for processors that require information beyond what's in the METS/PAGE XML.

The general workflow is:

resolver = Resolver(...)
workspace = Resolver.create_workspace(mets_url)
processor = FancyProcessorThatDoesGreatThings(workspace)
processor.process()

Reusable functionality moved to a ocrd.utils module (coordinate translation, string normalization, logging etc).

ocrd.cli contains the CLI (installed as ocrd) which has subcommands to process METS directly (process) and start services (server). Commands can be chained, e.g.

ocrd process -m /path/to/mets.xml characterize/exif segment_region/tesseract3 segment_line/tesseract3 recognize/tesseract3

Documentation (apidocs only atm, should move stuff from wiki there once it's agreed upon) can be built with sphinx (see Makefile).

TestCases are there but I couldn't get it running yet on circle ci because of tesseract bindings.

fixes #38
fixes #35
fixes #33
fixes #28
fixes #27
fixes #20
fixes #9
fixes #3
#16

…, rename ocrd_mets_file ocrd_file

…rom_xywh

wrznr

Great piece of work! Many thanks. Only minor changes requested. Mainly, I do not understand the 3 in Tesseract3.

wrznr · 2018-03-26T13:42:48Z

ocrd/processor/segment_region/tesseract3.py

+from ocrd.utils import getLogger, mets_file_id
+from ocrd.constants import MIMETYPE_PAGE
+
+import tesserocr


I do not understand why the class is called Tesseract3.... You are importing tesserocr which in turn uses the tesseract installation provided at your system (which in my case is Tesseract4).

I didn't realize that, because tesserocr bindings wouldn't build for me before with the 4.00 headers. But I realize now it was probably just a mismatch of headers and pkg-config. Sure, let's rename it everywhere tesseract3 -> tesseract (or tesserocr which would be clearer but conflict-prone, global package over local package priority change between 2/3).

BTW: If you have an idea why this won't build on Circle (https://circleci.com/gh/kba/pyocrd/12) or Travis (https://travis-ci.org/OCR-D/pyocrd/jobs/358407881) I'd appreciate it.

Get:7 http://us-central1.gce.archive.ubuntu.com/ubuntu trusty/universe amd64 libtesseract-dev amd64 3.03.02-3 [1,284 kB]

I see the problem now, tesserocr supports only libtesseract >= 3.04 (semver for
tesseract would be a good idea to spot stuff like this).

I'll see how I can get a newer version or a newer OS in the CI container.

wrznr · 2018-03-26T13:44:12Z

ocrd/webservice/processor.py

+        run_processor(ExifProcessor, request.args['mets_url'], resolver)
+        return 'DONE', 200
+
+    @app.route('/processor/segent_line/tesseract3', methods=['PUT'])


Typo: segment

wrznr · 2018-03-26T13:44:32Z

ocrd/webservice/processor.py

+        run_processor(Tesseract3LineSegmenter, request.args['mets_url'], resolver)
+        return 'DONE', 200
+
+    @app.route('/processor/segent_region/tesseract3', methods=['PUT'])


Typo: segment.

cneud

Excellent work! A major step forward in structure and control flow and covering a large amount of functionality. Only thing to keep in mind: while it is good to have something in place and working now for local/remote data management, we should align those parts closely with KIT and the LZA module project to ensure other parts of this implementation remain flexible enough to switch to using these repositories once they become available.

cneud · 2018-03-26T21:10:36Z

.travis.yml

+  - 2.7
+  - 3.6
+before_install:
+  - sudo add-apt-repository ppa:alex-p/tesseract-ocr


Tesseract 4.0 for Ubuntu 18.04 should be coming soon, then we can switch to using apt with no ppa.

See tesseract issue #1423 and recent release of 4.0.0-beta.1

Problem with travis is that you can only choose between 12.04 and 14.04. At least since 16.04 / Debian 7, 3.04 is shipped and tesserocr should build.

Even with the PPA it doesn't build for me since tesseract 4 seems to make use of some c++11 features that the 14.04 gcc version doesn't support.

I'm now using the same pre-install setup that tesserocr uses (download and build leptonica and tesseract). Very inefficient but it worked: https://travis-ci.org/OCR-D/pyocrd/jobs/358810995

cneud · 2018-03-26T21:14:06Z

Makefile

 		tesseract-ocr-eng \
 		tesseract-ocr-deu \
 		tesseract-ocr-deu-frak

 # Install python deps via pip
 deps-pip:
-	pip3 install --user -r requirements.txt
+	$(PIP) install -r requirements.txt

 # Clone the spec dir for sample files
 spec:
 	git clone https://github.com/OCR-D/spec


Just a note that the example files in spec need updating - the mets.xml should be updated to reflect recent discussions and ideally we should pick some sample images that are a) lightweight and b) for which we already have ground truth in ocr-d.de/daten.

Absolutely, tracking in #41

cneud · 2018-03-26T21:20:34Z

ocrd/constants.py

+    'mods': "http://www.loc.gov/mods/v3",
+    'xlink': "http://www.w3.org/1999/xlink",
+    'page': "http://schema.primaresearch.org/PAGE/gts/pagecontent/2017-07-15",
+    'xsl': 'http://www.w3.org/1999/XSL/Transform#',


Support for ALTO was seen as highly useful given current practices in libraries and possible uses in OCR-D post-processing of existing OCR. Current version is 3.1 with 4.0 due for release in April 2018.

Agreed, ALTO should be supported. From a programmer's POV it doesn't seem that hard since most features should be easy to translate between the two, i.e. you could use the same API and change the XML handling in the implementation. The same probably goes for hOCR BTW. But we need to discuss the specifics of how we support this and the repercussions on performance and long-term archival aspects etc.

cneud · 2018-03-26T21:28:07Z

ocrd/resolver.py

+    Handle Uploads, Downloads, Repository access and manage temporary directories
+    Optionally cache files.
+
+    Args:


kba · 2018-03-27T10:44:25Z

Builds now, the last thing I want to get right is fix a few python2-isms that make it break for 3.6 and then merge.

wrznr · 2018-03-27T13:06:18Z

@kba Is the failing Travis check preventing the merge?

kba · 2018-03-27T14:24:52Z

Yes, well not so much the checks but the python3 incompatibility, I'm on it now.

LABEL the image with git rev, build date, maintainer, microbadger bad…

kba added 30 commits March 23, 2018 16:09

Stubs for METS, PAGE, resolver and workspace, pylint, unittests

11bff34

OcrdMetsFile in its own file

abeb696

wip: ResolverCache

f22758b

model.ocrd_page: fix indexing off by one

c54a433

recognize works

1fc1ee7

gitignore libreoffice lock files

01045cf

wip: travis

dbaa95c

typo: segent -> segment

3a05a34

ResolverCache working

e3e46ae

'make test' to run all unit tests

919cec9

add EXIF constants

f3cc1ae

OcrdMets: cache fileGrps

db9eb66

📝 Update README

98ef09a

python3 compat (make PYTHON=python3 test)

f018657

create processor class, port exif to new api, extend page, test files…

18d12ee

…, rename ocrd_mets_file ocrd_file

rename ocrd.log -> ocrd.utils to contain reusable static code

6d81ea5

move to utils, export getLogger, coordinate_string_from_xywh

9641bbd

lazy logging

905aa1e

pylint: stop complaining about lxml

fef9b5c

page tag constants

ff1e92d

OcrdPage: methods for listing/creating regions/lines

62684c1

workspace: + save_mets method

728564e

utils: xywh_from_coordinate_string as opposite of coordinate_string_f…

8ecce18

…rom_xywh

WIP port segmenting to new api

fb09eba

pylint: stop complaining about tesserocr/cv2

cf724a8

MIMETYPE_PAGE = text/page+xml

9dbe659

processor: helpers for input/output of files

cfb536a

OcrdPage: prefer "X is not None" over "not X"

6cbb851

tests: assets module

48b3cee

mirror module structure in tests

b8a942b

kba added 10 commits March 25, 2018 21:18

basic setup for documentation with sphinx

30a97c9

move image manipulation to workspace.resolve_image_as_pil

32c2ce7

resolver: allow setting workspace directory explicitly (for testing)

b58e6f6

use xmllint --format to optionally canonicalize/pretty print XML

fa92fa9

canonical ID for mets:file: fileGrp@USE + 4-zero-padded index within grp

cabb273

page: helpers to work with TextLine

bfdf09a

processor.add_output_file: pass on ID

38c85a7

WIP recognition with tesseract3

9f3c82c

test assets

80d33da

🐛 resolver: use hyper-verbose but uniqe filenames based on url

c8232b7

kba requested review from cneud and wrznr March 26, 2018 13:07

wrznr suggested changes Mar 26, 2018

View reviewed changes

kba force-pushed the wip-api branch from bed3ff9 to dbaa95c Compare March 26, 2018 14:01

kba added 3 commits March 26, 2018 19:01

🎨 remove obsolete pylint exceptions

3693299

rename tesseract3 -> tesserocr

4e0fa6f

make 'test-profile' to list most time-consuming lines

38d29ab

kba mentioned this pull request Mar 26, 2018

Recognition of line images with tesseract #32

Closed

kba added 3 commits March 26, 2018 19:56

workspace: remove hard-coded reference to INPUT fileGrp

f911afa

💚 travis add @alex-p's tesseract-ocr PPA

1a2bd9b

properly skip recognize test, travis

daade7f

wrznr approved these changes Mar 26, 2018

View reviewed changes

cneud approved these changes Mar 26, 2018

View reviewed changes

kba mentioned this pull request Mar 27, 2018

Testing: More varied and feasible test data #41

Closed

kba merged commit 343ceae into OCR-D:master Mar 27, 2018

kba deleted the wip-api branch March 27, 2018 19:06

bertsky pushed a commit to bertsky/core that referenced this pull request Apr 26, 2023

Merge pull request OCR-D#40 from kba/label-images

f5fc4bf

LABEL the image with git rev, build date, maintainer, microbadger bad…

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restructure general workflow, cli, services, processors, docs, tests #40

Restructure general workflow, cli, services, processors, docs, tests #40

kba commented Mar 26, 2018 •

edited

Loading

wrznr left a comment

wrznr Mar 26, 2018

kba Mar 26, 2018

kba Mar 26, 2018

kba Mar 26, 2018

wrznr Mar 26, 2018

wrznr Mar 26, 2018

cneud left a comment

cneud Mar 26, 2018

kba Mar 27, 2018

cneud Mar 26, 2018

kba Mar 27, 2018

cneud Mar 26, 2018

kba Mar 27, 2018

cneud Mar 26, 2018

kba commented Mar 27, 2018

wrznr commented Mar 27, 2018

kba commented Mar 27, 2018

Restructure general workflow, cli, services, processors, docs, tests #40

Restructure general workflow, cli, services, processors, docs, tests #40

Conversation

kba commented Mar 26, 2018 • edited Loading

wrznr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cneud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kba commented Mar 27, 2018

wrznr commented Mar 27, 2018

kba commented Mar 27, 2018

kba commented Mar 26, 2018 •

edited

Loading