Unexpected UTF-8 problems

[mikegerber]

This was on a Linux system, and the "A~-" was an "Ö".

• Fix Ã. problem above
• Fix LookupError: unknown encoding: EUC-TW problem

For plain text files it would be best to

• Review CLI
  • cli.py (esp. process_dir)
  • ocrd_cli.py - any plain text files supported here?
  • cli_line_dirs.py
  • cli_summarize.py?
• add --plain-encoding option so users have the chance to give it manually
• Fall back to detecting, but over all files
• while warning about the auto detecting
• falling back to UTF-8 if the detected charset is way out there/unknown like EUC-TW
• What about the BOM now?

[mikegerber]

Happens with our merged test directory.

[mikegerber]

Another one, this time with test (current dataset):

(dinglehopper) mike.gerber@lx0246:~$ sh /data-ssd/mike.gerber/dta-gt-data/test-eval.sh
Traceback (most recent call last):
  File "/home/mike.gerber/.pyenv/versions/dinglehopper/bin/dinglehopper-line-dirs", line 8, in <module>
    sys.exit(main())
  File "/home/mike.gerber/.pyenv/versions/3.9.20/envs/dinglehopper/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/mike.gerber/.pyenv/versions/3.9.20/envs/dinglehopper/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/mike.gerber/.pyenv/versions/3.9.20/envs/dinglehopper/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mike.gerber/.pyenv/versions/3.9.20/envs/dinglehopper/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/mike.gerber/devel/dinglehopper/src/dinglehopper/cli_line_dirs.py", line 232, in main
    process(
  File "/home/mike.gerber/devel/dinglehopper/src/dinglehopper/cli_line_dirs.py", line 128, in process
    gt_text = plain_extract(gt_fn, include_filename_in_id=True)
  File "/home/mike.gerber/devel/dinglehopper/src/dinglehopper/ocr_files.py", line 167, in plain_extract
    with open(filename, "r", encoding=fileencoding) as f:
LookupError: unknown encoding: EUC-TW

[mikegerber]

chardet seems to be bad at dealing with these short tests:

In [2]: print(chardet.detect("Nur zum Prüfen von 'chardet'.".encode("utf-8")))
{'encoding': 'ISO-8859-9', 'confidence': 0.6587004243912733, 'language': 'Turkish'}

For plain text files it would be best to

• add --plain-encoding option so users have the chance to give it manually
• Fall back to detecting, but over all files
• while warning about the auto detecting
• falling back to UTF-8 if the detected charset is way out there/unknown like EUC-TW

[mikegerber]

Branch now has --plain-encoding and warns about auto-detecting (for dinglehopper-line-dirs)

[mikegerber]

This probably gives up problems with the UTF-8 BOM again, need to check.

[mikegerber]

We also need to review the CLIs again, I don't even remember we had an option to process directories (!= directories of lines)...

[mikegerber]

Note: working in the feat/flex-line-dirs branch on this, because 1. it came up there 2. the line dirs are especially affected because short texts are the input format there.

[mikegerber]

• cli.py now has a --plain-encoding option.
• cli_summarize.py only works with reports and is not affected.
• ocrd_cli.py also supports plain text files, so the OCR-D processor has a plain_encoding parameter now.