Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unexpected UTF-8 problems #123

Open
1 of 11 tasks
mikegerber opened this issue Dec 11, 2024 · 7 comments
Open
1 of 11 tasks

Unexpected UTF-8 problems #123

mikegerber opened this issue Dec 11, 2024 · 7 comments
Assignees
@mikegerber
Labels
bug Something isn't working

Comments

@mikegerber
Copy link
Member

mikegerber commented Dec 11, 2024 •
edited
Loading

image

This was on a Linux system, and the "A~-" was an "Ö".

  • Fix Ã. problem above
  • Fix LookupError: unknown encoding: EUC-TW problem

For plain text files it would be best to

  • Review CLI
    • cli.py (esp. process_dir)
    • ocrd_cli.py - any plain text files supported here?
    • cli_line_dirs.py
  • add --plain-encoding option so users have the chance to give it manually
  • Fall back to detecting, but over all files
  • while warning about the auto detecting
  • falling back to UTF-8 if the detected charset is way out there/unknown like EUC-TW
  • What about the BOM now?
@mikegerber mikegerber self-assigned this Dec 12, 2024
@mikegerber
Copy link
Member Author

Happens with our merged test directory.

@mikegerber
Copy link
Member Author

Another one, this time with test (current dataset):

(dinglehopper) mike.gerber@lx0246:~$ sh /data-ssd/mike.gerber/dta-gt-data/test-eval.sh
Traceback (most recent call last):
  File "/home/mike.gerber/.pyenv/versions/dinglehopper/bin/dinglehopper-line-dirs", line 8, in <module>
    sys.exit(main())
  File "/home/mike.gerber/.pyenv/versions/3.9.20/envs/dinglehopper/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/mike.gerber/.pyenv/versions/3.9.20/envs/dinglehopper/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/mike.gerber/.pyenv/versions/3.9.20/envs/dinglehopper/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mike.gerber/.pyenv/versions/3.9.20/envs/dinglehopper/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/mike.gerber/devel/dinglehopper/src/dinglehopper/cli_line_dirs.py", line 232, in main
    process(
  File "/home/mike.gerber/devel/dinglehopper/src/dinglehopper/cli_line_dirs.py", line 128, in process
    gt_text = plain_extract(gt_fn, include_filename_in_id=True)
  File "/home/mike.gerber/devel/dinglehopper/src/dinglehopper/ocr_files.py", line 167, in plain_extract
    with open(filename, "r", encoding=fileencoding) as f:
LookupError: unknown encoding: EUC-TW

@mikegerber mikegerber mentioned this issue Dec 18, 2024
Open
4 tasks
@mikegerber mikegerber added the bug Something isn't working label Dec 18, 2024
@mikegerber
Copy link
Member Author

chardet seems to be bad at dealing with these short tests:

In [2]: print(chardet.detect("Nur zum Prüfen von 'chardet'.".encode("utf-8")))
{'encoding': 'ISO-8859-9', 'confidence': 0.6587004243912733, 'language': 'Turkish'}

For plain text files it would be best to

  • add --plain-encoding option so users have the chance to give it manually
  • Fall back to detecting, but over all files
  • while warning about the auto detecting
  • falling back to UTF-8 if the detected charset is way out there/unknown like EUC-TW

@mikegerber
Copy link
Member Author

Branch now has --plain-encoding and warns about auto-detecting (for dinglehopper-line-dirs)

image

@mikegerber
Copy link
Member Author

This probably gives up problems with the UTF-8 BOM again, need to check.

@mikegerber
Copy link
Member Author

We also need to review the CLIs again, I don't even remember we had an option to process directories (!= directories of lines)...

@mikegerber
Copy link
Member Author

Note: working in the feat/flex-line-dirs branch on this, because 1. it came up there 2. the line dirs are especially affected because short texts are the input format there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mikegerber mikegerber

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mikegerber