TextCritical.net

• https://textcritical.net/work/berean-study-bible/Acts/6/11
• Surrogates not allowed happens when I use utf-8 when trying to persist to the database
  • https://github.com/mitmproxy/mitmproxy/issues/2075
• I was able to import with cp1252/surrogateescape; this fails in the tests though
• I tried importing via utf-8/surrogateescape; fails on database import which blows up complaining about

• Is there something missing where the chars are?
  • Double quotes
• What char is this?
  • https://www.codetable.net/decimal/147
• What encoding is this file?
  • UTF-8 works with the double-quotes but complains about the start character: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 40: invalid start byte
  • cp1252 cannot load: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 614: character maps to <undefined>
    • https://superuser.com/questions/301552/how-to-auto-detect-text-file-encoding
    • bsb.txt: Windows-1252 with confidence 0.73
  • iso-8859-1 fails to load the double-quotes
  • windows-1252: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 614: character maps to <undefined>
• What is the difference between cp1252 and windows-1252?
  • https://www.i18nqa.com/debug/table-iso8859-1-vs-windows-1252.html
  • This looks like utf-8 according to the table
• Where exactly is the problem in the file with cp1252?
  • That looks good actually
  • Tests fail on it though with surrogateescape. Strange because I imported it this way fine.

• https://docs.python.org/2.4/lib/standard-encodings.html

python3 manage.py import_berean_bible -f /db/bsb.txt