Fix for to_text and to_bytes error handlers

* surrogate_then_strict doesn't exist.  Switch to surrogate_or_strict
  instead.
* Found some bugs in the _text.py implementation
  * The composed error handlers (error handlers which are made up of two
    or more python encoding error handlers) had a wrong string in it,
    'surrogate_or_escape' doesn't exist.  Replaced that with
    'surrogate_or_replace' which is the correct handler name.
  * Left comment about the implicit conditions that are part of the
    surrogate_then_replace code path

Fixes #23865
Fixes #23861
This commit is contained in:
Toshio Kuratomi 2017-07-21 10:23:16 -07:00
commit fc5d71de0d
3 changed files with 16 additions and 13 deletions

View file

@ -44,7 +44,7 @@ except LookupError:
HAS_SURROGATEESCAPE = False
_COMPOSED_ERROR_HANDLERS = frozenset((None, 'surrogate_or_escape',
_COMPOSED_ERROR_HANDLERS = frozenset((None, 'surrogate_or_replace',
'surrogate_or_strict',
'surrogate_then_replace'))
@ -133,6 +133,9 @@ def to_bytes(obj, encoding='utf-8', errors=None, nonstring='simplerepr'):
return obj.encode(encoding, errors)
except UnicodeEncodeError:
if original_errors in (None, 'surrogate_then_replace'):
# We should only reach this if encoding was non-utf8 original_errors was
# surrogate_then_escape and errors was surrogateescape
# Slow but works
return_string = obj.encode('utf-8', 'surrogateescape')
return_string = return_string.decode('utf-8', 'replace')