UTF-8 Mode is Enabled by Default in Python 3.15

UTF-8 Mode is Enabled by Default in Python 3.15

When a text file is opened without specifying an encoding, Python has to choose one. Until now that choice came from the system locale, which means the same code could read and write different bytes on different machines. A file written on a Windows system using a legacy codepage and read on a Linux system using UTF-8 produces mojibake or raises UnicodeDecodeError, even though neither program specified anything.

For example, the encoding chosen for a newly opened file can be inspected directly:

import locale

f = open('data.txt', 'w')

print(f.encoding)  # utf-8
print(locale.getpreferredencoding(False))  # utf-8

f.close()

Before Python 3.15, the value varies by machine. Many Windows installations report a legacy codepage such as cp1252, while a system with a minimal C locale reports ASCII. Nothing in the source indicates which encoding will be used.

Since Python 3.15, UTF-8 mode is enabled by default, so UTF-8 is used for the default encoding of files, standard streams and pipes regardless of the locale.

The previous behaviour remains available. Setting the PYTHONUTF8=0 environment variable or passing the -X utf8=0 command-line option disables UTF-8 mode and restores the locale-based default, which is useful when an existing application depends on it.

Leave a Comment

Cancel reply

Your email address will not be published.