Title: Python3: guess text file charset using the BOM
Type: enhancement Stage:
Components: Unicode Versions: Python 3.4
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: lukasz.langa Nosy List: doerwalter, eric.araujo, flox, jdufresne, lukasz.langa, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2010-01-07 03:03 by vstinner, last changed 2015-01-08 00:51 by jdufresne. This issue is now closed.

File name Uploaded Description Edit
open_bom.patch vstinner, 2010-01-07 23:18 review
open_bom-2.patch vstinner, 2010-01-07 23:41 review
open_bom-3.patch vstinner, 2010-01-08 10:23 review
Messages (12)
msg97341 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-01-07 03:03
If the file starts with a BOM, open(filename) should be able to guess the charset. It would be helpful for many high level modules:

 - #7519: ConfigParser
 - #7185: csv
 - and any module using open() to read a text file

Actually, the user have to choose between UTF-8 and UTF-8-SIG to skip the UTF-8 BOM. For UTF-16, the user have to specify UTF-16-LE or UTF-16-BE, even if the file starts with a BOM (which should be the case most the time).

The idea is to delay the creation of the decoder and the encoder. Just after reading the first chunk: try to guess the charset by searching for a BOM (if the charset is unknown). If the BOM is found, fallback to current guess code (os.device_charset() or locale.getpreferredencoding()).

Concerned charsets: UTF-8, UTF-16-LE, UTF-16-BE, UTF-32-LE, UTF-32-BE. Binary files are not concerned. If the encoding is specified to open(), the behaviour is unchanged.

I wrote a proof of concept, but there are still open issues:

 - append mode: should we seek at zero to read the BOM?
   old=tell(); seek(0); bytes=read(4); seek(old); search_bom(bytes)
 - read+write: should we guess the charset using the BOM if the first action is a write? or only search for a BOM if the first action is a read?
msg97366 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-01-07 19:49
You should ask on the mailing-list (python-dev) because this is an important behaviour change which I'm not sure will get accepted.
msg97386 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-01-07 23:18
open_bom.patch is the proof of concept. It only works in read mode. The idea is to delay the creation of the encoding and the decoder. We wait for just after the first read_chunk().

The patch changes the default behaviour of open(): if the file starts with a BOM, the BOM is used but skipped. Example:
from _pyio import open

with open('test.txt', 'w', encoding='utf-8-sig') as fp:
    print("abc", file=fp)
    print("d\xe9f", file=fp)

with open('test.txt', 'r') as fp:
    print("open().read(): {!r}".format(

Unpatched Python displays '\ufeffabc\ndéf\n', whereas patched Python displays 'abc\ndéf\n'.
msg97389 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-01-07 23:41
Oops, fix read() method of my previous patch.
msg97406 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-01-08 10:23
New version of the patch which is shorter, cleaner, fix the last bug (seek) and don't change the default behaviour anymore (checking for BOM is now explicit):
 * BOM checking is now optional (explicit): use open(filename, encoding="BOM"). open(filename, "w", encoding="BOM") raises a ValueError.
 * Create a BOMS dictionary directly in the codecs module
 * Fix TextIOWrapper for seek(0) (add _has_bom attribute)
 * Add an unit test for read() and readlines()
 * Read the encoding property before the first read gives None

I also removed the _get_encoding() method (hack).
msg97455 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2010-01-09 10:45
IMHO this is the wrong approach.

As Martin v. Löwis suggested here the best solution would be a new codec (which he named sniff), that autodetects the encoding on reading. This doesn't require *any* changes to the IO library. It could even be developed as a standalone project and published in the Cheeseshop.
msg103082 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-04-13 20:12
The link has gone.  Is this the message you’re refering to?

msg103084 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2010-04-13 20:25
Yes, that's the posting I was referring to. I wonder why the link is gone.
msg111416 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2010-07-24 02:15
I agree with MvL that this is a broader issue that shouldn't be patched in user code (e.g. #7519) but on the codec level. The sniff codec idea seems neat.
msg164853 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2012-07-07 14:08
After reading the mailing list thread at

and waging on other concerns (e.g. how to behave on write-only and read-write modes), it looks like a PEP might be necessary to solve this once and for all.
msg164854 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2012-07-07 14:13
For the implementation part, there's something which already plays with the BOM in the tokenize module.

See, which uses tokenize.detect_encoding() to read the BOM in some cases.
msg178876 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-01-03 01:24
The idea was somehow rejected on the python-dev mailing list. I'm not really motivated to work on this issue since I never see any file starting with a BOM on Linux, and I'm only working on Linux. So I just close this issue.

If someone is motivated to work on this topic, I suppose that it would be better to reopen the discussion on the python-dev mailing list first.
Date User Action Args
2015-01-08 00:51:26jdufresnesetnosy: + jdufresne
2015-01-06 19:52:05r.david.murraylinkissue23178 superseder
2013-01-03 01:24:56vstinnersetstatus: open -> closed
resolution: rejected
messages: + msg178876
2012-07-07 14:13:21floxsetnosy: + flox
messages: + msg164854
2012-07-07 14:08:50lukasz.langasettype: enhancement
2012-07-07 14:08:34lukasz.langasetmessages: + msg164853
versions: + Python 3.4, - Python 2.7, Python 3.2
2012-03-20 12:33:40lukasz.langasetassignee: lukasz.langa
2012-03-20 12:32:44lukasz.langalinkissue14311 superseder
2010-07-25 09:09:02BreamoreBoylinkissue7519 superseder
2010-07-24 02:15:01lukasz.langasetnosy: + lukasz.langa
messages: + msg111416
2010-04-13 20:25:47doerwaltersetmessages: + msg103084
2010-04-13 20:12:43eric.araujosetnosy: + eric.araujo
messages: + msg103082
2010-01-09 10:45:56doerwaltersetnosy: + doerwalter
messages: + msg97455
2010-01-08 10:23:46vstinnersetfiles: + open_bom-3.patch

messages: + msg97406
2010-01-07 23:41:19vstinnersetfiles: + open_bom-2.patch

messages: + msg97389
2010-01-07 23:18:49vstinnersetfiles: + open_bom.patch
keywords: + patch
messages: + msg97386
2010-01-07 19:49:18pitrousetnosy: + pitrou
messages: + msg97366
2010-01-07 03:03:55vstinnercreate