In Python3, the following pattern becomes common:

        with open(fullname, 'rb') as fp:
            coding, line = tokenize.detect_encoding(fp.readline)
        with open(fullname, 'r', encoding=coding) as fp:
            ...

It opens the file is opened twice, whereas it is unnecessary: it's possible to reuse the raw buffer to create a text file. And I don't like the detect_encoding() API: pass the readline function is not intuitive.

I propose to create tokenize.open_python() function with a very simple API: just one argument, the filename. This function calls detect_encoding() and only open the file once.

Attached python adds the function with an unit test and a patch on the documentation. It patchs also functions currently using detect_encoding().

open_python() only supports read mode. I suppose that it is enough.

Looks good.

In the test, do you have to remove the TESTFN file manually?  Isn’t that handled by unittest or regrtest?

I think the name of the function could be better but I don’t have a proposal.

> In the test, do you have to remove the TESTFN file manually?  Isn’t that
> handled by unittest or regrtest?

The test uses TESTFN+'.py', not TESTFN ;-)

> I think the name of the function could be better but I don’t have a
> proposal.

Do you prefer a name like tokenize.open()?

> The test uses TESTFN+'.py', not TESTFN ;-)
Ah, right.  You can use addCleanup then.

> Do you prefer a name like tokenize.open()?
Hm, tokenize being “Tokenization help for Python programs”, I think this name is rather good.  Or tokenize.open_source.

+1 on adding the function.

Note that it is useful for opening any text file with an encoding cookie, not only python source code, so "tokenize.open()" sounds attractive.

Once we are at it, I would like to factor out and document code that extracts the cookie from a decoded string.  This functionality is useful for tools that produce annotated source code and want to preserve source encoding in their output.  See issue10329.

Also, to answer Victor's comment on issue10329, msg120601, while using this function simplifies dealing with source files in trace module, it does not solve all the issues: source file will still be opened separately in linecache.gettines() and find_executable_linenos() and  find_executable_linenos() still does not respect __loader__ interface.  See issue 10342.

> Note that it is useful for opening any text file with an encoding
> cookie, not only python source code,
This is cool.  doctest could benefit from that, I think.

> so "tokenize.open()" sounds attractive.
Agreed, even though the docstring of tokenize does talk about Python code.

On Sat, Nov 6, 2010 at 12:15 PM, Éric Araujo <report@bugs.python.org> wrote:
..
>> so "tokenize.open()" sounds attractive.
> Agreed, even though the docstring of tokenize does talk about Python code.

I still like "tokenize.open()" more, but for the sake of completeness
of discussion, what about open(.., encoding="fromcookie")?  (I know,
tokenize.detect_encoding() also looks for an utf-8 BOM, so
"fromcookie" can be improved.)

I don’t know if it would be okay to depend on tokenize in io.  Anyway, I tend to prefer tokenize.open over builtins.open(..., encoding='fromcookie').

+1 for the feature.  However, tokenize.open() sounds a bit unspecific.  But I don't really have better suggestions; open_cookie() is wrong too, since it doesn't open cookies :)

> what about open(.., encoding="fromcookie")?

Please don't do that. It remembers me the magical str.encode() method of 
Python2. I prefer simple API with limited features: if you want to open a 
Python script, use tokenize.open(), if you want to open an XML document use 
any XML library, etc. It remembers me also the discussion about detecting BOM 
in text files: issue #7651. There was no consensus, and so the issue is still 
open.

For this particular issue, I prefer a specific function tokenize.<choose a 
function name>.

On Saturday 06 November 2010 17:00:15 you wrote:
> Note that it is useful for opening any text file with an encoding cookie,
> not only python source code, so "tokenize.open()" sounds attractive.

Ok, the new patch (tokenize_open-2.patch) uses tokenize.open() name and adds a 
test for BOM without coding cookie (test utf-8-sig encoding).

On Sun, Nov 7, 2010 at 4:24 AM, STINNER Victor <report@bugs.python.org> wrote:
..
> Ok, the new patch (tokenize_open-2.patch) uses tokenize.open() name and adds a
> test for BOM without coding cookie (test utf-8-sig encoding).

Here are my comments on the new patch:

1. A nit-pick: tokenize.py is already inconsistent in this respect,
but I believe it is conventional not to start a docsting with an empty
line:

+def open(filename):
+    """Open a Python script in read mode with the right encoding.

instead of

+def open(filename):
+    """
+    Open a Python script in read mode with the right encoding.

Also, I would prefer a more neutral description that will not imply
that the method is only useful for Python source code.  For example:

"""Detect encoding and open a file in read mode with the right encoding.

(optionally a more detailed explanation or a reference to detect_encoding.)
"""

2. Another nit-pick: detect_encoding() returns a list of 0-2 lines in
the second item, not a single line, so it is better to write

+    encoding, lines = detect_encoding(buffer.readline)

or even

+    encoding, _ = detect_encoding(buffer.readline)

instead of

+    encoding, line = detect_encoding(buffer.readline)

3. In test_open(self), I think you should use self.assertEqual()
instead of a plain assert.

4. Last nit-pick (feel free to ignore as prejudice to %-formatting :-):

instead of

+                print("# coding: %s" % encoding, file=fp)

you can write

+                print("# coding: " + encoding, file=fp)

Changing the title to make the latest choice of function name more visible.

1. Agreed on both.  Extraneous blank lines seem to be favored by people who need to work around bugs in their editor.

2. Yes, “_” is recommended for a discarded object.

3. Agreed.

4. Even better: print('# coding:', encoding, file=fp).  No intermediary string!

Commited to Python 3.2 (r86346 + r86347). Thanks for your reviews.