classification
Title: improve linecache: reuse tokenize.detect_encoding() and io.open()
Type: Stage: patch review
Components: Library (Lib) Versions: Python 3.1
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, benjamin.peterson, haypo
Priority: normal Keywords: patch

Created on 2008-10-02 14:42 by haypo, last changed 2009-03-24 22:30 by benjamin.peterson. This issue is now closed.

Files
File name Uploaded Description Edit
linecache_refactor-2.patch haypo, 2008-12-12 18:44 Reuse existing code in linecache.updatecache()
Messages (11)
msg74165 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2008-10-02 14:42
linecache uses it own code to detect a Python script encoding whereas 
a function tokenize.detect_encoding() already exists. It does also 
convert bytes => unicode conversion of the file lines whereas open() 
already supports this feature.
msg74206 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2008-10-02 23:01
I wrote a different (and better) patch for tokenize module: moved to 
the issue4021.
msg74218 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-10-02 23:50
This patch looks good.
msg77643 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2008-12-11 23:21
"This patch looks good." ok and then?
msg77653 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-12-12 01:33
Applied in r67713.
msg77659 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-12-12 02:01
I had to revert this because tokenize imports itertools. This is a
problem when building (setup.py) because itertools doesn't exist, yet. I
also think we should consider hard adding more modules that are loaded
at startup time to py3k already huge list.
msg77684 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2008-12-12 18:44
It was easy to remove the itertools dependency: only itertools.chain() 
was really used and a simple "for ...: yield" is enough to get the 
same behaviour (test_tokenize.py runs fine). With the new version of 
the patch, the bootstrap works correctly.
msg78093 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2008-12-20 11:19
> I also think we should consider hard adding more modules 
> that are loaded at startup time to py3k already huge list.

My patch uses tokenize modules in setup.py bootstrap. But it doesn't 
affect Python classic usage "python" or "python myscript.py".
msg83839 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2009-03-20 01:17
@benjamin.peterson: The second version of my patch works correctly 
with the bootstraping.

> I also think we should consider hard adding more modules 
> that are loaded at startup time to py3k already huge list.

linecache is not loaded at startup time in py3k! I see that linecache 
is loaded by the warnings module, but the warnings module 
(Lib/warnings.py) is not loaded at startup. It was maybe the case with 
Python 2.x or older version of Python 3.x?

With my patch, loading linecache loads 2 extra modules: tokenize and 
token. It only impacts code using directly linecache or the warnings 
module.
msg83840 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2009-03-20 01:19
Oh, I see that setup.py uses warnings and so linecache is loaded by 
setup.py.
msg84118 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-03-24 22:30
Applied in r70587.
History
Date User Action Args
2009-03-24 22:30:33benjamin.petersonsetstatus: open -> closed
resolution: accepted
messages: + msg84118
2009-03-20 01:19:37hayposetmessages: + msg83840
2009-03-20 01:17:22hayposetmessages: + msg83839
2008-12-20 11:19:10hayposetmessages: + msg78093
2008-12-13 21:11:12pitrousetpriority: normal
stage: patch review
versions: + Python 3.1, - Python 3.0
2008-12-12 18:44:25hayposetfiles: - linecache_refactor.patch
2008-12-12 18:44:21hayposetfiles: + linecache_refactor-2.patch
messages: + msg77684
2008-12-12 02:01:55benjamin.petersonsetstatus: closed -> open
resolution: accepted -> (no value)
messages: + msg77659
2008-12-12 01:33:52benjamin.petersonsetstatus: open -> closed
resolution: accepted
messages: + msg77653
nosy: + benjamin.peterson
2008-12-11 23:21:56hayposetmessages: + msg77643
2008-10-02 23:50:51amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg74218
2008-10-02 23:01:27hayposetfiles: - tokenize_bom_utf8.patch
2008-10-02 23:01:23hayposetmessages: + msg74206
2008-10-02 14:43:13hayposetfiles: + tokenize_bom_utf8.patch
2008-10-02 14:42:03haypocreate