Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(16057)

Side by Side Diff: Doc/library/compileall.rst

Issue 16104: Use multiprocessing in compileall script
Patch Set: Created 5 years, 4 months ago
Left:
Right:
Use n/p to move between diff chunks; N/P to move between comments. Please Sign in to add in-line comments.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | Lib/compileall.py » ('j') | Lib/compileall.py » ('J')
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 :mod:`compileall` --- Byte-compile Python libraries 1 :mod:`compileall` --- Byte-compile Python libraries
2 =================================================== 2 ===================================================
3 3
4 .. module:: compileall 4 .. module:: compileall
5 :synopsis: Tools for byte-compiling all Python source files in a directory tr ee. 5 :synopsis: Tools for byte-compiling all Python source files in a directory tr ee.
6 6
7 7
8 This module provides some utility functions to support installing Python 8 This module provides some utility functions to support installing Python
9 libraries. These functions compile Python source files in a directory tree. 9 libraries. These functions compile Python source files in a directory tree.
10 This module can be used to create the cached byte-code files at library 10 This module can be used to create the cached byte-code files at library
(...skipping 47 matching lines...) Expand 10 before | Expand all | Expand 10 after
58 Read the file ``list`` and add each line that it contains to the list of 58 Read the file ``list`` and add each line that it contains to the list of
59 files and directories to compile. If ``list`` is ``-``, read lines from 59 files and directories to compile. If ``list`` is ``-``, read lines from
60 ``stdin``. 60 ``stdin``.
61 61
62 .. cmdoption:: -b 62 .. cmdoption:: -b
63 63
64 Write the byte-code files to their legacy locations and names, which may 64 Write the byte-code files to their legacy locations and names, which may
65 overwrite byte-code files created by another version of Python. The default 65 overwrite byte-code files created by another version of Python. The default
66 is to write files to their :pep:`3147` locations and names, which allows 66 is to write files to their :pep:`3147` locations and names, which allows
67 byte-code files from multiple versions of Python to coexist. 67 byte-code files from multiple versions of Python to coexist.
68
69 .. cmdoption:: -j N
70
71 Use *N* processes to compile the given directory.
Charles-François Natali 2014/04/27 16:28:46 The wording is strange, maybe "compile the files w
Claudiu.Popa 2014/04/27 18:13:36 Done.
72 If ``0`` is used, then the result of :func:`os.cpu_count()`
73 will be used.
68 74
69 .. versionchanged:: 3.2 75 .. versionchanged:: 3.2
70 Added the ``-i``, ``-b`` and ``-h`` options. 76 Added the ``-i``, ``-b`` and ``-h`` options.
71 77
78 .. versionchanged:: 3.5
79 Added the ``-j`` option.
80
72 There is no command-line option to control the optimization level used by the 81 There is no command-line option to control the optimization level used by the
73 :func:`compile` function, because the Python interpreter itself already 82 :func:`compile` function, because the Python interpreter itself already
74 provides the option: :program:`python -O -m compileall`. 83 provides the option: :program:`python -O -m compileall`.
75 84
76 Public functions 85 Public functions
77 ---------------- 86 ----------------
78 87
79 .. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, qu iet=False, legacy=False, optimize=-1) 88 .. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, qu iet=False, legacy=False, optimize=-1, processes=None)
80 89
81 Recursively descend the directory tree named by *dir*, compiling all :file:`. py` 90 Recursively descend the directory tree named by *dir*, compiling all :file:`. py`
82 files along the way. 91 files along the way.
83 92
84 The *maxlevels* parameter is used to limit the depth of the recursion; it 93 The *maxlevels* parameter is used to limit the depth of the recursion; it
85 defaults to ``10``. 94 defaults to ``10``.
86 95
87 If *ddir* is given, it is prepended to the path to each file being compiled 96 If *ddir* is given, it is prepended to the path to each file being compiled
88 for use in compilation time tracebacks, and is also compiled in to the 97 for use in compilation time tracebacks, and is also compiled in to the
89 byte-code file, where it will be used in tracebacks and other messages in 98 byte-code file, where it will be used in tracebacks and other messages in
(...skipping 11 matching lines...) Expand all
101 occur. 110 occur.
102 111
103 If *legacy* is true, byte-code files are written to their legacy locations 112 If *legacy* is true, byte-code files are written to their legacy locations
104 and names, which may overwrite byte-code files created by another version of 113 and names, which may overwrite byte-code files created by another version of
105 Python. The default is to write files to their :pep:`3147` locations and 114 Python. The default is to write files to their :pep:`3147` locations and
106 names, which allows byte-code files from multiple versions of Python to 115 names, which allows byte-code files from multiple versions of Python to
107 coexist. 116 coexist.
108 117
109 *optimize* specifies the optimization level for the compiler. It is passed t o 118 *optimize* specifies the optimization level for the compiler. It is passed t o
110 the built-in :func:`compile` function. 119 the built-in :func:`compile` function.
120
121 The argument *processes* specifies how many workers are used to
Charles-François Natali 2014/04/27 16:28:46 It's really nit-picking, but why aren't you lines
Claudiu.Popa 2014/04/27 18:13:36 Habit. ;)
122 compile files in parallel. The default is to not use multiple workers.
123 If :mod:`multiprocessing` is unavailable, then a :exc:`ValueError`
Charles-François Natali 2014/04/27 16:28:46 I think it should default to one, not None. If it
Claudiu.Popa 2014/04/27 18:13:36 Mm, I think that it's not good to default to one,
124 will be raised in case that *processes* is given.
125 If *processes* is lower or equal to ``0``, a :exc:`ValueError`
Charles-François Natali 2014/04/27 16:28:46 So there's no easy way to use all available cores
Claudiu.Popa 2014/04/27 18:13:36 Only with the cli interface. The users could call
126 will be raised.
111 127
112 .. versionchanged:: 3.2 128 .. versionchanged:: 3.2
113 Added the *legacy* and *optimize* parameter. 129 Added the *legacy* and *optimize* parameter.
114 130
131 .. versionchanged:: 3.5
132 Added the *processes* parameter.
133
115 134
116 .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=Fals e, legacy=False, optimize=-1) 135 .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=Fals e, legacy=False, optimize=-1)
117 136
118 Compile the file with path *fullname*. 137 Compile the file with path *fullname*.
119 138
120 If *ddir* is given, it is prepended to the path to the file being compiled 139 If *ddir* is given, it is prepended to the path to the file being compiled
121 for use in compilation time tracebacks, and is also compiled in to the 140 for use in compilation time tracebacks, and is also compiled in to the
122 byte-code file, where it will be used in tracebacks and other messages in 141 byte-code file, where it will be used in tracebacks and other messages in
123 cases where the source file does not exist at the time the byte-code file is 142 cases where the source file does not exist at the time the byte-code file is
124 executed. 143 executed.
(...skipping 38 matching lines...) Expand 10 before | Expand all | Expand 10 after
163 182
164 # Perform same compilation, excluding files in .svn directories. 183 # Perform same compilation, excluding files in .svn directories.
165 import re 184 import re
166 compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True) 185 compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True)
167 186
168 187
169 .. seealso:: 188 .. seealso::
170 189
171 Module :mod:`py_compile` 190 Module :mod:`py_compile`
172 Byte-compile a single source file. 191 Byte-compile a single source file.
OLDNEW
« no previous file with comments | « no previous file | Lib/compileall.py » ('j') | Lib/compileall.py » ('J')

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+