Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(46057)

Delta Between Two Patch Sets: Doc/library/compileall.rst

Issue 16104: Use multiprocessing in compileall script
Left Patch Set: Created 5 years, 6 months ago
Right Patch Set: Created 5 years ago
Left:
Right:
Use n/p to move between diff chunks; N/P to move between comments. Please Sign in to add in-line comments.
Jump to:
Left: Side by side diff | Download
Right: Side by side diff | Download
« no previous file with change/comment | « no previous file | Lib/compileall.py » ('j') | Lib/compileall.py » ('J')
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
LEFTRIGHT
1 :mod:`compileall` --- Byte-compile Python libraries 1 :mod:`compileall` --- Byte-compile Python libraries
2 =================================================== 2 ===================================================
3 3
4 .. module:: compileall 4 .. module:: compileall
5 :synopsis: Tools for byte-compiling all Python source files in a directory tr ee. 5 :synopsis: Tools for byte-compiling all Python source files in a directory tr ee.
6 6
7 7
8 This module provides some utility functions to support installing Python 8 This module provides some utility functions to support installing Python
9 libraries. These functions compile Python source files in a directory tree. 9 libraries. These functions compile Python source files in a directory tree.
10 This module can be used to create the cached byte-code files at library 10 This module can be used to create the cached byte-code files at library
11 installation time, which makes them available for use even by users who don't 11 installation time, which makes them available for use even by users who don't
12 have write permission to the library directories. 12 have write permission to the library directories.
13 13
14 14
15 Command-line use 15 Command-line use
16 ---------------- 16 ----------------
17 17
18 This module can work as a script (using :program:`python -m compileall`) to 18 This module can work as a script (using :program:`python -m compileall`) to
19 compile Python sources. 19 compile Python sources.
20 20
21 .. program:: compileall 21 .. program:: compileall
22 22
23 .. cmdoption:: [directory|file]... 23 .. cmdoption:: directory ...
24 file ...
24 25
25 Positional arguments are files to compile or directories that contain 26 Positional arguments are files to compile or directories that contain
26 source files, traversed recursively. If no argument is given, behave as if 27 source files, traversed recursively. If no argument is given, behave as if
27 the command line was ``-l <directories from sys.path>``. 28 the command line was ``-l <directories from sys.path>``.
28 29
29 .. cmdoption:: -l 30 .. cmdoption:: -l
30 31
31 Do not recurse into subdirectories, only compile source code files directly 32 Do not recurse into subdirectories, only compile source code files directly
32 contained in the named or implied directories. 33 contained in the named or implied directories.
33 34
(...skipping 24 matching lines...) Expand all
58 files and directories to compile. If ``list`` is ``-``, read lines from 59 files and directories to compile. If ``list`` is ``-``, read lines from
59 ``stdin``. 60 ``stdin``.
60 61
61 .. cmdoption:: -b 62 .. cmdoption:: -b
62 63
63 Write the byte-code files to their legacy locations and names, which may 64 Write the byte-code files to their legacy locations and names, which may
64 overwrite byte-code files created by another version of Python. The default 65 overwrite byte-code files created by another version of Python. The default
65 is to write files to their :pep:`3147` locations and names, which allows 66 is to write files to their :pep:`3147` locations and names, which allows
66 byte-code files from multiple versions of Python to coexist. 67 byte-code files from multiple versions of Python to coexist.
67 68
69 .. cmdoption:: -r
70
71 Control the maximum recursion level for subdirectories.
72 If this is given, then ``-l`` option will not be taken into account.
73 :program:`python -m compileall <directory> -r 0` is equivalent to
74 :program:`python -m compileall <directory> -l`.
75
68 .. cmdoption:: -j N 76 .. cmdoption:: -j N
69 77
70 Use multiple *N* processes to compile the given directory. 78 Use *N* workers to compile the files within the given directory.
79 If ``0`` is used, then the result of :func:`os.cpu_count()`
80 will be used.
71 81
72 .. versionchanged:: 3.2 82 .. versionchanged:: 3.2
73 Added the ``-i``, ``-b`` and ``-h`` options. 83 Added the ``-i``, ``-b`` and ``-h`` options.
74 84
75 .. versionchanged:: 3.5 85 .. versionchanged:: 3.5
76 Added the ``-j`` option. 86 Added the ``-j`` and ``-r`` options.
87
77 88
78 There is no command-line option to control the optimization level used by the 89 There is no command-line option to control the optimization level used by the
79 :func:`compile` function, because the Python interpreter itself already 90 :func:`compile` function, because the Python interpreter itself already
80 provides the option: :program:`python -O -m compileall`. 91 provides the option: :program:`python -O -m compileall`.
81 92
82 Public functions 93 Public functions
83 ---------------- 94 ----------------
84 95
85 .. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, qu iet=False, legacy=False, optimize=-1, processes=None) 96 .. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, qu iet=False, legacy=False, optimize=-1, workers=1)
86 97
87 Recursively descend the directory tree named by *dir*, compiling all :file:`. py` 98 Recursively descend the directory tree named by *dir*, compiling all :file:`. py`
88 files along the way. 99 files along the way.
89 100
90 The *maxlevels* parameter is used to limit the depth of the recursion; it 101 The *maxlevels* parameter is used to limit the depth of the recursion; it
91 defaults to ``10``. 102 defaults to ``10``.
92 103
93 If *ddir* is given, it is prepended to the path to each file being compiled 104 If *ddir* is given, it is prepended to the path to each file being compiled
94 for use in compilation time tracebacks, and is also compiled in to the 105 for use in compilation time tracebacks, and is also compiled in to the
95 byte-code file, where it will be used in tracebacks and other messages in 106 byte-code file, where it will be used in tracebacks and other messages in
(...skipping 12 matching lines...) Expand all
108 119
109 If *legacy* is true, byte-code files are written to their legacy locations 120 If *legacy* is true, byte-code files are written to their legacy locations
110 and names, which may overwrite byte-code files created by another version of 121 and names, which may overwrite byte-code files created by another version of
111 Python. The default is to write files to their :pep:`3147` locations and 122 Python. The default is to write files to their :pep:`3147` locations and
112 names, which allows byte-code files from multiple versions of Python to 123 names, which allows byte-code files from multiple versions of Python to
113 coexist. 124 coexist.
114 125
115 *optimize* specifies the optimization level for the compiler. It is passed t o 126 *optimize* specifies the optimization level for the compiler. It is passed t o
116 the built-in :func:`compile` function. 127 the built-in :func:`compile` function.
117 128
118 The optional argument *processes* gives the number of workers 129 The argument *workers* specifies how many workers are used to
119 used to compile files in parallel. 130 compile files in parallel. The default is to not use multiple workers.
131 If the platform can't use multiple workers and *workers* argument is given,
132 then a :exc:`NotImplementedError` will be raised.
133 If *workers* is lower than ``0``, a :exc:`ValueError` will be raised.
120 134
121 .. versionchanged:: 3.2 135 .. versionchanged:: 3.2
122 Added the *legacy* and *optimize* parameter. 136 Added the *legacy* and *optimize* parameter.
123 137
124 .. versionchanged:: 3.5 138 .. versionchanged:: 3.5
125 Added the *processes* parameter. 139 Added the *workers* parameter.
126 140
127 141
128 .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=Fals e, legacy=False, optimize=-1) 142 .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=Fals e, legacy=False, optimize=-1)
129 143
130 Compile the file with path *fullname*. 144 Compile the file with path *fullname*.
131 145
132 If *ddir* is given, it is prepended to the path to the file being compiled 146 If *ddir* is given, it is prepended to the path to the file being compiled
133 for use in compilation time tracebacks, and is also compiled in to the 147 for use in compilation time tracebacks, and is also compiled in to the
134 byte-code file, where it will be used in tracebacks and other messages in 148 byte-code file, where it will be used in tracebacks and other messages in
135 cases where the source file does not exist at the time the byte-code file is 149 cases where the source file does not exist at the time the byte-code file is
(...skipping 39 matching lines...) Expand 10 before | Expand all | Expand 10 after
175 189
176 # Perform same compilation, excluding files in .svn directories. 190 # Perform same compilation, excluding files in .svn directories.
177 import re 191 import re
178 compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True) 192 compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True)
179 193
180 194
181 .. seealso:: 195 .. seealso::
182 196
183 Module :mod:`py_compile` 197 Module :mod:`py_compile`
184 Byte-compile a single source file. 198 Byte-compile a single source file.
LEFTRIGHT
« no previous file | Lib/compileall.py » ('j') | Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Toggle Comments ('s')

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+