Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(23433)

Delta Between Two Patch Sets: Doc/library/compileall.rst

Issue 16104: Use multiprocessing in compileall script
Left Patch Set: Created 5 years, 4 months ago
Right Patch Set: Created 5 years ago
Left:
Right:
Use n/p to move between diff chunks; N/P to move between comments. Please Sign in to add in-line comments.
Jump to:
Left: Side by side diff | Download
Right: Side by side diff | Download
« no previous file with change/comment | « no previous file | Lib/compileall.py » ('j') | Lib/compileall.py » ('J')
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
LEFTRIGHT
1 :mod:`compileall` --- Byte-compile Python libraries 1 :mod:`compileall` --- Byte-compile Python libraries
2 =================================================== 2 ===================================================
3 3
4 .. module:: compileall 4 .. module:: compileall
5 :synopsis: Tools for byte-compiling all Python source files in a directory tr ee. 5 :synopsis: Tools for byte-compiling all Python source files in a directory tr ee.
6 6
7 7
8 This module provides some utility functions to support installing Python 8 This module provides some utility functions to support installing Python
9 libraries. These functions compile Python source files in a directory tree. 9 libraries. These functions compile Python source files in a directory tree.
10 This module can be used to create the cached byte-code files at library 10 This module can be used to create the cached byte-code files at library
(...skipping 48 matching lines...) Expand 10 before | Expand all | Expand 10 after
59 files and directories to compile. If ``list`` is ``-``, read lines from 59 files and directories to compile. If ``list`` is ``-``, read lines from
60 ``stdin``. 60 ``stdin``.
61 61
62 .. cmdoption:: -b 62 .. cmdoption:: -b
63 63
64 Write the byte-code files to their legacy locations and names, which may 64 Write the byte-code files to their legacy locations and names, which may
65 overwrite byte-code files created by another version of Python. The default 65 overwrite byte-code files created by another version of Python. The default
66 is to write files to their :pep:`3147` locations and names, which allows 66 is to write files to their :pep:`3147` locations and names, which allows
67 byte-code files from multiple versions of Python to coexist. 67 byte-code files from multiple versions of Python to coexist.
68 68
69 .. cmdoption:: -r
70
71 Control the maximum recursion level for subdirectories.
72 If this is given, then ``-l`` option will not be taken into account.
73 :program:`python -m compileall <directory> -r 0` is equivalent to
74 :program:`python -m compileall <directory> -l`.
75
69 .. cmdoption:: -j N 76 .. cmdoption:: -j N
70 77
71 Use *N* processes to compile the given directory. 78 Use *N* workers to compile the files within the given directory.
Charles-François Natali 2014/04/27 16:28:46 The wording is strange, maybe "compile the files w
Claudiu.Popa 2014/04/27 18:13:36 Done.
72 If ``0`` is used, then the result of :func:`os.cpu_count()` 79 If ``0`` is used, then the result of :func:`os.cpu_count()`
73 will be used. 80 will be used.
74 81
75 .. versionchanged:: 3.2 82 .. versionchanged:: 3.2
76 Added the ``-i``, ``-b`` and ``-h`` options. 83 Added the ``-i``, ``-b`` and ``-h`` options.
77 84
78 .. versionchanged:: 3.5 85 .. versionchanged:: 3.5
79 Added the ``-j`` option. 86 Added the ``-j`` and ``-r`` options.
87
80 88
81 There is no command-line option to control the optimization level used by the 89 There is no command-line option to control the optimization level used by the
82 :func:`compile` function, because the Python interpreter itself already 90 :func:`compile` function, because the Python interpreter itself already
83 provides the option: :program:`python -O -m compileall`. 91 provides the option: :program:`python -O -m compileall`.
84 92
85 Public functions 93 Public functions
86 ---------------- 94 ----------------
87 95
88 .. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, qu iet=False, legacy=False, optimize=-1, processes=None) 96 .. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, qu iet=False, legacy=False, optimize=-1, workers=1)
89 97
90 Recursively descend the directory tree named by *dir*, compiling all :file:`. py` 98 Recursively descend the directory tree named by *dir*, compiling all :file:`. py`
91 files along the way. 99 files along the way.
92 100
93 The *maxlevels* parameter is used to limit the depth of the recursion; it 101 The *maxlevels* parameter is used to limit the depth of the recursion; it
94 defaults to ``10``. 102 defaults to ``10``.
95 103
96 If *ddir* is given, it is prepended to the path to each file being compiled 104 If *ddir* is given, it is prepended to the path to each file being compiled
97 for use in compilation time tracebacks, and is also compiled in to the 105 for use in compilation time tracebacks, and is also compiled in to the
98 byte-code file, where it will be used in tracebacks and other messages in 106 byte-code file, where it will be used in tracebacks and other messages in
(...skipping 12 matching lines...) Expand all
111 119
112 If *legacy* is true, byte-code files are written to their legacy locations 120 If *legacy* is true, byte-code files are written to their legacy locations
113 and names, which may overwrite byte-code files created by another version of 121 and names, which may overwrite byte-code files created by another version of
114 Python. The default is to write files to their :pep:`3147` locations and 122 Python. The default is to write files to their :pep:`3147` locations and
115 names, which allows byte-code files from multiple versions of Python to 123 names, which allows byte-code files from multiple versions of Python to
116 coexist. 124 coexist.
117 125
118 *optimize* specifies the optimization level for the compiler. It is passed t o 126 *optimize* specifies the optimization level for the compiler. It is passed t o
119 the built-in :func:`compile` function. 127 the built-in :func:`compile` function.
120 128
121 The argument *processes* specifies how many workers are used to 129 The argument *workers* specifies how many workers are used to
Charles-François Natali 2014/04/27 16:28:46 It's really nit-picking, but why aren't you lines
Claudiu.Popa 2014/04/27 18:13:36 Habit. ;)
122 compile files in parallel. The default is to not use multiple workers. 130 compile files in parallel. The default is to not use multiple workers.
123 If :mod:`multiprocessing` is unavailable, then a :exc:`ValueError` 131 If the platform can't use multiple workers and *workers* argument is given,
Charles-François Natali 2014/04/27 16:28:46 I think it should default to one, not None. If it
Claudiu.Popa 2014/04/27 18:13:36 Mm, I think that it's not good to default to one,
124 will be raised in case that *processes* is given. 132 then a :exc:`NotImplementedError` will be raised.
125 If *processes* is lower or equal to ``0``, a :exc:`ValueError` 133 If *workers* is lower than ``0``, a :exc:`ValueError` will be raised.
Charles-François Natali 2014/04/27 16:28:46 So there's no easy way to use all available cores
Claudiu.Popa 2014/04/27 18:13:36 Only with the cli interface. The users could call
126 will be raised.
127 134
128 .. versionchanged:: 3.2 135 .. versionchanged:: 3.2
129 Added the *legacy* and *optimize* parameter. 136 Added the *legacy* and *optimize* parameter.
130 137
131 .. versionchanged:: 3.5 138 .. versionchanged:: 3.5
132 Added the *processes* parameter. 139 Added the *workers* parameter.
133 140
134 141
135 .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=Fals e, legacy=False, optimize=-1) 142 .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=Fals e, legacy=False, optimize=-1)
136 143
137 Compile the file with path *fullname*. 144 Compile the file with path *fullname*.
138 145
139 If *ddir* is given, it is prepended to the path to the file being compiled 146 If *ddir* is given, it is prepended to the path to the file being compiled
140 for use in compilation time tracebacks, and is also compiled in to the 147 for use in compilation time tracebacks, and is also compiled in to the
141 byte-code file, where it will be used in tracebacks and other messages in 148 byte-code file, where it will be used in tracebacks and other messages in
142 cases where the source file does not exist at the time the byte-code file is 149 cases where the source file does not exist at the time the byte-code file is
(...skipping 39 matching lines...) Expand 10 before | Expand all | Expand 10 after
182 189
183 # Perform same compilation, excluding files in .svn directories. 190 # Perform same compilation, excluding files in .svn directories.
184 import re 191 import re
185 compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True) 192 compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True)
186 193
187 194
188 .. seealso:: 195 .. seealso::
189 196
190 Module :mod:`py_compile` 197 Module :mod:`py_compile`
191 Byte-compile a single source file. 198 Byte-compile a single source file.
LEFTRIGHT
« no previous file | Lib/compileall.py » ('j') | Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Toggle Comments ('s')

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+