classification
Title: Python 3.8 regression: endless loop in shutil.copytree
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: cboltz, giampaolo.rodola, kinow
Priority: normal Keywords: 3.8regression, patch

Created on 2019-11-04 21:21 by cboltz, last changed 2019-11-09 13:26 by kinow.

Pull Requests
URL Status Linked Edit
PR 17098 open kinow, 2019-11-09 13:25
Messages (2)
msg355981 - (view) Author: Christian Boltz (cboltz) Date: 2019-11-04 21:21
The following test script works with Python 3.7 (and older), but triggers an endless loop with Python 3.8:


#!/usr/bin/python3

import shutil
import os

os.mkdir('/dev/shm/t')
os.mkdir('/dev/shm/t/pg')

with open('/dev/shm/t/pg/pol', 'w+') as f:
    f.write('pol')

shutil.copytree('/dev/shm/t/pg', '/dev/shm/t/pg/somevendor/1.0')


The important point is probably that 'pg' gets copied into a subdirectory of itsself. While this worked in Python up to 3.7, doing the same in Python 3.8 runs into an endless loop:

# python3 /home/abuild/rpmbuild/SOURCES/test.py
Traceback (most recent call last):
  File "/home/abuild/rpmbuild/SOURCES/test.py", line 15, in <module> 
    shutil.copytree('/dev/shm/t/pg', '/dev/shm/t/pg/somevendor/1.0')
  File "/usr/lib/python3.8/shutil.py", line 547, in copytree 
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/lib/python3.8/shutil.py", line 486, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
...
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/usr/lib/python3.8/shutil.py", line 547, in copytree 
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/lib/python3.8/shutil.py", line 449, in _copytree
    os.makedirs(dst, exist_ok=dirs_exist_ok)
  File "/usr/lib/python3.8/os.py", line 206, in makedirs 
    head, tail = path.split(name)
  File "/usr/lib/python3.8/posixpath.py", line 104, in split 
    sep = _get_sep(p)
  File "/usr/lib/python3.8/posixpath.py", line 42, in _get_sep 
    if isinstance(path, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object

I also reported this at https://bugzilla.opensuse.org/show_bug.cgi?id=1155839
msg356302 - (view) Author: Bruno P. Kinoshita (kinow) * Date: 2019-11-09 13:26
Hi,

I did a quick `git bisect` using the example provided, and looks like this regression was added in the fix for bpo-33695, commit 19c46a4c96553b2a8390bf8a0e138f2b23e28ed6.

It looks to me that the iterator returned by with os.scandir(...) is including the newly created dst directory (see the call for `os.makedirs(dst, exist_ok=dirs_exist_ok)` in https://github.com/python/cpython/blob/e27449da92b13730a5e11182f329d5da98a5e05b/Lib/shutil.py#L449).

This results in the function finding an extra directory, and repeating the steps for this folder and its subfolder recursively. This only happens because in the example in this issue, dst is a subdirectory of src.

The bpo-33695 commit had more changes, so I've reverted just this block of the copytree as a tentative fix, and added a unit test: https://github.com/python/cpython/pull/17098

--

Here's a simplified version of what's going on:

```
import os
import shutil

shutil.rmtree('/tmp/test/', True)
os.makedirs('/tmp/test')
with open('/tmp/test/foo', 'w+') as f:
  f.write('foo')

# now we have /tmp/test/foo, let's simulate what happens in copytree on master

with os.scandir('/tmp/test') as entries:
  # up to this point, /tmp/test/foo is the only entry
  os.makedirs('/tmp/test/1/2/3/', exist_ok=True) # <---- when the iterator starts below in `f in entries`, it will find 1 too
  # now 1 will have been added too
  for f in entries:
    print(f)
```
History
Date User Action Args
2019-11-09 13:26:16kinowsetnosy: + kinow
messages: + msg356302
2019-11-09 13:25:26kinowsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request16604
2019-11-08 23:33:11terry.reedysetkeywords: + 3.8regression
type: crash -> behavior
stage: needs patch
2019-11-04 21:24:13vstinnersetnosy: + giampaolo.rodola
2019-11-04 21:21:38cboltzcreate