Message393190
I have a directory with hundreds of thousands of text files. I wanted to explore one file, so I wrote the following code expecting it to happen basically instantaneously because of how generators work:
```python
from pathlib import Path
base_dir = Path("/path/to/lotta/files/")
files = base_dir.glob("*.txt") # return immediately
first_file = next(files) # doesn't return immediately
```
to my surprise, this took a long time to finish since `next` on a generator should be O(1).
A colleague pointed me to the following code: https://github.com/python/cpython/blob/adcd2205565f91c6719f4141ab4e1da6d7086126/Lib/pathlib.py#L431
I assume calling this list is to "freeze" a potentially changing directory since `scandir` relies on `os.stat`, but this causes a huge penalty and makes the generator return-type a bit disingenuous. In any case, I think this is bug worthy in someo sense. |
|
Date |
User |
Action |
Args |
2021-05-07 15:46:22 | Elijah Rippeth | set | recipients:
+ Elijah Rippeth |
2021-05-07 15:46:22 | Elijah Rippeth | set | messageid: <1620402382.41.0.425186476785.issue44069@roundup.psfhosted.org> |
2021-05-07 15:46:22 | Elijah Rippeth | link | issue44069 messages |
2021-05-07 15:46:22 | Elijah Rippeth | create | |
|