classification
Title: glob returns empty list with "[" character in the folder name
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.2
process
Status: open Resolution: invalid
Dependencies: Superseder:
Assigned To: Nosy List: Aquinas, eric.smith, ezio.melotti, flacs, george.hu, kveretennicov, l0nwlf, terry.reedy
Priority: normal Keywords:

Created on 2010-04-15 00:51 by george.hu, last changed 2011-11-11 14:34 by ezio.melotti.

Messages (15)
msg103160 - (view) Author: george hu (george.hu) Date: 2010-04-15 00:51
Have this problem in python 2.5.4 under windows. 
I'm trying to return a list of files in a directory by using glob. It keeps returning a empty list until I tested/adjusted folder name by removing "[" character from it. Not sure if this is a bug.

glob.glob("c:\abc\afolderwith[test]\*") returns empty list
glob.glob("c:\abc\afolderwithtest]\*") returns files
msg103163 - (view) Author: Shashwat Anand (l0nwlf) Date: 2010-04-15 01:06
When you do :
glob.glob("c:\abc\afolderwith[test]\*") returns empty list
It looks for all files in three directories:
c:\abc\afolderwitht\*
c:\abc\afolderwithe\*
c:\abc\afolderwiths\*

Ofcourse they do not exist so it returns empty list

06:35:05 l0nwlf-MBP:Desktop $ ls -R test
1 2 3
06:35:15 l0nwlf-MBP:Desktop $ ls -R test1
alpha beta  gamma

>>> glob.glob('/Users/l0nwlf/Desktop/test[123]/*')
['/Users/l0nwlf/Desktop/test1/alpha', '/Users/l0nwlf/Desktop/test1/beta', '/Users/l0nwlf/Desktop/test1/gamma']

As you can see, by giving the argument test[123] it looked for test1, test2, test3. Since test1 existed, it gave all the files present within it.
msg103164 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-04-15 01:09
See the explanation at http://docs.python.org/library/fnmatch.html#module-fnmatch , which uses the same rules.
msg103165 - (view) Author: george hu (george.hu) Date: 2010-04-15 01:16
Ok, what if the name of the directory contains "[]" characters? What is the escape string for that?
msg103168 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-04-15 01:27
The documentation for fnmatch.translate, which is what ultimately gets called, says:
   There is no way to quote meta-characters.
Sorry.

If you want to see this changed, you could open a feature request. If you have a patch, that would help!

You probably want to research what the Unix shells use for escaping globs.
msg103171 - (view) Author: Shashwat Anand (l0nwlf) Date: 2010-04-15 01:34
glob module does not provide what you want.
As a workaround you can try:

os.listdir("c:\abc\afolderwith[test]")

07:02:52 l0nwlf-MBP:Desktop $ ls -R test\[123\]/
1 2 3
>>> os.listdir('/Users/l0nwlf/Desktop/test[123]')
['1', '2', '3']

Changing type to 'Feature Request'
msg103173 - (view) Author: george hu (george.hu) Date: 2010-04-15 01:40
Well, the listdir doesn't support "wildcard", for example,
listdir("*.app"). I know the glob is kind of unix shell style expanding, but
my program is running under windows, it's my tiny script to walk through a
huge directory in my NAS. And there are many directories named with "[]" and
"()" characters amid. May the only way is to program a filter on the
listdir.

On Wed, Apr 14, 2010 at 6:34 PM, Shashwat Anand <report@bugs.python.org>wrote:

>
> Shashwat Anand <anand.shashwat@gmail.com> added the comment:
>
> glob module does not provide what you want.
> As a workaround you can try:
>
> os.listdir("c:\abc\afolderwith[test]")
>
> 07:02:52 l0nwlf-MBP:Desktop $ ls -R test\[123\]/
> 1 2 3
> >>> os.listdir('/Users/l0nwlf/Desktop/test[123]')
> ['1', '2', '3']
>
> Changing type to 'Feature Request'
>
> ----------
> status: pending -> open
> type: behavior -> feature request
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue8402>
> _______________________________________
>
msg103174 - (view) Author: george hu (george.hu) Date: 2010-04-15 01:43
Well, the listdir doesn't support "wildcard", for example, listdir("*.app"). I know the glob is kind of unix shell style expanding, but my program is running under windows, it's my tiny script to walk through a huge directory in my NAS. And there are many directories named with "[]" and "()" characters amid. May be the only way is to write a filter on the listdir.
msg103175 - (view) Author: Shashwat Anand (l0nwlf) Date: 2010-04-15 01:46
You repeated the same comment twice and added an 'unnamed' file. I assume you did it by mistake.
msg106545 - (view) Author: Dan Gawarecki (Aquinas) Date: 2010-05-26 17:09
Shouldn't the title be updated to indicate the fnmatch is the true source of the behavior (I'm basing this on http://docs.python.org/library/glob.html indicating the fnmatch is invoked by glob).  I'm not using glob, but fnmatch in my attempt to find filenames that look like "Ajax_[version2].txt".  

If nothing else, it would have helped me if the documentation would state whether or not the brackets could be escaped.  It doesn't appear from my tests (trying "Ajax_\[version2\].txt" and "Ajax_\\[version2\\].txt") that 'escaping' is possible, but if the filter pattern gets turned into a regular expression, I think escaping *would* be possible.  Is that a reasonable assumption?

I'm running 2.5.1 under Windows, and this is my first ever post to the bugs list.
msg106548 - (view) Author: Dan Gawarecki (Aquinas) Date: 2010-05-26 17:17
Following up...
I saw Eric Smith's 2nd note (2010-04-15 @1:27) about fnmatch.translate documentation stating that
   "There is no way to quote meta-characters."

When I looked at:
http://docs.python.org/library/fnmatch.html#module-fnmatch

did not see this statement appear anywhere.  Would this absence be because someone is working on making this enhancement?
msg106550 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-05-26 17:32
I don't think so. That quote came from the docstring for fnmatch.translate.

>>> help(fnmatch.translate)
Help on function translate in module fnmatch:

translate(pat)
    Translate a shell PATTERN to a regular expression.
    
    There is no way to quote meta-characters.
msg109682 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-07-09 05:06
The 3.1.2 doc for fnmatch.translate no longer says "There is no way to quote meta-characters." If that is still true (no quoting method is given that I can see), then that removal is something of a regression.
msg109743 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-07-09 14:15
The note about no quoting meta-chars is in the docstring for fnmatch.translate, not the documentation. I still see it in 3.1. I have a to-do item to add this to the actual documentation. I'll add an issue.
msg147434 - (view) Author: (flacs) Date: 2011-11-11 14:33
As a workaround, it is possible to make every glob character a character set of one character (wrapping it with [] ). The gotcha here is that you can't just use multiple replaces because you would escape the escape brackets.

Here is a function adapted from [1]:

def escape_glob(path):
    transdict = {
            '[': '[[]',
            ']': '[]]',
            '*': '[*]',
            '?': '[?]',
            }
    rc = re.compile('|'.join(map(re.escape, transdict)))
    return rc.sub(lambda m: transdict[m.group(0)], path)

[1] http://www.daniweb.com/software-development/python/code/216636
History
Date User Action Args
2011-11-11 14:34:09ezio.melottisetnosy: + ezio.melotti
2011-11-11 14:33:11flacssetnosy: + flacs
messages: + msg147434
components: + Library (Lib)
2010-09-27 20:41:29kveretennicovsetnosy: + kveretennicov
2010-07-09 14:15:33eric.smithsetmessages: + msg109743
2010-07-09 05:06:33terry.reedysetnosy: + terry.reedy

messages: + msg109682
versions: + Python 3.2, - Python 2.5
2010-05-26 17:32:59eric.smithsetmessages: + msg106550
2010-05-26 17:17:10Aquinassetmessages: + msg106548
2010-05-26 17:09:09Aquinassetnosy: + Aquinas
messages: + msg106545
2010-04-15 02:21:42r.david.murraysetfiles: - unnamed
2010-04-15 01:51:14l0nwlfsettitle: glob returns empty list with " -> glob returns empty list with "[" character in the folder name
2010-04-15 01:46:53l0nwlfsetmessages: + msg103175
2010-04-15 01:43:17george.husetmessages: + msg103174
2010-04-15 01:40:19george.husetfiles: + unnamed

messages: + msg103173
title: glob returns empty list with "[" character in the folder name -> glob returns empty list with "
2010-04-15 01:34:35l0nwlfsetstatus: pending -> open
type: behavior -> enhancement
messages: + msg103171
2010-04-15 01:27:56eric.smithsetstatus: open -> pending

messages: + msg103168
2010-04-15 01:16:54george.husetstatus: closed -> open

messages: + msg103165
2010-04-15 01:09:30eric.smithsetstatus: open -> closed

nosy: + eric.smith
messages: + msg103164

resolution: invalid
2010-04-15 01:06:44l0nwlfsetnosy: + l0nwlf
messages: + msg103163
2010-04-15 00:51:25george.hucreate