Merge pull request #31 from facelessuser/extmatch-fixes

facelessuser · web-flow · commit a366a0544dcc · 2019-04-23T20:44:55.000-06:00
- allow literal dots as extmatch patterns should not be treated like character sequences. - extended match patterns should be allowed to match '.' and '..' directories just like bash if the extended pattern starts with '.'. - extended pattern inverse should handle path separators correctly. - `?` or `[.]` should not trigger matching directories `.` and `..` Closes #30 #32
diff --git a/.coveragerc b/.coveragerc
@@ -1,7 +1,7 @@
 [run]
 omit=
-    backrefs/pep562.py
+    wcmatch/pep562.py
 
 [report]
 omit=
-    backrefs/pep562.py
+    wcmatch/pep562.py
diff --git a/docs/src/markdown/changelog.md b/docs/src/markdown/changelog.md
@@ -1,5 +1,12 @@
 # Changelog
 
+## 2.2.1
+
+- **FIX**: `EXTMATCH`/`EXTGLOB` should allow literal dots and should not treat dots like sequences do.
+- **FIX**: Fix `!(...)` extended match patterns in `glob` and `globmatch` so that they properly match `.` and `..` if their pattern starts with `.`.
+- **FIX**: Fix `!(...)` extended match patterns so that they handle path separators correctly.
+- **FIX**: Patterns such as `?` or `[.]` should not trigger matching directories `.` and `..` in `glob` and `globmatch`.
+
 ## 2.2.0
 
 - **NEW**: Officially support Python 3.8.
diff --git a/docs/src/markdown/fnmatch.md b/docs/src/markdown/fnmatch.md
@@ -29,7 +29,7 @@ Pattern           | Meaning
 
 - Slashes are generally treated as normal characters, but on windows they will be normalized: `/` will become `\\`. There is no need to explicitly use `\\` in patterns on Windows, but if you do, it will be handled.  This applies to matching patterns and the file names the patterns are applied to.
 - If case sensitivity is applied on a Windows system, slashes will not be normalized and pattern and file names will be treated as a Linux/Unix path.
-- By default, `.` is *not* matched by `*`, `?`, `[]`, and extended patterns such as `*(...)`. See the [`DOTMATCH`](#fnmatchdotmatch) flag to match `.` at the start of a filename without a literal `.`.
+- By default, `.` is *not* matched by `*`, `?`, and `[]`. See the [`DOTMATCH`](#fnmatchdotmatch) flag to match `.` at the start of a filename without a literal `.`.
 
 --8<-- "posix.txt"
 
@@ -145,7 +145,7 @@ When `MINUSNEGATE` is used with [`NEGATE`](#fnmatchnegate), negate patterns are
 
 #### `fnmatch.DOTMATCH, fnmatch.D` {: #fnmatchdotmatch}
 
-By default, [`glob`](#fnmatchfnmatch) and related functions will not match file or directory names that start with dot `.` unless matched with a literal dot. `DOTMATCH` allows the meta characters (such as `*`) to match dots like any other character. Dots will not be matched in `[]`, `*`, `?`, or extended patterns like `+(...)`.
+By default, [`glob`](#fnmatchfnmatch) and related functions will not match file or directory names that start with dot `.` unless matched with a literal dot. `DOTMATCH` allows the meta characters (such as `*`) to match dots like any other character. Dots will not be matched in `[]`, `*`, or `?`.
 
 #### `fnmatch.EXTMATCH, fnmatch.E` {: #fnmatchextmatch}
 
diff --git a/docs/src/markdown/glob.md b/docs/src/markdown/glob.md
@@ -34,7 +34,7 @@ Pattern           | Meaning
 - Windows drives are recognized as either `C:\\` or `\\\\Server\\mount\\` (or `C:/` and `//Server/mount/`).
 - Meta characters have no effect when inside a UNC path: `\\\\Server?\\mount*\\`.
 - If case sensitivity is applied on a Windows system, slashes will not be normalized and pattern and paths will be treated as if on Linux/Unix. Also Windows drives are no longer handled special. One exception is when using the functions [`glob`](#globglob) or [`iglob`](#globiglob).  Since `glob` and `iglob` work on the actual file system of the host, it *must* normalize slashes and handle drives to work properly on the system.
-- By default, file and directory names starting with `.` are only matched with literal `.`.  The patterns `*`, `?`, `[]`, and extended patterns like `*(...)` will not match a leading `.`.  To alter this behavior, you can use the [`DOTGLOB`](#globdotglob) flab, but even with `DOTGLOB`, `*` and `**` will not match a directory `.` or `..`.  But a pattern like `.*` will match `.` and `..`.
+- By default, file and directory names starting with `.` are only matched with literal `.`.  The patterns `*`, `**`, `?`, and `[]` will not match a leading `.`.  To alter this behavior, you can use the [`DOTGLOB`](#globdotglob) flag, but even with `DOTGLOB` these special tokens will not match a special directory (`.` or `..`).  But when a literal `.` is used, for instance in the pattern `.*`, the pattern will match `.` and `..`.
 - Relative paths and patterns are supported.
 
     ```pycon3
@@ -268,7 +268,7 @@ When `MINUSNEGATE` is used with [`NEGATE`](#globnegate), negate patterns are rec
 
 #### `glob.DOTGLOB, glob.D` {: #globdotglob}
 
-By default, [`glob`](#globglob) and [`globmatch`](#globglobmatch) will not match file or directory names that start with dot `.` unless matched with a literal dot. `DOTGLOB` allows the meta characters (such as `*`) to glob dots like any other character. Dots will not be matched in `[]`, `*`, `?`, or extended patterns like `+(...)`.
+By default, [`glob`](#globglob) and [`globmatch`](#globglobmatch) will not match file or directory names that start with dot `.` unless matched with a literal dot. `DOTGLOB` allows the meta characters (such as `*`) to glob dots like any other character. Dots will not be matched in `[]`, `*`, or `?`.
 
 Alternatively `DOTMATCH` will also be accepted for consistency with the other provided libraries. Both flags are exactly the same and are provided as a convenience in case the user finds one more intuitive than the other since `DOTGLOB` is often the name used in Bash.
 
diff --git a/tests/test_fnmatch.py b/tests/test_fnmatch.py
@@ -112,10 +112,10 @@ class TestFnMatch:
         ['?abc', '.abc', False, fnmatch.D],
         ['*abc', '.abc', False, fnmatch.D],
         ['[.]abc', '.abc', False, fnmatch.D],
-        ['*(.)abc', '.abc', False, fnmatch.E | fnmatch.D],
-        [r'*(\.)abc', '.abc', False, fnmatch.E | fnmatch.D],
+        ['*(.)abc', '.abc', True, fnmatch.E | fnmatch.D],
+        [r'*(\.)abc', '.abc', True, fnmatch.E | fnmatch.D],
         ['*(?)abc', '.abc', False, fnmatch.E | fnmatch.D],
-        ['*(?|.)abc', '.abc', False, fnmatch.E | fnmatch.D],
+        ['*(?|.)abc', '.abc', True, fnmatch.E | fnmatch.D],
         ['*(?|*)abc', '.abc', False, fnmatch.E | fnmatch.D],
         ['a.bc', 'a.bc', True, fnmatch.D],
         ['a?bc', 'a.bc', True, fnmatch.D],
diff --git a/tests/test_glob.py b/tests/test_glob.py
@@ -363,12 +363,16 @@ class Testglob(_TestGlob):
         [('aa?',), [('aaa',), ('aab',)]],
         [('aa[ab]',), [('aaa',), ('aab',)]],
         [('*q',), []],
+        [('.',), [('.',)]],
+        [('?',), [('a',)]],
+        [('[.a]',), [('a',)]],
+        [('*.',), []],
 
         # Glob inverse
         [
             ('a*', '**'),
             [
-                ('EF',), ('ZZZ',), ('',)
+                ('EF',), ('ZZZ',), ('',), ('a',), ('aaa', ), ('aab', )
             ] if not can_symlink() else [
                 ('EF',), ('ZZZ',), ('',), ('sym1',), ('sym3',), ('sym2',),
                 ('a',), ('aaa', ), ('aab', ), ('sym3', 'efg'), ('sym3', 'efg', 'ha'), ('sym3', 'EF')
@@ -716,7 +720,7 @@ class Testglob(_TestGlob):
         [
             ('a*', '**'),
             [
-                ('EF',), ('ZZZ',)
+                ('EF',), ('ZZZ',), ('a',), ('aaa', ), ('aab', )
             ] if not can_symlink() else [
                 ('EF',), ('ZZZ',),
                 ('a',), ('aaa', ), ('aab', ), ('sym1',), ('sym3',), ('sym2',), ('sym3', 'efg'), ('sym3', 'efg', 'ha'),
diff --git a/tests/test_globmatch.py b/tests/test_globmatch.py
@@ -382,6 +382,46 @@ class TestGlobFilter:
         ['!(test)', ['.abc', 'abc'], glob.D | glob.M],
         ['.!(test)', ['.', '..', '.abc'], glob.M],
         ['.!(test)', ['.', '..', '.abc'], glob.D | glob.M],
+        ['!(.)', ['..', '.abc', 'abc'], glob.M],
+        [r'!(\.)', ['..', '.abc', 'abc'], glob.M],
+        [r'!(\x2e)', ['..', '.abc', 'abc'], glob.M | glob.R],
+        ['@(!(.))', ['..', '.abc', 'abc'], glob.M],
+        ['!(@(.))', ['..', '.abc', 'abc'], glob.M],
+        ['+(!(.))', ['..', '.abc', 'abc'], glob.M],
+        ['!(+(.))', ['.abc', 'abc'], glob.M],
+        ['!(?)', ['abc'], glob.M],
+        ['!(*)', [], glob.M],
+        ['!([.])', ['abc'], glob.M],
+        ['!(.)', ['..', '.abc', 'abc'], glob.M | glob.D],
+        [r'!(\.)', ['..', '.abc', 'abc'], glob.M | glob.D],
+        [r'!(\x2e)', ['..', '.abc', 'abc'], glob.M | glob.R | glob.D],
+        ['@(!(.))', ['..', '.abc', 'abc'], glob.M | glob.D],
+        ['!(@(.))', ['..', '.abc', 'abc'], glob.M | glob.D],
+        ['+(!(.))', ['..', '.abc', 'abc'], glob.M | glob.D],
+        ['!(+(.))', ['.abc', 'abc'], glob.M | glob.D],
+        ['!(?)', ['.abc', 'abc'], glob.M | glob.D],
+        ['!(*)', [], glob.M | glob.D],
+        ['!([.])', ['.abc', 'abc'], glob.M | glob.D],
+
+        # More extended pattern dot related tests
+        ['*(.)', ['.', '..']],
+        [r'*(\.)', ['.', '..']],
+        ['*([.])', []],
+        ['*(?)', ['abc']],
+        ['@(.?)', ['..']],
+        ['@(?.)', []],
+        ['*(.)', ['.', '..'], glob.D],
+        [r'*(\.)', ['.', '..'], glob.D],
+        ['*([.])', [], glob.D],
+        ['*(?)', ['.abc', 'abc'], glob.D],
+        ['@(.?)', ['..'], glob.D],
+        ['@(?.)', [], glob.D],
+
+        GlobFiles(['folder/abc', 'directory/abc', 'dir/abc']),
+        # Test that inverse works properly mid path.
+        ['!(folder)/*', ['directory/abc', 'dir/abc'], glob.M],
+        ['!(folder)dir/abc', ['dir/abc'], glob.M],
+        ['!(dir)/abc', ['directory/abc', 'folder/abc'], glob.M],
 
         # Slash exclusion
         GlobFiles(
@@ -725,23 +765,26 @@ def test_glob_translate(self, mock__iscase_sensitive):
         if util.PY37:
             value = (
                 [
-                    '^(?s:(?:(?!(?:/|^)\\.).)*?(?:^|$|/)+(?![/.])[\x00-\x7f]/+stuff/+(?=.)'
+                    '^(?s:(?:(?!(?:/|^)\\.).)*?(?:^|$|/)+'
+                    '(?!(?:\\.{1,2})(?:$|/))(?![/.])[\x00-\x7f]/+stuff/+(?=.)'
                     '(?!(?:\\.{1,2})(?:$|/))(?:(?!\\.)[^/]*?)?[/]*?)$'
                 ],
                 []
             )
         elif util.PY36:
             value = (
                 [
-                    '^(?s:(?:(?!(?:\\/|^)\\.).)*?(?:^|$|\\/)+(?![\\/.])[\x00-\x7f]\\/+stuff\\/+(?=.)'
+                    '^(?s:(?:(?!(?:\\/|^)\\.).)*?(?:^|$|\\/)+'
+                    '(?!(?:\\.{1,2})(?:$|\\/))(?![\\/.])[\x00-\x7f]\\/+stuff\\/+(?=.)'
                     '(?!(?:\\.{1,2})(?:$|\\/))(?:(?!\\.)[^\\/]*?)?[\\/]*?)$'
                 ],
                 []
             )
         else:
             value = (
                 [
-                    '(?s)^(?:(?:(?!(?:\\/|^)\\.).)*?(?:^|$|\\/)+(?![\\/.])[\x00-\x7f]\\/+stuff\\/+(?=.)'
+                    '(?s)^(?:(?:(?!(?:\\/|^)\\.).)*?(?:^|$|\\/)+'
+                    '(?!(?:\\.{1,2})(?:$|\\/))(?![\\/.])[\x00-\x7f]\\/+stuff\\/+(?=.)'
                     '(?!(?:\\.{1,2})(?:$|\\/))(?:(?!\\.)[^\\/]*?)?[\\/]*?)$'
                 ],
                 []
diff --git a/tox.ini b/tox.ini
@@ -12,6 +12,7 @@ deps=
 commands=
     {envbindir}/py.test --cov wcmatch --cov-append tests
     {envbindir}/coverage html -d {envtmpdir}/coverage
+    {envbindir}/coverage report --show-missing
 
 [testenv:lint]
 deps=
diff --git a/wcmatch/__meta__.py b/wcmatch/__meta__.py
@@ -186,5 +186,5 @@ def parse_version(ver, pre=False):
     return Version(major, minor, micro, release, pre, post, dev)
 
 
-__version_info__ = Version(2, 2, 0, "final")
+__version_info__ = Version(2, 2, 1, "final")
 __version__ = __version_info__._get_canonical()
diff --git a/wcmatch/_wcparse.py b/wcmatch/_wcparse.py
@@ -102,6 +102,7 @@
 _ONE_OR_MORE = r'+'
 # End of pattern
 _EOP = r'$'
+_PATH_EOP = r'(?:$|%(sep)s)'
 # Divider between `globstar`. Can match start or end of pattern
 # in addition to slashes.
 _GLOBSTAR_DIV = r'(?:^|$|%s)+'
@@ -603,6 +604,8 @@ def __init__(self, pattern, flags=0):
             self.bslash_abort = False
             self.sep = '/'
         sep = {"sep": re.escape(self.sep)}
+        self.path_eop = _PATH_EOP % sep
+        self.no_dir = _NO_DIR % sep
         self.seq_path = _PATH_NO_SLASH % sep
         self.seq_path_dot = _PATH_NO_SLASH_DOT % sep
         self.path_star = _PATH_STAR % sep
@@ -643,11 +646,18 @@ def update_dir_state(self):
         elif not self.dir_start and self.after_start:
             self.reset_dir_track()
 
+    def _restrict_extended_slash(self):
+        """Restrict extended slash."""
+
+        return self.seq_path if self.pathname else ''
+
     def _restrict_sequence(self):
         """Restrict sequence."""
 
         if self.pathname:
             value = self.seq_path_dot if self.after_start and not self.dot else self.seq_path
+            if self.after_start:
+                value = self.no_dir + value
         else:
             value = _NO_DOT if self.after_start and not self.dot else ""
         self.reset_dir_track()
@@ -794,7 +804,7 @@ def _sequence(self, i):
             else:
                 result = [value]
 
-        if self.pathname or (self.after_start and not self.dot):
+        if self.pathname or self.after_start:
             return self._restrict_sequence() + ''.join(result)
 
         return ''.join(result)
@@ -814,23 +824,25 @@ def _references(self, i, sequence=False):
                     value = self.get_path_sep() + _ONE_OR_MORE
                     self.set_start_dir()
                 else:
-                    value = self._restrict_sequence() + value
+                    value = self._restrict_extended_slash() + value
         elif c == '/':
             # \/
             if sequence and self.pathname:
                 raise PathNameException
             if self.pathname:
                 value = r'\\'
                 if self.in_list:
-                    value = self._restrict_sequence() + value
+                    value = self._restrict_extended_slash() + value
                 i.rewind(1)
             else:
                 value = re.escape(c)
-        elif self.in_list and not sequence and self.after_start and c == '.':
-            value = _NO_DOT + re.escape(c)
         else:
             # \a, \b, \c, etc.
             value = re.escape(c)
+            if c == '.' and self.after_start and self.in_list:
+                self.allow_special_dir = True
+                self.reset_dir_track()
+
         return value
 
     def _handle_star(self, i, current):
@@ -923,7 +935,7 @@ def _handle_star(self, i, current):
         else:
             current.append(value)
 
-    def clean_up_inverse(self, current, default=None):
+    def clean_up_inverse(self, current):
         """
         Clean up current.
 
@@ -941,18 +953,16 @@ def clean_up_inverse(self, current, default=None):
         if not self.inv_ext:
             return
 
-        if default is None:
-            default = ''
-
         index = len(current) - 1
         while index >= 0:
             if isinstance(current[index], InvPlaceholder):
                 content = current[index + 1:]
-                current[index] = (''.join(content) if content else default) + (_EXCLA_GROUP_CLOSE % str(current[index]))
+                content.append(_EOP if not self.pathname else self.path_eop)
+                current[index] = (''.join(content)) + (_EXCLA_GROUP_CLOSE % str(current[index]))
             index -= 1
         self.inv_ext = 0
 
-    def parse_extend(self, c, i, current):
+    def parse_extend(self, c, i, current, reset_dot=False):
         """Parse extended pattern lists."""
 
         # Save state
@@ -961,6 +971,8 @@ def parse_extend(self, c, i, current):
         temp_in_list = self.in_list
         temp_inv_ext = self.inv_ext
         self.in_list = True
+        if reset_dot:
+            self.allow_special_dir = False
 
         # Start list parsing
         success = True
@@ -979,14 +991,15 @@ def parse_extend(self, c, i, current):
                     pass
                 elif c == '*':
                     self._handle_star(i, extended)
-                elif c == '.' and not self.dot and self.after_start:
-                    extended.append(_NO_DOT + re.escape(c))
+                elif c == '.' and self.after_start:
+                    extended.append(re.escape(c))
+                    self.allow_special_dir = True
                     self.reset_dir_track()
                 elif c == '?':
                     extended.append(self._restrict_sequence() + _QMARK)
                 elif c == '/':
                     if self.pathname:
-                        extended.append(self._restrict_sequence())
+                        extended.append(self._restrict_extended_slash())
                     extended.append(re.escape(c))
                 elif c == "|":
                     self.clean_up_inverse(extended)
@@ -1026,17 +1039,18 @@ def parse_extend(self, c, i, current):
                 # If pattern is at the end, anchor the match to the end.
                 current.append(_EXCLA_GROUP % ''.join(extended))
                 if self.pathname:
-                    if temp_after_start and not self.dot:
+                    if not temp_after_start or self.allow_special_dir:
+                        star = self.path_star
+                    elif temp_after_start and not self.dot:
                         star = self.path_star_dot2
-                    elif temp_after_start:
-                        star = self.path_star_dot1
                     else:
-                        star = self.path_star
+                        star = self.path_star_dot1
                 else:
-                    if temp_after_start and not self.dot:
-                        star = _NO_DOT + _STAR
-                    else:
+                    if not temp_after_start or self.dot:
                         star = _STAR
+                    else:
+                        star = _NO_DOT + _STAR
+
                 if temp_after_start:
                     star = _NEED_CHAR + star
                 # Place holder for closing, but store the proper star
@@ -1108,7 +1122,7 @@ def root(self, pattern, current):
         for c in i:
 
             index = i.index
-            if self.extend and c in EXT_TYPES and self.parse_extend(c, i, current):
+            if self.extend and c in EXT_TYPES and self.parse_extend(c, i, current, True):
                 # Nothing to do
                 pass
             elif c == '*':
@@ -1146,7 +1160,7 @@ def root(self, pattern, current):
 
             self.update_dir_state()
 
-        self.clean_up_inverse(current, default=_EOP)
+        self.clean_up_inverse(current)
         if self.pathname:
             current.append(_PATH_TRAIL % self.get_path_sep())
 
