Glob API

[mwichmann]

Over time, various SCons users have requested the ability to glob recursively with Glob. #3851 codifies this request explicitly, but it also appears in earlier issues. This is an effort to start a discussion about the API of the Glob function, as there are several questions (some touched on in the referenced issue).

1. Python's glob module introduced a new function, iglob: Return an iterator which yields the paths matching a pathname pattern. iglob is now the implementation for glob, and glob itself is reduced to return list(iglob(...args...). SCons could introduce a similar interface, but upon further thought, it doesn't seem to make much sense - Glob has to sort its results to assure stable file lists across runs, and sorting means you blow away the benefits of returning an iterator. So, proposing we not pursue this bit of alignment with Python's stdlib. An iglob-type implementation could be used internally (not as a new public API) if it makes things easier (yield one match at a time instead of having to collect - might make simpler convenience routines internally).
2. The syntax added by glob.glob (Python 3.5) is to add a new keyword argument, recursive, which, if true, enables special interpretation of ** in the pattern, making it recursive. The interpretation of this syntax is a little confusing: the ** gets a different treatment depending on whether it's inside a pattern sequence, or followed by a pathname separator (**/). The internet seems has conflicting wording on this, the Python docs only very recently added a sentence on this, which to me does not actually clarify: If the pattern is followed by an os.sep or os.altsep then files will not match
3. The syntax in pathlib.Path.glob is different, it uses this wording: Patterns are the same as for fnmatch, with the addition of “**” which means “this directory and all subdirectories, recursively”. It does not require a special flag to enable this interpretation. The wording added for the case mentioned at the end of the previous bullet is: Changed in version 3.11: Return only directories if pattern ends with a pathname components separator (sep or altsep).
4. Pathlib also exposes an rglob method, which behaves as if the **/ were supplied without the need to do so.
5. Very recent glob.glob (py3.11) has added a new flag: If include_hidden is true, “**” pattern will match hidden directories. Would that be a useful addition? Or are directories starting with "." not a particularly useful case for SCons?
6. (unrelated to all the above, sorry!) The existing Glob API includes a source parameter, which reportedly gives you matches in the source directory if using variant dirs. Don't see any evidence that this works, and wording is inconsistent about it anyway (is it expected to always work, or work only if duplicate=False?). The actual implementation is SCons.Node.FS.Dir._glob1(), which takes the source argument passed down, and never uses it at all. Is source useful and should be implemented, or...

So: if we do a recursive addition, should it use the glob.glob approach of requiring a flag to be set, or the pathlib approach of not requiring a flag, or the alternate pathlib approach of exposing a new function (maybe RGlob?) which does recursive search, and perhaps leave Glob alone?

[bdbaddog]

We probably can't change the default behavior of Glob() without a deprecation cycle, but if we added recurse=False as an argument to it, that wouldn't break any existing code and new code (or updated old code) would be able to take advantage of the change.

[mwichmann]

I suspect that's why glob.glob chose to add an argument - so unless you set it true the behavior is exactly as before, but it's not likely that ** would occur naturally either...

[loonycyborg]

I'd favor adding support for ** to Glob, possibly gated by default recursive=False. But I doubt much code would be broken by just enabling ** always.

[mwichmann]

Cross-linking updates: Item 5 above is tracked as #4231, and Item 6 as #4233.