How to search for files with a wildcard in Common Lisp?

Question

I am not satisfied to find files matching a string like this:

(remove-if-not (lambda (it)
                   (search "wildcard" (namestring it)))
               (uiop:directory-files "./"))
;; I'll ignore case with str:contains?
;; https://github.com/vindarel/cl-str

How would one search for files with unix-style wildcards ?

If it is not built-in, I'd enjoy a solution with uiop. Maybe there is with Osicat or cl-fad (with which it doesn't seem so, the documentation oftentimes says "non-wild pathname").

Bonus if it is possible to use the double wildcard to traverse directories recursively (./**/*.jpg).

edit: I have tried variants of (directory #p"./**/*.jpg") and it returns nil :( Also tried #p".*jpg", #p"./.*jpg",…

(wild-pathname-p (pathname "*.jpg"))
(:WILD :WILD-INFERIORS)


(make-pathname :name :wild :type "jpg")
#P"*.jpg"

The following gets me files by jpg extension, but it isn't a proper wildcard yet:

(directory *)
(#P"/home/vince/cl-cookbook/AppendixA.jpg"
 #P"/home/vince/cl-cookbook/AppendixB.jpg"
 #P"/home/vince/cl-cookbook/AppendixC.jpg")

Documentation on pathnames and make-pathname: http://gigamonkeys.com/book/files-and-file-io.html (no mentions of wildcards)

What do you get if you do `(wild-pathname-p (pathname "*.jpg"))`? What about `(make-pathname :name :wild :type "jpg")`? — Dan Robertson, Feb 16 '18 at 18:59
I answered by updating my question. `(directory …)` returns nil, `(directory (make-pathnam … :type "jpg")` works. — Ehvince, Feb 16 '18 at 19:19
That will probably work for some pathnames. Pathnames are difficult in Lisp because the api is designed to be general enough to accommodate all the operating systems that were around in the mid 1980s but this has meant that every CL implementation of pathnames for unix is slightly different. I would still look for a portable way to do this if I were you. I don’t know of one. — Dan Robertson, Feb 16 '18 at 19:22
Which implementation? For example, SBCL supports stars in the middle of names, but this is an extension from the standard. — coredump, Feb 16 '18 at 19:24

coredump · Answer 1 · 2018-02-20T17:55:48.603

SBCL

SBCL supports wildcards in names. First, create some files:

(loop 
  with stem = #P"/tmp/stack/_.txt"
  initially (ensure-directories-exist stem)
  for name in '("abc" "def" "cadar" "cdadr" "cddr")
  for path = (make-pathname :name name :defaults stem)
  do (open path :direction :probe :if-does-not-exist :create))

Then, list all files that contains an "a":

CL-USER> (directory #P"/tmp/stack/*a*.txt")
(#P"/tmp/stack/abc.txt" #P"/tmp/stack/cadar.txt" #P"/tmp/stack/cdadr.txt")

The pathname contains an implementation-specific (valid) name component:

CL-USER> (describe #P"/tmp/stack/*a*.txt")
#P"/tmp/stack/*a*.txt"
  [structure-object]

Slots with :INSTANCE allocation:
  HOST       = #<SB-IMPL::UNIX-HOST {10000F3FF3}>
  DEVICE     = NIL
  DIRECTORY  = (:ABSOLUTE "tmp" "stack")
  NAME       = #<SB-IMPL::PATTERN :MULTI-CHAR-WILD "a" :MULTI-CHAR-WILD>
  TYPE       = "txt"
  VERSION    = :NEWEST
; No value

SBCL also defines sb-ext:map-directory, which process files one by one, instead of first collecting all files in a list.

Portable solutions

If you need to stick to standard pathname components, you can first call directory with normal wildcards, and filter the resulting list:

CL-USER> (remove-if-not (wildcard "*a*")
                        (directory #P"/tmp/stack/*.txt")
                        :key #'pathname-name)

(#P"/tmp/stack/abc.txt" #P"/tmp/stack/cadar.txt" #P"/tmp/stack/cdadr.txt")

... where wildcard might be based on regex (PPCRE):

(defun parse-wildcard (string)
  (delete ""
          (map 'list
               (lambda (string)
                 (or (cdr (assoc string
                                 '(("*" . :wild)
                                   ("?" . :char))
                                 :test #'string=))
                     string))
               (ppcre:split '(:sequence
                              (:negative-lookbehind #\\)
                              (:register (:alternation #\* #\?)))
                            string
                            :with-registers-p t))
          :test #'string=))

(note: the above negative lookbehind does not eliminate escaped backslashes)

(defun wildcard-regex (wildcard)
  `(:sequence
    :start-anchor
    ,@(loop
        for token in wildcard
        collect (case token
                  (:char :everything)
                  (:wild '(:greedy-repetition 0 nil :everything))
                  (t token)))
    :end-anchor))

(defun wildcard (string)
  (let ((scanner (ppcre:create-scanner
                  (wildcard-regex (parse-wildcard string)))))
    (lambda (string)
      (ppcre:scan scanner string))))

Intermediate functions:

CL-USER> (parse-wildcard "*a*a\\*a?\\?a")
(:WILD "a" :WILD "a\\*a" :CHAR "\\?a")

CL-USER> (wildcard-regex (parse-wildcard "*a*a\\*a?\\?a"))
(:SEQUENCE :START-ANCHOR #1=(:GREEDY-REPETITION 0 NIL :EVERYTHING) "a" #1# "a\\*a" :EVERYTHING "\\?a" :END-ANCHOR)

Thanks. However I can't get the following to return sthg: `(directory #P"/tmp/stack/.*a.*")`, and I tried with regexp variants. Looks like we must specify an extension, and not only `txt` but `.txt`, the following doesnt work either: `(directory #P"/tmp/stack/*a*txt")`. — Ehvince, Feb 17 '18 at 00:06
@Ehvince In case of doubt, use `describe` to see how strings are parsed into pathname objects. The first one looks for a file whose name component starts with a dot, ends with an "a" and whose type is wild. The syntax might not match the expectations we have when working with linux. — coredump, Feb 17 '18 at 07:06

Rainer Joswig · Answer 2 · 2018-02-17T09:15:06.180

no current directory and no home directory characters

The concept of . denoting the current directory does not exist in portable Common Lisp. This may exist in specific filesystems and specific implementations.

Also ~ to denote the home directory does not exist. They may be recognized by some implementations as non-portable extensions.

In pathname strings you have * and ** as wildcards. This works in absolute and relative pathnames.

defaults for the default pathname

Common Lisp has *default-pathname-defaults* which provides a default for some pathname operations.

Examples

CL-USER 46 > (directory "/bin/*")
(#P"/bin/[" #P"/bin/bash" #P"/bin/cat"   ....   )

Now in above it is already slightly undefined or diverging what implementations do on Unix:

resolve symbolic links?
include 'hidden' files?
include files with types?

CL-USER 47 > (directory "/bin/*sh")
(#P"/bin/zsh" #P"/bin/tcsh" #P"/bin/sh" #P"/bin/ksh" #P"/bin/csh" #P"/bin/bash")

Using a relative pathname:

CL-USER 48 > (let ((*default-pathname-defaults* (pathname "/bin/")))
               (directory "*sh"))
(#P"/bin/zsh" #P"/bin/tcsh" #P"/bin/sh" #P"/bin/ksh" #P"/bin/csh" #P"/bin/bash")

Files in your home directory:

CL-USER 49 > (let ((*default-pathname-defaults* (user-homedir-pathname)))
              (directory "*"))

The same:

CL-USER 54 > (directory (make-pathname :name "*"
                                       :defaults (user-homedir-pathname)))

Finding all files ending with sh in /usr/local/ and below:

CL-USER 54 > (directory "/usr/local/**/*sh")

Constructing pathnames with MAKE-PATHNAME

Three ways to find all .h files under /usr/local/:

(directory "/usr/local/**/*.h")

(directory (make-pathname :name :wild
                          :type "h"
                          :defaults "/usr/local/**/")

(directory
   (make-pathname :name :wild
                  :type "h"
                  :directory '(:ABSOLUTE "usr" "local" :WILD-INFERIORS)))

Problems

There are a lot of different interpretations of implementations across platforms ('windows', 'unix', 'mac', ...) and even on the same platform (especially 'windows' or 'unix'). Stuff like unicode in pathnames creates additional complexity - not describe in the CL standard.

We still have a lot of different filesystems ( https://en.wikipedia.org/wiki/List_of_file_systems ), but they are different or different in capabilities from what was typical when Common Lisp was designed. Implementations may have tracked the changes, but not necessarily in portable ways.

Thanks. Your examples involve a wildcard + a file extension. Do you have a way to search for files containing a string, without precising the extension ? — Ehvince, Feb 21 '18 at 12:41

How to search for files with a wildcard in Common Lisp?

2 Answers2

SBCL

Portable solutions

Linked