How to make defsystem use "everything"?

Question

I am working on project euler problems in SBCL and keep a short file for every solution. Every problem has some 5am-based tests, which are referenced from a "main" test suite. These tests are run when "tests.lisp" is run. Since I got bored with maintaining the list of files by hand, I wrote some code to do it for me:

(defpackage #:euler/asdf
  (:use :cl :asdf))
(in-package #:euler/asdf)

;; directory containing the problem files
(defparameter +dir+ "/home/stefan/quicklisp/local-projects/euler")

;; build file list for package components
(defun files-for-problems (dir)
  (mapcar #'(lambda (p) (list :file (pathname-name p) :depends-on '("package")))
      (directory (concatenate 'string dir "/e????.lisp"))))

;; build dependency list for all tests component
(defun depends-on-problems (dir)
  (mapcar #'pathname-name
      (directory (concatenate 'string dir "/e????.lisp"))))

;; define euler system
(defsystem euler
    :name "euler"
    :author "Stefan Schmiedl"
    :description "Solutions to problems at http://projecteuler.net"
    :depends-on ("iterate" "fiveam" "cl-csv")
    :components #.`((:file "package")
            ,@(files-for-problems +dir+)
         #.`(:file "tests" :depends-on ,(depends-on-problems +dir+))))

In short, defsystem euler uses all e????.lisp files as components and tests.lisp depends on all of these files.

Is this a good idea? Is there an "official" way to make defsystem use all files in a directory or all files matching a given filename pattern?

I feel like I'm missing something elementary here, especially after reading some ELS slides on github about a "more declarative defsystem" where the thing I've done above would probably be frowned upon.

After some fiddling with Fare's suggestion here is what I now have:

;; define private package for defsystem
(defpackage #:euler-system
  (:use :cl :uiop :asdf))
(in-package #:euler-system)


;; define euler system
(defsystem "euler"
  :author "Stefan Schmiedl"
  :description "Solutions to problems at http://projecteuler.net"
  :depends-on ("iterate" "fiveam" "cl-csv")
  :components ((:module "package"
                        :pathname ""
                        :components ((:file "package")))
               (:module "problems"
                        :pathname ""
                        :depends-on ("package")
                        :components #.(mapcar #'(lambda (p) (list :file (pathname-name p)))
                                              (directory-files (pathname-directory-pathname
                                                                (uiop/lisp-build:current-lisp-file-pathname))
                                                               "e*.lisp")))
               (:module "tests"
                        :pathname ""
                        :depends-on ("package" "problems")
                        :components ((:file "tests")))))

Thanks for the feedback.

FWIW, I would discourage putting your tests in your system in general: someone using your Euler system might well want to run it without having to install and load your test library. (I realize that for a project Euler exercise, this concern may not apply, which is why I said "in general.") For that case, I would add an "euler/test" system to replace your module, and add an in-order-to to run the test-op on euler, you need to run test-op on euler/test. We have a FiveAM-asdf add on to make it easier to specify such TEST-OPs. — Robert P. Goldman, Jun 13 '14 at 19:40
That is, in fact, on my schedule once I've reached problem 100 :-) — Stefan Schmiedl, Jun 15 '14 at 07:50

score 6 · Accepted Answer · answered Jun 12 '14 at 03:27

For the directory part, I recommend using relative pathnames. You could do it several ways.

1- Thou shalt not use an absolute pathname. Use relative pathname like that, possibly via a variable: (subpathname (current-file-pathname) #p"e????.lisp")

2- I'm not sure how portable ? is as a wildcard character — if you can live with it, * is much more portable.

3- uiop:directory-files is safer than cl:directory in this and many contexts.

4- for an "official" way of handling wildcard patterns without #. or (eval `...), get inspiration from asdf/contrib/wild-modules.lisp — that said, for a one-off, #. is totally acceptable, especially since we're so far from purely declarative .asd files.

5- for grouped dependencies, you could use

(defsystem "euler"
  :depends-on ("iterate" "fiveam" "cl-csv")
  :serial t
  :components
   ((:module "package" :pathname ""
       :components ((:file "package")))
    (:module "problems" :pathname "" :depends-on ("package")
       :components #.(mapcar ...))
    (:module "tests" :pathname ""
       :components ((:file "tests")))))

6- Instead of modules, you could be using secondary systems, at which point system-relative-pathname would be available:

(defsystem "euler" :depends-on ("euler/tests"))
(defsystem "euler/tests"
  :depends-on ("euler/package")
  :components ((:file "package")))
(defsystem "euler/problems"
  :depends-on ("euler/package")
  :components
    #.(mapcar ... (directory-files (system-relative-pathname "euler" #p"e*.lisp")))))
(defsystem "euler/tests"
  :depends-on ("euler/problems")
  :components ((:file "tests")))

7- In the above I assume asdf3, and that you use uiop without prefix:

(defpackage :euler-system (:use :cl :uiop :asdf))
(in-package :euler-system)

If you don't define any function or variable or class, you could directly (in-package :asdf)

I'm glad you enjoyed my talk at ELS 2013. I gave another one at ELS 2014, in the same repository.

The 2014 talk is where I got the repo URL from, I watched the video of your presentation and have the extended version really close to the top of my reading list. Thank you for your work. — Stefan Schmiedl, Jun 12 '14 at 05:38
I'm going to switch to modules (your #5), as it removes the duplication in my code. Also, the second defsystem in your #6 seems to suffer from copypasteritis ;-) — Stefan Schmiedl, Jun 12 '14 at 05:57
If a module has only one file, it needn't be a module, actually — just use the :file. My bad. Also, you needn't qualify something with uiop/foo: if you're already :use'ing :uiop. Lastly, since you don't define any function or variable, you can just (in-package :asdf) instead of creating your own package. — Faré, Jun 13 '14 at 21:56

score 3 · Answer 2 · answered Jun 11 '14 at 20:05

ASDF provides three built-in component-types, you're using the simple :file component type only in your system definition. Typically for grouping some files together, one would introduce separate modules (which pretty much directly translate to different directories), but modules still require you to specify the (sub-)components and then you're back to where you started. I looked briefly if there would be an ASDF extension that supports the functionality you have built, but haven't found anything. So, while there might be some minor problems with your code (such as the wildcard syntax probably not being portable across implementations), your general approach looks fine to me.

To address your second question of whether this is a good idea: Looking at make's implicit rules, I think that having something like this might be useful. However, more often than not you have dependencies between the various files and as soon as you need to specify these, you're basically back to having to list the components and their dependencies. The entire idea of defsystem(s) is to be able to specify dependencies and required serializations. So your use case might not be too common which might explain why you don't find an easily provided solution for this.

The plan is to make a "library" module and have the solution files depend on that. I don't think that this practice project warrants the effort for finely tuned dependencies. — Stefan Schmiedl, Jun 11 '14 at 20:10

How to make defsystem use "everything"?

2 Answers2