Dynamically generating Rules from the content of an (Action a)

Question

I'm currently testing porting our build system from make to shake and have hit a roadblock:

Given the following project structure:

static/a.js
static/b.coffee

build/a.js
build/b.js

That is, various input extensions map to identical output extensions, so a straightforward "build//*.js" %> rule isn't going to work.

I wanted to avoid using priority if possible, and writing an ad-hoc build rule that checks for the existence of either possible input felt clunky (especially since this situation occurs with other filetypes as well), so I wrote the following:

data StaticFileMapping a = StaticFileMapping String String (FilePath -> FilePath -> Action a)

staticInputs :: FilePath -> StaticFileMapping a -> Action [FilePath]
staticInputs dir (StaticFileMapping iExt _ _) = (findFiles (dir </> "static") [iExt])

staticInputToOutput :: StaticFileMapping a -> FilePath -> FilePath
staticInputToOutput (StaticFileMapping _ oExt _) = (remapDir ["build"]) . (-<.> oExt)

staticTargets :: FilePath -> StaticFileMapping a -> Action [FilePath]
staticTargets dir sfm = (map $ staticInputToOutput sfm) <$> staticInputs dir sfm

rules :: FilePath -> StaticFileMapping a -> Rules ()
rules dir sfm@(StaticFileMapping _ _ process) = join $ mconcat . (map buildInputRule) <$> staticInputs dir sfm
    where buildInputRule :: FilePath -> Rules ()
          buildInputRule input = (staticInputToOutput sfm input) %> (process input)

That way I can define a mapping for each input type (.coffee -> .js, .svg -> .png) and so on, with only a tiny amount of code implementing the transformation for each. And it almost works.

But it seems impossible to go from (Action a) to Rules _ without throwing the value inside the Action away first, as far as I can tell.

Is there a function with type (Action a) -> (a -> Rules ()) -> Rules () or (Action a) -> (Rules a)? Can I implement either one myself, or do I need to modify the library's code?

Or is this entire approach hare-brained and I should take some other route?

Answer 1

First off, using priority would not work, as that picks a rule statically then runs it - it doesn't backtrack. It's also important that Shake doesn't run any Action operations to produce Rules (as per the two functions you propose) since the Action might call need on a Rule that it itself defines, or is defined by another action rule, thus making the ordering of those Action calls visible. You could add IO (Rules ()) -> Rules (), which might be enough for what you are thinking of (directory listing), but it isn't currently exposed (I have an internal function that does exactly that).

To give a few example approaches it's useful to define plausible commands to convert .js/.coffee files:

cmdCoffee :: FilePath -> FilePath -> Action ()
cmdCoffee src out = do
    need [src]
    cmd "coffee-script-convertor" [src] [out]

cmdJavascript :: FilePath -> FilePath -> Action ()
cmdJavascript = copyFile'

Approach 1: Use doesFileExist

This would be my standard approach, writing something like:

"build/*.js" %> \out -> do
    let srcJs = "static" </> dropDirectory1 out
    let srcCf = srcJs -<.> "coffee"
    b <- doesFileExist srcCf
    if b then cmdCoffee srcCf out else cmdJavascript srcJs out

This accurately captures the dependency that if the user adds a .coffee file in the directory then the rule should be rerun. You could imagine sugaring up the doesFileExist if this is a common pattern for you. You could even drive it from you list of StaticFileMapping structures (do a group on the oExt field to add one rule per oExt than calls doesFileExists on each iExt in turn). An advantage of this approach is that if you do shake build/out.js it doesn't need to do a directory listing, although likely that cost is negligible.

Approach 2: List the files before calling shake

Instead of writing main = shakeArgs ... do:

import System.Directory.Extra(listFilesRecursive) -- from the "extra" package

main = do
    files <- listFilesRecursive "static"
    shakeArgs shakeOptions $ do
        forM_ files $ \src -> case takeExtension src of
            ".js" -> do
                 let out = "build" </> takeDirectory1 src
                 want [out]
                 out %> \_ -> cmdJavascript src out
            -- rules for all other types you care about
            _ -> return ()

Here you operate in IO to get the list of files, then can add rules by referring to that previously captured value. Adding rulesIO :: IO (Rules ()) -> Rules () would allow you to list the files inside shakeArgs.

Approach 3: List the files inside the rules

You can define a mapping between file names and outputs, driven from the directory listing:

buildJs :: Action (Map FilePath (Action ()))
buildJs = do
    js <- getDirectoryFiles "static" ["*.js"]
    cf <- getDirectoryFiles "static" ["*.coffee"]
    return $ Map.fromList $
        [("build" </> j, cmdJavascript ("static" </> j) ("build" </> j)) | j <- js] ++
        [("build" </> c, cmdCoffee ("static" </> c) ("")) | c <- cf]

Then lift that into a set of rules:

action $ do
    mpJs <- buildJs
    need $ Map.keys mpJs
"//*.js" %> \out -> do
    mpJs <- buildJs
    mpJs Map.! out

However, that recomputes the directory listing for every file we build, so we should cache it and make sure it's only computed once:

mpJs <- newCache $ \() -> buildJs
action $ do
    mpJs <- mpJs ()
    need $ Map.keys mpJs
"//*.js" %> \out -> do
    mpJs <- mpJs ()
    mpJs Map.! out

This solution is probably closest to your original approach, but I find it the most complex.