How To Process Inputs and Outputs in Separate Directories in Rake?

Question

I am trying to convert Haml files to HTML files using Rake, so that if the Haml files have not changed, I do not re-generate existing HTML.

However, I do not want my input files, output files, and Rakefile all smushed into one directory. Instead, I want my input files to be in src/ and my output files to be in build/output/. So, I want to start with:

Rakefile
src/
  slides.haml

and I want to end with:

Rakefile
src/
  slides.haml
build/
  output/
    slides.html

I have tried several bits of Rake code, with no luck.

First, I tried the caveman approach, hard-coding the precise files I am trying:

task "build/output/slides.html" => "src/slides.haml" do
  touch task.name
end

task :slides => "src/slides.haml"
task :default => "slides"

Running rake --trace results in:

** Invoke default (first_time)
** Invoke slides (first_time)
** Invoke src/slides.haml (first_time, not_needed)
** Execute slides
** Execute default

The directory is not built, and I do not have an empty build/output/slides.html file.

Ideally, I'd prefer a rule-and-FileList approach, using stuff like this SO answer or this "Rake cheatshet" entry. However, I tried a few variations on that theme, and I get the same results, suggesting that I have something more profoundly messed up. I have trimmed my Rakefile back to the caveman approach, just to try to grok what's going on here.

Why is Rake not recognizing my "build/output/slides.html" task? Is there some magic to using subdirectories (input or output) in Rake?

zanedp · Answer 1 · 2019-05-16T14:21:31.177

Here's something I put together today that I believe accomplishes the same thing you were trying to accomplish. First, I define mv_rule (can be in a separate ruby file that you import/require), then define a directory task for the output directory, and finally make a rule that describes the output pattern (including directory) and input pattern (including directory).

# Declare a rule for auto-tasks that have input and output in different directories
#
# Example:
#  mv_rule 'obj/*.o' => 'src/*.c' do |t|
#    sh "gcc -c #{t.source} -o #{t.name}"
#
def mv_rule(*args, &block) # :doc:
  outFile, inFile = args[0].keys[0], args[0].values[0]
  outExt = File.extname outFile
  outDir = File.dirname outFile
  inExt = File.extname inFile
  inDir = File.dirname inFile
  destPattern = Regexp.new("#{outDir}/.*\\#{outExt}")
  srcPattern = ->(o){o.pathmap("%{^#{outDir}/,#{inDir}/}X#{inExt}")}
  Rake::Task.create_rule(destPattern => [srcPattern, outDir], &block)
end

directory "build/output"

mv_rule 'build/output/*.html' => 'src/*.haml' do |t|
    touch t.name
end

task :all => "build/output/slides.html"

task :default => [:all]

I can run it with rake to execute the :default task. Or, since I specified a general rule for making html from haml in the appropriate directories, I can run it with rake build/output/more_slides.html (assuming there's a src/more_slides.haml).

I kind of surprised there isn't a built-in way to specify a different output directory. It seems like a pretty common scenario.

I made my mv_rule to make a general solution to the problem with a pretty straight-forward syntax. It probably has limitations I haven't even thought about, yet. I used http://www.virtuouscode.com/2014/04/23/rake-part-3-rules/ and http://www.virtuouscode.com/2014/04/24/rake-part-4-pathmap/ for inspiration.

score 0 · Answer 2 · answered Nov 08 '15 at 20:10

First of all, tasks are declared as task :name => dependency. Your :slides task names a source instead of an output dependency. Also, your :default task depends on a file "slides" instead of the task :slides. Change those lines to be

task :slides => "build/output/slides.html"
task :default => :slides

Next, the block of code that will actually execute should be a rule, not a task, and you can access the internal information of the rule via #{t.<param>}:

rule "build/output/slides.html" => "src/slides.html" do |t|
  sh "mkdir -p $(dirname #{t.name})"
  sh "touch #{t.name}"
end

There are better ways of making the rule more generic, building up lists of source files and mapping them to outputs, etc. Some useful tutorials are here https://github.com/ruby/rake#presentations-and-articles-about-rake

There are probably also more Ruby-esque ways of making the build/output/ directory if it does not exist, rather than invoking a shell to do it.

Thanks for this! I'll grant you the bounty, mostly because you triggered me (while trying to make sense of where I was going wrong with respect to your answer) to notice exactly where my caveman approach had gone awry. — CommonsWare, Nov 08 '15 at 20:38

score 0 · Accepted Answer · answered Nov 08 '15 at 20:36

I was missing that file tasks use file instead of task, and that there is a directory task as well.

The working version of the caveman approach is:

namespace :build do
  directory "build/output"

  file "build/output/slides.html" => ["build/output", "src/slides.haml"] do |t|
    touch t.name
  end

  task :all => "build/output/slides.html"
end

task :default => ['build:all']

Steve · Answer 4 · 2019-10-01T22:40:31.110

The non-caveman tool for this job is pathmap.

SOURCES = FileList['src/*.haml']

directory 'build/output'

rule '.html' do |t|
  sh 'touch', t.name
end

task :default => ['build/output', SOURCES.pathmap('build/output/%n.html')]

Output:

$ mkdir src && touch src/slides.haml
$ rake --trace
** Invoke default (first_time)
** Invoke build/output (first_time)
** Execute build/output
mkdir -p build/output
** Invoke build/output/slides.html (first_time)
** Execute build/output/slides.html
touch build/output/slides.html
** Execute default

For this kind of task, you might also be interested in rake/clean for generating tasks to restore your project to a pristine state:

require 'rake/clean'
CLOBBER.include 'build' # running 'rake clobber' will remove the build directory

How To Process Inputs and Outputs in Separate Directories in Rake?

4 Answers4