1

My question might be a duplicate of Shake depending on on node_modules directory

I'm new to shake-build and got stumped with the very first thing that I wanted to do. Basically, I want to run yarn install whenever the yarn.lock OR package.json file changes.

I tried the following first, but it ended up running always:

want ["foo"]

phony "foo" $ do
  need ["yarn.lock", "package.json"]
  cmd_ ["yarn", "install"]

Then I tried the following:

  want ["yarn.lock"]

  "yarn.lock" %> \out -> do
    need ["package.json"]
    cmd_ ["yarn", "install"]

And it seems to work, but not quite. It re-runs only when yarn.lock or package.json changes, but fails to run after an rm -rf node_modules (sadly, a very common occurence, btw!)

Is Shake depending on on node_modules directory the correct way to set this up? Does one have to resort to these fake files for all such cases? Shouldn't this be automagically handled somehow by the shake database?

Edit: As an extension to this question, what's the right way to setup a common rule that can do a yarn install in any given directory?

Saurabh Nanda
  • 6,373
  • 5
  • 31
  • 60

1 Answers1

1

For the first snippet, a phony rule is always rerun, so there's no point doing a need within it (unless it needs those outputs to do its job) - which explains why it was always rerun.

For the second snippet, you have said to build yarn.lock depend on package.json. It has no idea that the contents of node_modules are the important ones. To capture that, and be tolerant to node_modules being deleted, the stamp file approach works well.

The reason stamp files are necessary is that directories aren't really a tracked concept in Shake - they don't have identity, a good modification time, they don't have contents, they come and go freely in the build system. Adding a dependency on a directory always goes wrong sooner or later (Shake actually used to allow it).

For a common reusable rule, one approach is to encode the directory name in the stamp file (which I've named .yarn_stamp). For example:

"*.yarn_stamp" *> \out -> do
    let name = dropDirectory out
    need ["package.lock", name </> "package.json"]
    cmd_ "yarn install" (Cwd name)
    writeFile' out ""

As to whether this could be handled for you by Shake, very little goes deep into the Shake database - even the notion of files themselves are wrapped at a higher level. I looked at whether we could wrap the directory creation pattern, and it seems we're one simple function away from doing it - see https://github.com/ndmitchell/shake/issues/759 for that feature request (and please weigh in on the ticket if it seems useful for you).

Neil Mitchell
  • 9,090
  • 1
  • 27
  • 85
  • So, IIUC, the rule should depend on THREE things to be correct - `package.json`, `yarn.lock`, AND `node_modules/`. But since this doesn't fit the model of "compile X to produce Y" (and therefore Y is the build target), we need to introduce a fake build target in the form of `.stamp` files. Am I right? – Saurabh Nanda Jun 03 '20 at 04:21
  • And the fake `.stamp` files are treated as the build target and they in-turn depend on the three things. But, do they? Will the `.stamp` approach work if I do `rm -rf node_modules/`. Will shake be able to detect the missing package and re-run `yarn install`? – Saurabh Nanda Jun 03 '20 at 04:24
  • Actually, now that I think of it, a better mental model is that `yarn.lock` actually determines what directories should be present **inside** `node_modules`. And `package.json` determines the contents of `yarn.lock`. Is this something that can be modelled in `shake`? – Saurabh Nanda Jun 03 '20 at 04:30
  • Reading the manual more carefully now - specifically https://shakebuild.com/manual#directory-listing-dependencies - I'm wondering if this could be setup by adding dependencies on `yarn.lock`, `package.json` AND all files inside `node_modules`. This would cover the following scenarios: (a) package is added/changed in package.json => will trigger the build (b) directory is removed from `node_modules` => will trigger the build. Is depending on a LARGE number of files in a directory problematic? Will it cause perf issues? – Saurabh Nanda Jun 03 '20 at 04:33
  • Most rules need to produce something, a specific named something. If every node_modules compilation produced a known `node_output_lump.js` or similar then that would work instead of a stamp file. Your final comment about depending on the package.json, yarn.lock and the outputs is probably a reasonable thing to do, and works if you delete node_modules, however, the default directory listing functions assume they are working on source files. Depending on a large number of files/dirs will cause perf issues if it's enough, but it can cope with 10K, so probably not an issue. – Neil Mitchell Jun 04 '20 at 21:27