Is it possible to depend on an in-memory value (instead of a file) in shake-build?

Question

I'm writing a static site builder using shake and am trying to implement an efficient way of persisting the site's navigation (along with some metadata for each page). This is turning out to be more complex than I imagined:

• The site-metadata is a tree data-structure where each node in the tree is a permalink, along with the page's metadata, like title, short blurb, author, etc.
• This site-metadata is used by almost all pages to render a site-wide navigation (either in the header or the sidebar).
• Whenever a page is modified, its build rule modifies its own node in the site-metadata.
• The complete site-metadata is stored in a file (as opposed to individual file's for each page's metadata)

If the site-metadata is updated at the end of each page's build action, it will result in the the site-metadata being deserialised, updated, serialised, and written to disk, potentially hundreds of times (once for oeach build-rule that is executed).

Instead, I was thinking of reading the site-metadata into an IORef at the very top, and updating the in-memory IORef at the end of each page's build-action.

However, this can lead to two problems:

• how do I make the sitemap.xml rule depend on this IORef and how do I ensure that it runs at the very end?
• how do I make each page depend on this IORef because there is actually a cyclic-dependency here - each page, both, depends and updates the site-metadata.

If I've made this more complex that it needs to be, what is an alternate, but efficient way of implementing this?

Answer 1

Firstly, before optimising, it might be worth double checking your assumption that the serialisation etc is expensive. In many cases, it isn't that problematic. But let's proceed assuming it is too expensive.

The biggest problem is the cyclic dependency. If you can design that out, there are quite a few solutions in Shake that can avoid repeated serialisation etc. I'd instead consider designing the dependencies as:

1. Each page produces its metadata individually.
2. A single rule that collects all the metadata from all pages, that depends on each rule in step 1.
3. Each page depends on its own metadata plus the collected/aggregated metadata as per 2.

Now you have eliminated the cycle. You also cause rule 2 to aggregate/serialise only once.

But there's still a concern you are deserialising multiple times, and that can be eliminated with newCache which can introduce a dependency on a deserialised form of a file - the rules still depend on the file, but via an applied transformation.

As a final note, if going with the above formulation, you might want step 2 to produce the sidebar/topbar contents directly, and then all work of rendering those features can be shared too.