Webpack rebuild is slow using webpack-dev-middleware when building stylus

Question

I'm using webpack 2, with the webpack-dev-middleware

 {
    test: /\.styl$/,
    use: [
      'style-loader',
      'css-loader',
      {
        loader: 'stylus-loader',
        options: {
          use: [nib()],
        },
      },
    ]
  },
  {
    test: /\.jpe?g$|\.gif$|\.png$|\.svg$|\.woff$|\.ttf$|\.wav$|\.mp3$/,
    use: [ {
      loader: 'file-loader',
      options: {
        publicPath: '/dist/'
      }
    } ]
  }

Relevant server code:

let compiler = webpack(webpackConfig);
app.use(webpackDevMiddleware(compiler, {
  publicPath: ('/dist/')
}));

On the initial build (either using webpack from the console, or on first running my server) the whole process takes ~2000ms. Changing .js files doesn't take long at all (<200ms), but changing stylus files takes a very long time (>90s). If I make a change to the stylus then manually trigger a rebuild with webpack it's really quick, but ideally I'd like the watch to do its magic and for my css to be reloaded in place quickly...

Any ideas why the rebuild is taking such a long with stylus, or how I can debug the issue?