5

I have a monorepo with a very basic setup available for reproducing this issue here:

It is a single nestjs app with 2 packages that it reads from.

@nestjs/core among other dependencies is needed for both the packages and the main app to work, and it is enforced to be the exact same fixed version not only on their own local package.json's but also with the resolutions {} config in the main package.json.

I can inspect the lockfile and find out that although the same version is used -- the hashes are different, causing major issues with nestjs, not being able to import injectable dependencies reliably causing it to break on bootstrap.

Is there a way to prevent this? to force linking the exact same hash/dependency?

SebastianG
  • 8,563
  • 8
  • 47
  • 111

3 Answers3

2

When a dependency has peer dependencies, it might be written to node_modules several times if the peer dependencies are resolved differently in various parts of the dependency graph.

In your case, @nestjs/core is in the dependencies of the graphql-server project and the @myapp/entities project. @nestjs/core has @nestjs/platform-express as an optional peer dependency.

@nestjs/platform-express is in the dependencies of the graphql-server project, so pnpm links it to @nestjs/platform-express. You can see it in the lockfile:

  /@nestjs/core/8.4.7_fkqgj3xrohk2pflugljc4sz7ea:
    resolution: {integrity: sha512-XB9uexHqzr2xkPo6QSiQWJJttyYYLmvQ5My64cFvWFi7Wk2NIus0/xUNInwX3kmFWB6pF1ab5Y2ZBvWdPwGBhw==}
    requiresBuild: true
    peerDependencies:
      '@nestjs/common': ^8.0.0
      '@nestjs/microservices': ^8.0.0
      '@nestjs/platform-express': ^8.0.0
      '@nestjs/websockets': ^8.0.0
      reflect-metadata: ^0.1.12
      rxjs: ^7.1.0
    peerDependenciesMeta:
      '@nestjs/microservices':
        optional: true
      '@nestjs/platform-express':
        optional: true
      '@nestjs/websockets':
        optional: true
    dependencies:
      '@nestjs/common': 8.4.7_47vcjb2de6lyibr6g4enoa5lyu
      '@nestjs/platform-express': 8.4.7_7tsmhnugyerf5okgqzer2mfqme # <------HERE
      '@nuxtjs/opencollective': 0.3.2
      fast-safe-stringify: 2.1.1
      iterare: 1.2.1
      object-hash: 3.0.0
      path-to-regexp: 3.2.0
      reflect-metadata: 0.1.13
      rxjs: 7.5.5
      tslib: 2.4.0
      uuid: 8.3.2
    transitivePeerDependencies:
      - encoding

In the other project (@myapp/entities), @nestjs/platform-express is not in the dependencies, so when installing @nestjs/core, pnpm cannot resolve the optional peer dependency. As a result, pnpm needs to create another instance of @nestjs/core, which doesn't have this optional peer linked in. As you can see in the lockfile, the other entry doesn't have @nestjs/platform-express:

  /@nestjs/core/8.4.7_g7av3gvncewo44y4rurz3mgav4:
    resolution: {integrity: sha512-XB9uexHqzr2xkPo6QSiQWJJttyYYLmvQ5My64cFvWFi7Wk2NIus0/xUNInwX3kmFWB6pF1ab5Y2ZBvWdPwGBhw==}
    requiresBuild: true
    peerDependencies:
      '@nestjs/common': ^8.0.0
      '@nestjs/microservices': ^8.0.0
      '@nestjs/platform-express': ^8.0.0
      '@nestjs/websockets': ^8.0.0
      reflect-metadata: ^0.1.12
      rxjs: ^7.1.0
    peerDependenciesMeta:
      '@nestjs/microservices':
        optional: true
      '@nestjs/platform-express':
        optional: true
      '@nestjs/websockets':
        optional: true
    dependencies:
      '@nestjs/common': 8.4.7_47vcjb2de6lyibr6g4enoa5lyu
      '@nuxtjs/opencollective': 0.3.2
      fast-safe-stringify: 2.1.1
      iterare: 1.2.1
      object-hash: 3.0.0
      path-to-regexp: 3.2.0
      reflect-metadata: 0.1.13
      rxjs: 7.5.5
      tslib: 2.4.0
      uuid: 8.3.2
    transitivePeerDependencies:
      - encoding

To solve this, you can add @nestjs/platform-express to the dependencies of the @myapp/entities project. It should be the same version as in the other project.

Zoltan Kochan
  • 5,180
  • 29
  • 38
  • Hi Zoltan, is there other way to solve this issue? We are facing with a kind of similar problem in a workspace. All packages use `@mui/x-date-pickers`, but it has a longer list of optional peerDependencies. Some of the peerDependencies are used in some of our packages, but not in others. Adding "unused" dependencies to the package.json seems to be a too fragile solution, because these deps, without an explicit import in the codebase, might be removed later, accidentally. I wonder if there's an other solution, maybe using `hooks.afterAllResolved` in .pnpnfile.cjs, do you think this is feasible? – bencergazda Nov 30 '22 at 16:04
  • fwiw my issue was more of a critical incompatibility with turborepo & the way it handles dependencies together with pnpm -- for this reason and a dozen other major pain points we switched our project to NX and haven't had an issue in months. – SebastianG Dec 01 '22 at 00:41
  • Zoltan adding/tracking optional dependencies could work for small projects but it is not manageable for large projects. We switched to pnpm solely for the purpose of ensuring a single package/instance (that @nestjs requires). Is there any other way to enforce it for the same version? (I tried "overrides" but that didn't help either). – Andrej K Dec 01 '22 at 05:54
  • You can try to set the `auto-install-peers=true` setting. This will tell pnpm to automatically install missing peer dependencies. I believe it should reduce the amount of issues. – Zoltan Kochan Dec 01 '22 at 13:12
  • Zoltan `auto-install-peers` helps resolve the warning during install but not the issues of multiple package copies of the same version. I'm asking whether there is way to enforce single version & single copy in monorepo project (without manually chasing all nested optional peer dependencies) or at least a way to identify the root/chain why duplicate copies are created. – Andrej K Dec 04 '22 at 22:19
  • The copies are created because these package have peer dependencies and these peer dependencies are resolved from different versions. You can use overrides to force a single version for the peer dependencies. Overrides: https://pnpm.io/package_json#pnpmoverrides – Zoltan Kochan Dec 05 '22 at 23:07
  • The copies are created because there are optional peers installed only in one of the projects (not because there's different versions) and overrides doesn't seem to have any effect preventing multiple copies of the same version. – Andrej K Dec 06 '22 at 18:06
  • There is no automatic way at the moment. There will be probably because this seems like a problem to many. – Zoltan Kochan Dec 07 '22 at 08:48
2

Edit:

With pnpm v7.29.0, you no longer have to perform the hack described below, but just leaving it here for educational purposes.

Now the solution is just to set dedupe-peer-dependents=true (e.g. in your .npmrc).


From pnpm docs

- foo-parent-1
  - bar@1.0.0
  - baz@1.0.0
  - foo@1.0.0
- foo-parent-2
  - bar@1.0.0
  - baz@1.1.0
  - foo@1.0.0

In the example above, foo@1.0.0 is installed for foo-parent-1 and foo-parent-2. Both packages have bar and baz as well, but they depend on different versions of baz. As a result, foo@1.0.0 has two different sets of dependencies: one with baz@1.0.0 and the other one with baz@1.1.0. To support these use cases, pnpm has to hard link foo@1.0.0 as many times as there are different dependency sets.

For your specific case, foo === @nestjs/core, baz === @nestjs/microservices. Although the example used here is for "different versions", the same applies for optional peer dependencies. So to re-illustrate the example, in your context:

- my-nestjs-app
  - @nestjs/microservices@9.1.4
  - @nestjs/core@9.1.4
- my-other-nestjs-app
  - @nestjs/core@9.1.4

Normally, if a package does not have peer dependencies, it is hard linked to a node_modules folder next to symlinks of its dependencies, like so:

However, if foo [@nestjs/core] has peer dependencies, there may be multiple sets of dependencies for it, so we create different sets for different peer dependency resolutions

^ This is usually ok for most packages out there. However @nestjs/core is special. It's stateful so that it can take care of all the runtime dependency injections. pnpm creating multiple copies of @nestjs/core in a monorepo will result in the confusing behaviour you're seeing, as your app could depend on 1 copy, while other NestJS libs depend on another. This seems like a common problem felt by devs using pnpm + nest, according to the NestJS discord.

Solution

Use pnpm hooks to modify nestjs packages' peerDependenciesMeta at resolution time:

// .pnpmfile.cjs in your monorepo's root

function readPackage(pkg, context) {
  if (pkg.name && pkg.name.startsWith('@nestjs/')) {
    context.log(`${pkg.name}: make all peer dependencies required`);
    pkg.peerDependenciesMeta = {}; 
  }
  return pkg;
}

module.exports = {
  hooks: {
    readPackage,
  }
};

This is a hack IMO, and it's really annoying to deal with because Renovate / Dependabot will ignore the .pnpmfile.cjs when it performs dependency updates. I'd suggest going with Nx or some other package manager that Nest / stateful packages work better with.

Eli
  • 36
  • 1
  • 3
  • That seems quite well thought out, unfortunately I moved away from using the tech stack that caused this issue and can't confirm if it works. If anyone can confirm, happy to mark this as the right answer. – SebastianG Feb 10 '23 at 13:08
  • With [pnpm v7.29.0](https://github.com/pnpm/pnpm/releases/tag/v7.29.0), you no longer have to do this hack – Eli Mar 07 '23 at 01:41
0
  1. Set use-lockfile-v6=true in your .npmrc

  2. Then pnpm install to get the new lockfile

  3. Then analyze the top of the pnpm-lock.yaml to see which packages (yours) of your monorepo appear with different versions.

It's a bit a manual work but starts from top to bottom and look into the importers: block. It's all the direct dependencies you listed into your package.json files.

Stop at each dependency that looks with a suffix, like version: 0.31.1(react@18.2.0) (note that something with no risk of duplication would be version: 0.31.1).

In my case for example I found for my packageA:

      '@mui/material':
        specifier: ^5.10.16
        version: 5.10.16(@emotion/react@11.10.5)(@emotion/styled@11.10.5)(@types/react@18.0.26)(react-dom@18.2.0)(react@18.2.0)

Then copy into the clipboard '@mui/material': and search for all occurrences into the importer: section. If you find a different pattern of version it means it's deduplicated, for example I had for my packageB:

      '@mui/material':
        specifier: ^5.10.16
        version: 5.10.16(@types/react@18.0.26)(react-dom@18.2.0)(react@18.2.0)

In my case my packageA has @emotion/* dependencies that are not specified into packageB, making them a mismatch since @mui/material lists them as peerDependencies. Since I was no longer needing @emotion/* dependencies I just deleted them. Then pnpm install and there is only now version: 5.10.16(@types/react@18.0.26)(react-dom@18.2.0)(react@18.2.0). Meaning it will uses the exact same package for both packageA and packageB.

In case I would have needed @emotion/* in packageA I would have been able to add them into packageB, it would make the same result of merging the dependency. The idea is just to fix the versions of peerDependencies of my different @mui/material so they can match and be merged. It's explained in https://pnpm.io/how-peers-are-resolved .

I think it can also help to make sure YOUR direct dependencies use same version everywhere in the monorepo (it's a first step before aligning peerDependencies. For this I use into my root package.json:

  "pnpm": {
    "overrides": {
      "@mui/material": "^5.10.16",
      "react": "^18.2.0",
      "react-dom": "^18.2.0",
    }
  },

(repeat the process for each dependency that you don't want to see "duplicated"...)

(note that it's from my own experience, it's maybe not perfect, but it helped me at least ^^...)

Thomas Ramé
  • 438
  • 4
  • 10