Hash mismatch on Angular 16 Migration and switching from Webpack to Vite

Question

After Updating to Angular 16 (from 15) and Switching from Webpack to Vite

   "architect": {
        "build": {
          "builder": "@angular-devkit/build-angular:browser-esbuild",

My Service Worker gets and "Version_Installation_Failed" Error for all my assets after I deploy the app via firebase hosting:

"Hash mismatch (cacheBustedFetchFromNetwork): https://myUrl.com/favicon.ico: expected 27d22f56faabc6ede22b52aad474a2806263ea9e, got a1b8ccd7e91b162f833472126c73929012a05ca7 (after cache busting)
Error: Hash mismatch (cacheBustedFetchFromNetwork): https://myUrl.com/favicon.ico: expected 27d22f56faabc6ede22b52aad474a2806263ea9e, got a1b8ccd7e91b162f833472126c73929012a05ca7 (after cache busting)
    at PrefetchAssetGroup.cacheBustedFetchFromNetwork (https://myUrl.com/ngsw-worker.js:474:21)

This is my current ngsw-config.json:

{
  "$schema": "./node_modules/@angular/service-worker/config/schema.json",
  "index": "/index.html",
  "assetGroups": [
    {
      "name": "app",
      "installMode": "prefetch",
      "resources": {
        "files": ["/favicon.ico", "/index.html", "/manifest.json", "/*.css", "/*.js"]
      }
    },
    {
      "name": "assets",
      "installMode": "lazy",
      "updateMode": "prefetch",
      "resources": {
        "files": ["/assets/**", "/*.(eot|svg|cur|jpg|png|webp|gif|otf|ttf|woff|woff2|ani)"]
      }
    }
  ]
}

I already tried to deploy the app without the ngsw-config or deleting all assets + deploying and adding everyhting again but I am getting the same error.

I would be ok if all users have to download all assets again, but I do not want every user to delete their browser cache (which could help?!).

Update Oh and this is how I check for new versions:

if (this.swUpdate.isEnabled) {
  this.subs.sink = this.swUpdate.versionUpdates.subscribe((update) => {
    console.log(update);
    if (!this.snackBar._openedSnackBarRef && update.type == "VERSION_READY")
      this.openSnackBar();
  });
}