PWA ServiceWorker does not work with Asp.net core Spa Extenstions

Question

I created a sample project aspnet.core 2.1 webapplication with angular and added pwa with ng add @angular/pwa

Then created a small app with two controllers to test.

Now if I run the app with angular client in spaextension, the serviceworker is registered and running but does not cache assets or datagroups at all. Client is generated with --prod and spaExtension does not start a dev server, it loads the files from the 'dist' folder.

if I host the client and the backend with two hosts (backend localhost:5555, Angular Client localhost:5556) it works as expected, app, images, fonts and api is cache as configured.

    services.AddSpaStaticFiles(configuration =>
    {
        configuration.RootPath = "ClientApp/dist";
    });

    app.UseSpa(spa =>
    {
        // To learn more about options for serving an Angular SPA from ASP.NET Core,
        // see https://go.microsoft.com/fwlink/?linkid=864501

        spa.Options.SourcePath = "ClientApp";

        //if (env.IsDevelopment())
        //{
        //    spa.UseAngularCliServer(npmScript: "start");
        //}
    });

I have no idea why it doesn't work when the client is running on the same host but does work if hosted separately. API route could be wrong, but I tried all possible ways. Assets don't have a route so it should work anyway.

here the ngsw-config, but since it works when hosted separately I don't think the problem is here.

{
  "index": "/index.html",
  "assetGroups": [
    {
      "name": "app",
      "installMode": "prefetch",
      "resources": {
        "files": [
          "/favicon.ico",
          "/index.html",
          "/*.css",
          "/*.js"
        ]
      }
    }, {
      "name": "assets",
      "installMode": "lazy",
      "updateMode": "prefetch",
      "resources": {
        "files": [
          "/assets/**",
          "/*.(eot|svg|cur|jpg|png|webp|gif|otf|ttf|woff|woff2|ani)"
        ]
      }
    }, {
      "name": "fonts",
      "resources": {
        "urls": [
          "https://fonts.googleapis.com/**",
          "https://fonts.gstatic.com/**"
        ]
      }
    }
  ],
  "dataGroups": [
    {
      "name": "api-freshness",
      "urls": [
        "https://localhost:5001/api/news"
      ],
      "cacheConfig": {
        "maxSize": 100,
        "maxAge": "3d",
        "timeout": "1m",
        "strategy": "freshness"
      }
    },
    {
      "name": "api-performance",
      "urls": [
        "https://localhost:5001/api/list"
      ],
      "cacheConfig": {
        "maxSize": 100,
        "maxAge": "10d",
        "timeout": "10s",
        "strategy": "performance"
      }
    }
  ]
}