How to properly configure swc-loader with webpack, when switching from ts-loader for better build performance

Question

I have a ts-loader configuration in webpack,

test: /\.tsx?$/,
use: [
  {
    loader: 'ts-loader',
    options: {
      transpileOnly: true
    }
  }
]

This was taking like 5 mins of build time. so, to improve the build time I switched to swc-loader.

[!] I can't change webpack now because there is a lot of logic and configurations which can break if I switch to another tool.

I switched to swc-loader with the following configuration to make it work.

test: /\.tsx?$/,
use: [
  {
    loader: 'swc-loader',
    options: {
      jsc: {
        parser: {
          syntax: 'typescript',
          tsx: true,
        },
      },
    },
  }
]

when I benchmarked using time yarn build, I get the following result For ts-loader

real    5m45.844s
user    9m10.575s
sys 0m15.470s

real    5m44.468s
user    9m6.640s
sys 0m14.864s

real    5m40.006s
user    9m7.087s
sys 0m15.133s

For swc-loader

real    6m9.500s
user    9m47.997s
sys 0m17.619s

real    6m12.210s
user    9m52.899s
sys 0m18.758s

real    6m11.122s
user    9m47.258s
sys 0m17.932s

Please help me understand on how to configure the swc-loader to gain the benefits for the same use-case and gain performance benefits if possible.