Is there any way to get Jest to handle importing libraries with esmodules?

Question

Historically I've just used the transformIgnorePatterns option in Jest to convert esmodule libraries from esmodules to commonjs:

module.exports = {
    transformIgnorePatterns: [
        '<rootDir>/node_modules/(?!my-library)'
    ]
}

However this adds a lot of extra compile time, especially since I have a bunch of libraries in these ignore patterns now.

I'm trying to avoid needing to transform esmodules to commonjs. I'm using Jest & SWC to try and do this.

Here is my Jest config:

import fs from 'fs';

const swcConfig = JSON.parse(fs.readFileSync('./.swcrc', 'utf8'));

export default {
    testEnvironment: 'jsdom',
    transform: {
        '^.+\.jsx?$': [
            '@swc/jest',
            swcConfig
        ]
    }
};

And my package.json is setup the following way:

{
    "type": "module",
    "scripts": {
        "test": "NODE_OPTIONS=--experimental-vm-modules npx jest"
    }
}

This seems to be enough to add support for local esmodule code. However, library code still fails. The following example is a super simple test I've created for this. I'm just trying to make an import from a library whose code is in esmodules work:

import { MyComponent } from 'my-library';

describe('My Test', () => {
    it('can import successfully', () => {
        expect(MyComponent).not.toBeNull();
    });
});

The import from my-library will fail with a Jest encountered an unexpected token error, referring to the export line at the top of the index file in the library.

Is there some way to make this work without transformIgnorePatterns?