How to run test suite in parallel but not the single tests

Question

I'm using Playwright.dev to automate our UI tests. Currently I face the following issue:

In a single spec.ts file I have different test suites. Those test suites should run in parallel but not each test. I can not split those test suites into separate files because they are created dynamically. Why I want to run the tests of each test suite serial is, to reuse the existing page. Because it's much faster to just reuse the page without doing a complete refresh of the page the whole time. I'll try to explain my problem with some pseudo-code:

catalogs.forEach((objectIdsOfCatalog, catalogId) => {
    // each suite could run in parallel because they do not
    // depend on each other
    test.describe('Test catalog "' + catalogId + '"', () => {
        let newVersion: PageObject;
        let actualVersion: PageObject;

        test.beforeAll(async ({browser}) => {
            console.log('New page for', catalogId);
            const {actualUrl, newUrl} = getConfig();
            const context = await browser.newContext();
            actualVersion = new PageObject(await context.newPage(), actualUrl);
            newVersion = new PageObject(await context.newPage(), newUrl);
        });

        test.afterAll(async () => {
            console.log('Close page for', catalogId);
            actualVersion.close();
            newVersion.close();
            actualVersion = null;
            newVersion = null;
        });
        // those tests should ran serial because it's faster
        // if we just navigate on the existing page due to the
        // client side caching of the web app
        for (const objectId of objectIdsOfCatalog) {
            test('Testing "' + objectId + '"', async () => {
            });
        }
    });
});

Is there some way to achieve the following behavior in Playwright or do I have to rethink my approach?

score 0 · Answer 1 · answered Nov 24 '21 at 19:57

I don't know if multiple test.describe.serial blocks could be nested in a test.describe.parallel (and if that works), but maybe that's worth a try.

Another option could be to not generate real tests, but just to generate steps (test.step) inside tests inside a test.describe.parallel block.

And where do the catalogs come from? Maybe instead of generating describe blocks, you could generate projects in the playwright.config.ts? Projects run in parallel by default I think. But don't know if that approach would work if the data is coming from some async source.

Hey @refactoreric thanks for your response :-) I think the idea with the projects could be nice but currently I face the problem that I only have access to the project within a test but I would need the data of the project outside of a test so I can generate the tests. — stephan.niedermayr, Nov 25 '21 at 12:11

score 0 · Answer 2 · answered Mar 28 '23 at 06:15

You have a few options which will achieve the result you desire, or at least something similar.

First things first, the forEach block executes synchronously and also cannot exit or return early (always loops over every single item in an array and will return immediately even if you try to await within it).

You will instead want to nest the loop inside an async function like so (untested, also written in TypeScript):

async function tests() {
  const catalogs = [[{ objectIdsOfCatalog: { key: 'value' } }]];
  // tell PlayWright test runner to run separate describe blocks in parallel
  test.describe.configure({ mode: 'parallel' });

  await Promise.all(
    catalogs.map(async (objectIdsOfCatalog, catalogId) => {
      // tell PlayWright test runner nested tests inside these describe
      // blocks should be run sequentially 
      test.describe.serial('Test catalog "' + catalogId + '"', () => {
        let newVersion: PageObject;
        let actualVersion: PageObject;

        test.beforeAll(async ({ browser }) => {
          // ...setup
        });

        test.afterAll(async () => {
          // ...cleanup
        });

        for (const objectId of objectIdsOfCatalog) {
          // removed async keyword to ensure test function is serial
          test('Testing "' + objectId + '"', () => {});
        }
      });
    })
  );
}

To be honest I'm not sure generating tests like this is supported by PlayWright - that is - the relationship between the map function and how it "unrolls" the code in order for the test runner to interpret is something you would need to either test or deep dive into their code base to understand.

It could be that the runner doesn't understand the instructions here and there isn't a neat way around it.

If you want to tackle this in a more "PlayWright" friendly way, I would recommend:

Using a custom worker scoped fixture which encapsulates the page object initialisation you want done
Using image matching. It's not clear what you are actually testing based on the code you shared - but if you are comparing images or screenshots of new and old version definitely look into the toHaveScreenshot() assertion.
Consider testing specific items within the catalogue that have contrasting or unexpected values. Testing every single item inside the catalogue, especially if they have the same shape/schema, is unnecessary. Sometimes less is more when it comes to testing (and especially E2E testing). Find some problematic/unique (but valid) items in your catalogue and use only those for the E2E tests.

How to run test suite in parallel but not the single tests

2 Answers2