Load/stress test in a SPA with Hasura Cloud Graphql as a backend and subscriptions

Question

I'm trying to do a performance test on a

• SPA with a Frontend in React, deployed with Netlify
• As a backend we're using Hasura Cloud Graphql (std version) https://hasura.io/, where everything from the client goes directly through Hasura to the DB.
• DB is in Postgress housed in Heroku (Std 0 tier).
• We're hoping to be able to have around 800 users simultaneous.

The problem is that i'm loss about how to do it or if i'm doing it correctly, seeing how most of our stuff are "subscriptions/mutations" that I had to transform into queries. I tried doing those test with k6 and Jmeter but i'm not sure if i'm doing them properly.

k6 test

At first, i did a quick search and collected around 10 subscriptions that are commonly used. Then i tried to create a performance test with k6 https://k6.io/docs/using-k6/http-requests/ but i wasn't able to create a working subscription test so i just transform each subscription into a query and perform a http.post with this setup:

export const options = {
  stages: [
    { duration: '30s', target: 75 },
    { duration: '120s', target: 75 },
    { duration: '60s', target: 50 },
    { duration: '30s', target: 30 },
    { duration: '10s', target: 0 }
  ]
};

export default function () {
  var res = http.post(prod,  
    JSON.stringify({
    query: listaQueries.GetDesafiosCursosByKey(
      keys.desafioCursoKey
    )}), params);
  sleep(1)
}

I did this for every query and ran each test individually. Unfortunately, the numbers i got were bad, and somehow our test environment was getting better times than production. (The only difference afaik is that we're using Hasura Cloud for production).

I tried to implement websocket, but i couldn't getthem work and configure them to do a stress/load test.

K6 result

Jmeter test

After that, i tried something similar with Jmeter, but again i couldn't figure how to set up a subscription test (after i while, i read in a blog that jmeter doesn't support it https://qainsights.com/deep-dive-into-graphql-in-jmeter/ ) so i simply transformed all subscriptions into a query and tried to do the same, but the numbers I was getting were different and much higher than k6.

Jmeter query Config 1

Jmeter query config 2

Jmeter thread config

Questions

I'm not sure if i'm doing it correctly, if transforming every subscription into a query and perform a http request is a correct approach for it. (At least I know that those queries return the data correctly).

Should i just increase the number of VUS/threads until i get a constant timeout to simulate a stress test? There were some test that are causing a graphql error on the website Graphql error, and others were having a

""WARN[0059] Request Failed error="Post \"https://xxxxxxx-xxxxx.herokuapp.com/v1/graphql\": EOF""

in the k6 console. Or should i just give up with k6/jmeter and try to search for another tool to perfom those test?

Thanks you in advance, and sorry for my English and explanation, but i'm a complete newbie at this.

Answer 1

I'm not sure if i'm doing it correctly, if transforming every subscription into a query and perform a http request is a correct approach for it. (At least I know that those queries return the data correctly).

Ideally you would be using WebSocket as that is what actual clients will most likely be using.

For code samples, check out the answer here.

Here's a more complete example utilizing a main.js entry script with modularized Subscription code in subscriptions\bikes.brands.js. It also uses the Httpx library to set a global request header:

// main.js
import { Httpx } from 'https://jslib.k6.io/httpx/0.0.5/index.js';

import { getBikeBrandsByIdSub } from './subscriptions/bikes-brands.js';

const session = new Httpx({
  baseURL: `http://54.227.75.222:8080`
});

const wsUri = 'wss://54.227.75.222:8080/v1/graphql';

const pauseMin = 2;
const pauseMax = 6;

export const options = {};

export default function () {
  session.addHeader('Content-Type', 'application/json');

  getBikeBrandsByIdSub(1);
}

// subscriptions/bikes-brands.js
import ws from 'k6/ws';

/* using string concatenation */
export function getBikeBrandsByIdSub(id) {
  const query = `
    subscription getBikeBrandsByIdSub {
      bikes_brands(where: {id: {_eq: ${id}}}) {
        id
        brand
        notes
        updated_at
        created_at
      }
    }
  `;

  const subscribePayload = {
    id: "1",
    payload: {
      extensions: {},
      operationName: "query",
      query: query,
      variables: {},
    },
    type: "start",
  }

  const initPayload = {
    payload: {
      headers: {
        "content-type": "application/json",
      },
      lazy: true,
  
    },
    type: "connection_init",
  };

  console.debug(JSON.stringify(subscribePayload));

  // start a WS connection
  const res = ws.connect(wsUri, initPayload, function(socket) {
    socket.on('open', function() {
      console.debug('WS connection established!');

      // send the connection_init:
      socket.send(JSON.stringify(initPayload));

      // send the chat subscription:
      socket.send(JSON.stringify(subscribePayload));
    });

    socket.on('message', function(message) {
      let messageObj;
      try {
        messageObj = JSON.parse(message);
      }
      catch (err) {
        console.warn('Unable to parse WS message as JSON: ' + message);
      }

      if (messageObj.type === 'data') {
        console.log(`${messageObj.type} message received by VU ${__VU}: ${Object.keys(messageObj.payload.data)[0]}`);
      }

      console.log(`WS message received by VU ${__VU}:\n` + message);
    });
  });
}

Should i just increase the number of VUS/threads until i get a constant timeout to simulate a stress test?

Timeouts and errors that only happen under load are signals that you may be hitting a bottleneck somewhere. Do you only see the EOFs under load? These are basically the server sending back incomplete responses/closing connections early which shouldn't happen under normal circumstances.

Answer 2

My expectation is that your test should be replicating the real user activity as close as possible. I doubt that real users will be sending requests to GraphQL directly and well-behaved load test must replicate the real life application usage as close as possible.

So I believe you should move to HTTP protocol level and mimic the network footprint of the real browser instead of trying to come up with individual GraphQL queries.

With regards to JMeter and k6 differences it might be the case that k6 produces higher throughput given the same hardware and running requests at maximum speed as it evidenced by kind of benchmark in the Open Source Load Testing Tools 2021 article, however given you're trying to simulate real users using real browsers accessing your applications and the real users don't hammer the application non-stop, they need some time to "think" between operations you should be getting the same number of requests for both load testing tools, if JMeter doesn't give you the load you want to conduct make sure to follow JMeter Best Practices and/or consider running it in distributed mode .