Unable to hear participants audio to each other in voice conference

Question

Have created a sample twilio voice conference app with java script SDK. At nodeJS, two endpoints, one for getting the access token and other with /voice for conference. Now, from js side, we request token and initialised the device with token and then made an outbound call to a device (through webhook -> /voice) and caller answered it (here its the moderator). And once it's done, add another participant to the conference. Now, in the deskop audio, where the JS UI running, is able to hear participants talks, but participants cannot hear each other.

1: code to get the access token:

const accessToken = new AccessToken(
      config.accountSid,
      config.apiKey,
      config.apiSecret
  );
  
  accessToken.identity = identity;
  const grant = new VoiceGrant({
    outgoingApplicationSid: config.twimlAppSid,
    incomingAllow: true,
  });
  accessToken.addGrant(grant);
  return {
    identity: identity,
    token: accessToken.toJwt(),
  };

2: /voice end point:

const toNumberOrClientName = requestBody.To;
  const callerId = config.callerId;

  const MODERATOR = 'xxxxxxxxxxx';
  const twiml = new VoiceResponse();  
  
  if (toNumberOrClientName == MODERATOR) {
    console.log('moderator');
    const dial = twiml.dial({callerId: callerId, answerOnBridge: true});
    // dial.client(identity);

    const attr = isAValidPhoneNumber(toNumberOrClientName) ? 'number' : 'client';
    dial[attr]({}, toNumberOrClientName);

    dial.conference(requestBody.conferenceId, {
      startConferenceOnEnter: true,
      endConferenceOnExit: true,
      beep: true,
    });

    client.conferences(requestBody.conferenceId)
        .participants
        .create({
          label: 'moderator',
          earlyMedia: true,
          beep: 'onEnter',
          statusCallback: '',
          statusCallbackEvent: ['ringing'],
          record: false,
          from: callerId,
          to: toNumberOrClientName,
        })
        .then((participant) => {
          console.log('moderator added : ' + participant.callSid);
        });

  } else {
    console.log('participant');
    const dial = twiml.dial({callerId: callerId, answerOnBridge: true});

    const attr = isAValidPhoneNumber(toNumberOrClientName) ? 'number' : 'client';
    dial[attr]({}, toNumberOrClientName);

    dial.conference(requestBody.conferenceId, {
      startConferenceOnEnter: false,
      beep: true,
    });

    client.conferences(requestBody.conferenceId)
        .participants
        .create({
          label: 'participant',
          earlyMedia: true,
          beep: 'onEnter',
          statusCallback: '',
          statusCallbackEvent: ['ringing'],
          coaching: false,
          record: false,
          from: callerId,
          to: toNumberOrClientName,
        })
        .then((participant) => {
          console.log('participants added : ' + participant.callSid);
        });
  }

Anything missing in this to get the audio to each participant?