Twilio + React: How to change audio or video input device during a call?

Question

I've been looking at the documentation but I can't figure it out. When a user selects a different device, the other users can't hear that person anymore. This must mean that something is going right with the unpublishing of tracks, right? I'm not sure.

This is my code for a user to change devices:

const setDevice = (device) => {
    if(!room) return
    let deviceId = device.deviceId
    const localParticipant = room.localParticipant

    if(device.kind === 'audioinput'){
        setSelectedAudioDevice(device.label)

        Video.createLocalAudioTrack({
            deviceId: {exact: deviceId}
        }).then((localAudioTrack) => {
            const tracks = localParticipant.audioTracks

            tracks.forEach((track) => {
                localParticipant.unpublishTrack(track.track)
            })

            localParticipant.publishTrack(localAudioTrack)
        })
    } else if(device.kind === 'videoinput'){
        setSelectedVideoDevice(device.label)

        Video.createLocalVideoTrack({
            deviceId: {exact: deviceId}
        }).then((localVideoTrack) => {
            const tracks = localParticipant.videoTracks

            tracks.forEach((track) => {
                localParticipant.unpublishTrack(track.track)
            })

            localParticipant.publishTrack(localVideoTrack)
        })
    }
}

Each participant has its own component in which they subscribe to tracks. However, this code was from one of the Twilio examples, so I'm not entirely sure how it works.

const trackpubsToTracks = (trackMap) =>
    Array.from(trackMap.values())
      .map((publication) => publication.track)
      .filter((track) => track !== null);

useEffect(() => {
    setVideoTracks(trackpubsToTracks(participant.videoTracks));
    setAudioTracks(trackpubsToTracks(participant.audioTracks));

    const trackSubscribed = (track) => {
      if (track.kind === "video") {
        setVideoTracks((videoTracks) => [...videoTracks, track]);
      } else if (track.kind === "audio") {
        setAudioTracks((audioTracks) => [...audioTracks, track]);
      }
    };

    const trackUnsubscribed = (track) => {
      if (track.kind === "video") {
        setVideoTracks((videoTracks) => videoTracks.filter((v) => v !== track));
      } else if (track.kind === "audio") {
        setAudioTracks((audioTracks) => audioTracks.filter((a) => a !== track));
      }
    };

    participant.on("trackSubscribed", trackSubscribed);
    participant.on("trackUnsubscribed", trackUnsubscribed);

    return () => {
      setVideoTracks([]);
      setAudioTracks([]);
      participant.removeAllListeners();
    };
  }, [participant]);

  useEffect(() => {
    const videoTrack = videoTracks[0];
    if (videoTrack) {
      videoTrack.attach(videoRef.current);

      return () => {
        videoTrack.detach();
      };
    }
  }, [videoTracks]);

  useEffect(() => {
    const audioTrack = audioTracks[0];
    if (audioTrack) {
      audioTrack.attach(audioRef.current);

      return () => {
    audioTrack.detach();
      };
    }
  }, [audioTracks]);

If anyone knows how I can handle device switching mid-call, I'd greatly appreciate it.

Answer 1

Twilio developer evangelist here.

I have found that the best order of operations here is:

1. Unpublish the local participant’s existing track from the room, this will trigger the trackRemoved event on the room for any other participants
2. Detach the existing track from the page
3. Stop the track completely
4. Request the new track with createLocal(Video|Audio)Track
5. Attach the new track to the page
6. Publish the new track to the room, triggering the trackAdded event on the room for the other participants

This is especially true for iOS devices which do not let you access more than one camera at a time.

Here is some code I've used before, though not in a React application:

function stopTracks(tracks) {
  tracks.forEach(function(track) {
    if (track) { track.stop(); }
  })
}

function updateVideoDevice(event) {
  const select = event.target;
  const localParticipant = activeRoom.localParticipant;
  if (select.value !== '') {
    const tracks = Array.from(localParticipant.videoTracks.values()).map(
      function(trackPublication) {
        return trackPublication.track;
      }
    );
    localParticipant.unpublishTracks(tracks);
    detachTracks(tracks);
    stopTracks(tracks);
    Video.createLocalVideoTrack({
      deviceId: { exact: select.value }
    }).then(function(localVideoTrack) {
      localParticipant.publishTrack(localVideoTrack);
      log(localParticipant.identity + ' added track: ' + localVideoTrack.kind);
      const previewContainer = document.getElementById('local-media');
      attachTracks([localVideoTrack], previewContainer);
    });
  }
}

You can see the entire application in this repo on GitHub and I wrote about it here.

I think the React example you're referring to was one of mine too. I actually had a go at adding camera changes to that repo on a branch. It was apparently a year ago, but you can see the updates here. Hopefully that can point you in the right direction too.