How to use $addToSet to add a new object only if the object does not exist in the array already

Question

I have a post schema where there is data like this inside it

  const PostSchema = new Schema({
    post votes: {
      upvotes: [{
        votedby: {
          type: Schema.Types.ObjectId,
          ref: 'User',
        },
        votedtime: {
          type: Date,
          default: Date.now,
        },
      }],
      downvotes: [{
        votedby: {
          type: Schema.Types.ObjectId,
          ref: 'User',
        },
        votedtime: {
          type: Date,
          default: Date.now,
        },
      }],
    },
    ... other data in it
  })

I need to add users to the upvotes or downvotes only if they are not present in the list already.

I am using something like this to update my post, but what happens is the user is still added to the list even though it exists already.

        //if downvoted
        const adduser = {
          votedby: userid,
        };
        const sendPost = await Post.findByIdAndUpdate(postid,
          {
            $addToSet: { 'postvotes.downvotes': adduser },
            $pull: { 'postvotes.upvotes': adduser },
          }, { new: true })

Also, I want the post to be returned everytime, either the votes been updated or not.

Thanks