16

There have been numerous news reports of scammers using AI voice cloning to trick people into thinking that their kid has been kidnapped:

  • NY Post

    An Arizona mom claims that scammers used AI to clone her daughter’s voice so they could demand a $1 million ransom from her as part of a terrifying new voice scheme.

  • NBC News

    The FTC is sounding the alarm on artificial intelligence being used to simulate someone’s voice in imposter scams, which was the most commonly reported fraud in 2022. NBC News’ Emilie Ikeda spoke to one father who got a call that sounded like his daughter and said she was being held hostage.

  • Washington Post

    Scammers are using artificial intelligence to sound more like family members in distress. People are falling for it and losing thousands of dollars.

I'm skeptical – not that people are getting scammed, but that AI voice generation is involved. It seems like it would be a lot of trouble to get hold of audio of someone's kid and feed it into AI (in "terrified kidnapped child" mode?), when you can just use a generic terrified-kid soundbite and rely on the power of suggestion. None of the sources I've found present any actual evidence that AI is being used; they just repeat the victims' claims that the voice over the phone absolutely sounded like their child.

Even the FTC is claiming this is a thing. But I wonder if they have evidence or if they're just following the news reports. No one's ever going to get in trouble for telling people to watch out for scams, after all.

Are scammers using AI voice cloning to fake kidnappings?

Oddthinking
  • 140,378
  • 46
  • 548
  • 638
joshuahhh
  • 261
  • 2
  • 1
    There have been other reports of AI being used to generate convincing fake voices. For example: https://www.npr.org/2023/03/22/1165448073/voice-clones-ai-scams-ftc https://www.forbes.com/sites/thomasbrewster/2021/10/14/huge-bank-fraud-uses-deep-fake-voice-tech-to-steal-millions https://www.forbes.com/sites/jessedamiani/2019/09/03/a-voice-deepfake-was-used-to-scam-a-ceo-out-of-243000 – Kyralessa Apr 14 '23 at 12:09
  • @Kyralessa Those sources seem similar to the problems in OPs post. It's very believable that this would be done to target organizations, but seems less likely for targeting someone's grandparents. – JMac Apr 14 '23 at 12:50
  • 3
    One question is how they get the voices. I wonder whether these are families with a lot of videos of their kids online. Even so, it would be surprising if the audio quality were high enough to be convincing by an AI. But then if it's a mobile phone call, maybe it doesn't need very high audio quality, given how bad mobile sound quality is in the first place. – Kyralessa Apr 14 '23 at 14:50
  • 6
    @Kyralessa: We can easily speculate about the amount of video online, but we re looking for evidence that it happened, not handwaving about how much children use TikTok and Twitch. – Oddthinking Apr 14 '23 at 16:29
  • 1
    It's not at all difficult to clone a voice, and it only requires 5 seconds of recordings to do so. Obviously the more source material you have the better the voice. You can also use your own voice to generate text which then generates the face voice, in real time. It's completely free and open source too. https://youtu.be/-O_hYhToKoA Literally anyone who's shared their voice online is susceptible - and that's most people. – Andy Gee Apr 18 '23 at 21:40
  • 1
    A recording of an actor with a similar voice. Few adjustments of the pitch and other variables to make the voice as close as possible. Then add some drama to the situation and even if the voice is not exactly the same it is not difficult to persuade the victim under stress that they are hearing the voices of their kids. – FluidCode May 05 '23 at 09:36

0 Answers0