How this Canadian producer is using AI to allow artists to sing in different languages

When Canadian producer Jordan Young started playing around with AI voice-cloning technology — ultimately creating a fake Kanye West song that sounded, well, just like Kanye West — it immediately prompted him to think about the ethical implications of the technology. 

“As soon as I made it, I started to think … if anybody can make music with anyone’s voice, what does this mean for artists? What does this mean for music?” said Young, who has worked with Beyoncé, BTS and Dua Lipa.

He and his company Hooky recently developed a technology that could make an artist sound like they were singing in a different language from their own.

That resonated with American pop singer Lauv, who at the time had just finished a Korean tour and wanted to find a meaningful way to thank his fans. Why not develop a new version of his song Love U Like That, transformed to make it sound like he was performing in Korean?

The process began with Young’s company tapping Kevin Woo, a South Korean singer-songwriter who is fluent in English and Korean, to rewrite and sing the song in a new tongue. That interpretation has to capture the nuances of the Korean language, explained Young.

“It’s not just, throw it into a translator website and whatever it spits out, that’s your song. You have to be artistic about it,” Young said.

Help shape the future of CBC article pages by taking a quick survey.

The recordings also had to match musically. For example, if Lauv sang in falsetto on a few bars, Woo did the same in his Korean-language demo.

Enter artificial intelligence.

The technology is trained on “every corner” of Lauv’s voice, as Young puts it — from the highest frequencies and tones to the lowest — to build a comprehensive model of his vocals.

The Korean-language demo written and sung by Woo is then combined with the AI model of Lauv’s voice, creating a Korean-language song that sounds like it’s sung by Lauv.

WATCH | The Korean-language version of LAUV’s Love U Like That: 

It’s been a year of reckoning for creative industries wrestling with artificial intelligence.

The music business in particular has seen an explosion of AI-generated songs made to sound like the world’s most popular artists — such as the viral track Heart On My Sleeve, which featured computer-generated vocals that sounded like Drake and The Weeknd.

Some artists are fighting the non-sanctioned use of their voices and likeness, while others like Grimes, John Legend and Demi Lovato are embracing the tech to varying degrees. 

“At minimum, we should give artists control over how their voice is used,” said Young, who is from Toronto, noting that’s where the idea came from to use AI to change languages.

“It was about creating a control mechanism for artists to be able to manage their likeness in a world where AI can totally disrupt that.”

Other companies are working on technology that focuses on music and translation.

AudioShake, a San Francisco company that specializes in audio separation, isolates a song’s vocal track from the music, then uses AI to transcribe and translate the song, building technology that maps the original English with the new language so that fans can follow the lyrics of the original song in their language. Pop singer Sia is one of their clients.

Meanwhile, South Korea’s largest record label HYBE used in-house AI technology so that its artist Lee Hyun, known as MIDNATT, could record his song Masquerade in six different languages: English, Korean, Spanish, Japanese, Chinese and Vietnamese.

The singer recorded his song in each language, which was then combined with demos of native speakers reading out the lyrics. Hyun speaks Korean as well as limited English and Chinese.

“It’s a scary thing that people don’t fully understand, and often AI is portrayed as this monolithic thing, and it’s not,” said Young.

“AI is actually many different technologies that can come together in many different ways.”

Artists already trying to reach international fans

Even before AI came to the forefront this year, artists had already been trying to use different languages to target fans, according to Kristin Robinson with Billboard Magazine.

She points to Despacito, a 2017 song by Puerto Rican singer Luis Fonsi and Puerto Rican rapper Daddy Yankee, that saw its remixed version featuring Justin Bieber become a huge hit. 

“Having Justin Bieber hop onto a Latin track is not something I would have ever anticipated. Having Beyoncé try to sing in Spanish is not something I had on my bingo card for Beyoncé to do.”

But with AI now factored into this already successful trend, Robinson says people will likely have mixed feelings about “the authenticity” of the process.

It remains to be seen if this type of music will be well received. Robinson says that using the technology in a way that is both ethically and culturally appropriate is an important part of the conversation.

That was reflected in some of the online reaction to Lauv’s Korean-language version of Love U Like That. 

Some fans said they were “obsessed” with the new version, while others questioned the optics of a white artist using technology to insert his voice over a song sung by a Korean artist. Still others suggested different ways to have Lauv and Woo sing on the same track.

Getting ‘nuance’ of language important: Toronto artist

Toronto jazz vocalist and songwriter Jacqueline Teh hopes to incorporate her family’s language — Hokkien, a Chinese dialect from the Fujian province — into her future projects. As a multilingual artist, she can see the benefits and the downsides of using AI to create songs in different languages.

She says the technology seemed exciting at first, adding that it can be a “great springboard” for artists to access other languages.

But she says credit and agency are incredibly important to artists — and that she wouldn’t want AI to come between a potential relationship with a language that’s new to her, or with a person who speaks a language different from her own.

“I think with language there’s so much nuance and there’s so much slang and there’s so much that can’t really be translated just through technology,” she said, which is why working with a native speaker makes the song more meaningful to her. 

She says as long as artists aren’t using AI to bypass the hard work that goes into creating meaningful relationships with fans and other artists, “there’s so much that is accessible to us with technology.”

Young, the producer, is working on a new translation project — potentially with a Canadian artist. He says now that the genie is out of the bottle, the industry has to figure out how to steer the tech in the right direction.

“I’m never going to build something that removes the human being from the creative process,” he said. “And there are lots of companies that are attempting to do that right now.”