21 Mar “Frank Sinatra” Sings A New Song After Death With Deepfake Audio
Copyright issues in AI-generated music
“It’s Christmas time! It’s hot tub time!” sings Frank Sinatra.
At least, it sounds like him. The song, ‘Hot Tub Time’, is the outcome of artificial intelligence algorithms developed by OpenAI. Frank Sinatra died 23 years ago in 1998, but his voice has been recreated using deepfake audio.
Frank Sinatra singing about hot tubs more than 20 years after his death.
With the same, familiar easy swing, cheery conviviality, and subtle brass and string flourishes, the track sounds like it could be some long-lost unreleased demo. Even that rich-toned voice is eerily familiar, yet it sounds not quite right, like it was recorded at the bottom of a swimming pool.
What Is an Audio Deepfake?
An audio deepfake is an AI recreation of someone’s voice. It is generated by artificial intelligence through machine learning, where a large amount of data is input through computer algorithms that can improvise by ingesting and recompiling. The outcome is a hyper realistic audio of something that never actually happened, complete with the corresponding lyrics.
‘Hot Tub Time’ was produced by OpenAI’s artificial intelligence system known as Jukebox which can generate new songs and vocals which sound almost exactly like the real artists. The Jukebox system was trained by feeding 1.2 million songs with its corresponding lyrics and track information, such as artist names, genres and years of release. Along with Sinatra, they’ve done deepfakes of Katy Perry, Elvis, Simon and Garfunkel, 2Pac, Céline Dion and more. Using data, Jukebox can recreate basically any new music samples that never existed.
AI researchers and enthusiasts have also recreated an old 1999 Eminem song with updated cultural references in the lyrics for a 2021 audience. YouTuber, 30 Hertz, uploaded the modernised version of ‘My Name Is’ using AI technology on the vocals – kind of like if Eminem was a Gen Z of today. The result is a scarily realistic new take on the original in both lyricism and delivery. You’d find that you’ll constantly try to remind yourself that it’s not The Real Slim Shady on the vocals, because it’s exactly sort of bars he would’ve written.
Eminem is 49 years old now, but what would he write if he was a teen of today?
Source: YouTube
Immediately, moral and legal questions around audio deepfake technology comes to mind. Discussions involving man versus technology often come with so much complexities and intricacies and potentially presents interesting, new issues for the music industry.
Legal Standpoint: Can You Copyright a Voice?
It’s not hard to foresee how such deepfakes could lead to intellectual property issues.
The human voice is not protected under copyright law. This means vocal artists can be put in a difficult situation if someone tries to create an audio deepfake of them. At this point, we are treading new waters in copyright laws.
If a voice cannot be copyrighted, what are the
protections against the unlawful use of a voice?
With music there are two separate copyrights. One is in the song, which is the lyrics and the music notation component, and another is in the sound recording, which is what recording labels are involved.
So technically, copyright holders of a singer’s sound recordings could potentially still claim copyright infringement if they could prove their recordings were copied in the process of creating a deepfake, whether someone made money from it or not. That means you must show that a recording already exists, plus you own copyright in that recording. Then, you have to prove that someone has somehow gotten unauthorized access and copied that recording. In short, it means you can exercise your copyrights over recordings of a voice, not the person’s voice itself.
This is where it becomes complicated. If an AI system simply listened to sound recordings and then an entirely new sound recording was created without including any part of the original sound recordings, it’s highly unlikely that copyright infringement of the original sound recordings would have occurred. That would just be the equivalent of a regular human songwriter listening to hundreds of rock music and creating his or her own music which contains elements of prominent rock artists. In this situation, you can’t really say copyright laws have been infringed.
Audio deepfakes will create many issues which
are not yet addressed by our copyright laws.
The situation will be even more complicated if a newly created deepfake contained just a little part of an existing copyright-protected musical work, for instance, a small snippet of an existing melody. Deepfake audio is already challenging to explain, analyse, and define, so it’s even more challenging to sue over. It’s too new in contrast to copyright laws that could be hundreds of years old, and the boundaries of copyright around audio deepfakes are still being tested.
Another situation worth pondering over is this. If an advertising company didn’t want to pay the market rate for using an established artist’s music in a film, TV show or commercial, could they just use their own imitation? Ultimately, can streaming services, radio stations and others increasingly avoid paying humans, and instead rely on AI for new music?
JAY-Z’s voice was deepfakes to say the Shakespeare’s “To Be, Or Not to Be” soliloquy from Hamlet.
Moral Standpoint: Keeping the Dead “Alive”?
As a piece of engineering, it’s really impressive, but, would deepfake audio be accepted by fans?
An artist is so much more than their voice. Artists often write and sing about their personal lives and connect to their fanbase through lyrics that speak out to fans. Artists recreate themselves constantly throughout their own life journey, and fans love to relate to an artists’ personal journey and growth.
Also, the music industry is all about novelty. If we held onto AI generated voices that is basically just a “fake” version of an artist that anyone can recreate, is it really still the same? Novelty is what makes original recordings and live performances so valuable. Even so, as technology continues to evolve, we’re likely going to see this sort of AI tech become more sophisticated. The accuracy of audio cuts like this will only get better and better.
Nonetheless, at what point is it too much,
too creepy, too much like a singing phantom of a
person long gone?
Admirable as the technical achievement is, there’s something absolutely horrifying about the songs generated of artists who have long since died, like sad, trapped ghosts, lost in the music machine.
I’m sure we can all agree that there is a sense of emotional investment in what our favourite artists do. These systems are trained on human expression and will augment it, but we will lose something fundamental to music – the human side of music creation.
At the moment, it seems to be just a curiosity in the music industry as it is unlikely to replace great artists in the industry at the moment. There is no shortage of fresh talent and it’s not that we don’t have enough talented singers and musicians to have the need for AI music.
However, that doesn’t mean we shouldn’t be watching the technology closely.
Mother meets recreation of her deceased child speaking with a deep faked voice in VR for one last goodbye.
The Law Needs to Catch Up
As artificial intelligence continues to evolve, the law must progress as well.
In the Economic Impact of the Malaysian Music Industry: Final Report (2016), the Malaysian music industry provides significant economic contribution at RM5.5 -RM6.1 billion per year. Every Ringgit spent in the music industry generates about RM2.20 of output for the Malaysian economy, consistent with typical output for other industries, giving a total output of RM12.2 RM13.4 billion per year.
There needs to be a push for greater protection of the common person against vocal deepfakes as Malaysia is not too far behind the global pace in matters regarding technology and AI. How the industry will deal with this brave new world and what steps might be taken to protect artists and the integrity of their work is important to protect the copyrights of our local creative minds.
Do you really understand your copyrights? You might be protected automatically for free and don’t even know it. Talk to a us to see if you need a copyright registration and to fully understand your intellectual property rights.