Google’s Deep Mind has showcased its latest results from its generative AI video-to-audio research. The system combines what is seen on screen with the user’s written prompt to create synced audio.
Called V2A AI, it can be paired with video-generation models such as Veo. It can create soundtracks, sound effects, and dialogue for on-screen action.
Deep Mind also claims it can generate “an unlimited number of soundtracks for any video input” by tuning the model with positive and negative prompts.
It works by encoding and compressing the video input, then leverages it to iteratively refine the desired audio effects from background noise, based on the user’s text prompt and the visual input.
The audio output is then decoded and exported as a waveform which can be recombined with the video input.
The user isn’t required to go in and manually sync the audio and video tracks, because the system does it automatically.
The Deep Mind team said, “By training on video, audio and the additional annotations, our technology learns to associate specific audio events with various visual scenes while responding to the information provided in the annotations or transcripts.”
The system isn’t entirely flaw-free yet. One; the output audio quality is dependent on the fidelity of the video input, and two; the system can mess up when video artifacts or distortions are present.
Deep Mind revealed syncing dialogue to the audio track is still a challenge as well.
“V2A attempts to generate speech from the input transcripts and synchronize it with characters’ lip movements. But the paired video-generation model may not be conditioned on transcripts. This creates a mismatch, often resulting in uncanny lip-syncing, as the video model doesn’t generate mouth movements that match the transcript.”
The team also revealed the system still has to undergo “rigorous safety assessments and testing” before it’s released to the public.
Stability AI also released a similar product last week, and ELevenLabs released its sound effects tool last month.
Categories
Select Category
Bauhn
Automation
Drones
Hisense
Wearables
Klipsch
MWC 2019
Latest News
TV
Promotion
Software
Video Streaming
Soundbars
Sharp
Cyrus Audio
Fujifilm
Brabantia
Automation
Comment
Accessories
Accessories
Cameras
Lenovo
Hitachi
Smartwatches
Video Downloads
Air Conditioning
Webcams
Optus
Electric Vehicles
Denon
Matter
nbn
IFA 2023
Display and TV
Content and Downloads
Plantronics
Appliances
Wearables
Dell
Computex 2019
Chrome
Desktop PC
Heating
Headphones
Telecomms
CES 2021
Arlo
Technics
Bang & Olufsen
Wave Audio
Wearables
Microsoft
Razer
Automation
Tablets & Computers
Nokia
IFA 2019
Surface Tablet
Hardware
Dash Cam
Keyboards
Coronavirus
Swann
Netflix
Lithe Audio
Westan
Withings
Gaming Hardware
Netgear
DJI
Cameras
Content & Downloads
House of Marley
TCL
Sound Buds
Monitors
Nextbase
Laptops
Leaks
MSI
Kayo
Liquid Ears
Pioneer
CES 2024
Home Office
Alcatel
Kaiser Baas
Cars & Bikes
Tablets & Computers
Android
CES 2020
Tablets
Security
Kitchen
Mouse
Laundry
Xiaomi
Uniden
Marshall
Ecovac
Acer
Oppo
LG
Comment
Events
Smart Home Devices
Netflix
e sports
Xbox
Devialet
ACCC
Action Camera
pet tech
M&K Sound
Cambridge Audio
Michi
Google
Brands
BenQ
Content and Downloads
IFA 2017
HP
Miele
Arlo
Bowers & Wilkins
Cygnett
Broadband
Contactless Payment
Apparel
Electric scooters
KEF
JL Audio
Smartphones
Samsung
Panasonic
Display and TV
Apartments
MWC 2018
Belkin
Connected Home
FitBit
Wireless Charging
NBN Co
Finance
Cleaning
Sonus Faber
MartinLogan
Fetch TV
Sonos
Apple
HyperX
Gaming Hardware
Test Category
Windows
Gaming
Security Cameras
Telstra
Foxtel
Latest Review
Point Of Sale
Outdoor
RØDE X
Definitive Technology
Asus
Sound
Archive
Hisense
Home Entertainment
Blender
Dyson
Home Security
Streaming
Xbox
Wireless
Office
Display
IsoTek
BenQ
Beats
BlueAnt
Reviews
Audio
Roccat
Home Office
Apple
Logitech
Networking
Cloud Gaming
Console
Turntables
Storage
Telstra
Alogic
Suunto
Pro-Ject
Bose
Appliances
Huawei
Home Entertainment
Sennheiser
News
CES 2018
Yamaha
Sponsored
Disney+
Gaming Controllers
Cybersecurity
Binge
Public Relations
McIntosh
Earbuds
Skullcandy
Sharp
B&O
ZTE
HTC
Polaroid
Reviews
D-Link
Ultimate Ears
MSI
Headphones
Automotive
Android
iPhone
Sales & Marketing
Peleton
Leica
Garmin
Alienware
Jabra
News
Cars & Bikes
Amazon
Smartphones
JBL
IFA 2018
5G
Coronavirus
Motor Cars
RealMe
Phones
Content
GoPro
Projectors
ECOVACS
Shokz
Motorola
Philips
Cameras
Linksys
Sound
Sony
CES 2019
Communication
Industry
BIG W
Phones
Xiaomi
Portable Speakers
Activision Blizzard
Nakamichi
ELAC
Ecovacs
