Google Introduces Dreamix, A Text-Conditioned Video Editing Model

Google Research recently released Dreamix, an AI model that will be able to perform text-based motion and appearance editing of general videos. 

While there have been several image-editing tools released based on diffusion models, Dreamix is the first diffusion-based method for video editing. 

The model essentially functions in three ways: 

To ensure high quality output and fidelity to the input image and prompt, the team will be to enable a text-conditioned video diffusion model (VDM) with two main ideas: first, rather than using pure-noise as initialisation of the model, they used a degraded version of the original video by downscaling it and adding noise, keeping only low spatio-temporal information; and second, they further improved the fidelity to the original video by introducing a mixed fine-tuning approach, wherein the VDMs are also fine-tuned on the collection of individual frames of the input video while discarding their temporal order by masing the temporal attention.  

Additionally, the researchers also proposed a new framework for image animation for applications that include animating the objects and background in an image, creating dynamic camera motion, and more. This is done by employing simple image processing operations, such as frame replication or geometric image transformation, to generate a coarse video. The video is then passed through the Dreamix video editor for editing. 

On comparing this method against the state-of-the-art baselines, their model showed superior results on factors like quality, fidelity, and alignment.