Can Midjourney Give Birth to a Movie? It Just Did!

You have heard of AI-generated images and videos, but how would you react to an AI-generated movie? Fabian Stelzer, a Berlin-based entrepreneur and AI enthusiast, has done just that. Taking the frenzy of text-to-image generation tools like Midjourney to another level, Stelzer has produced a movie named ‘Salt’. 

"I am carrying this project entirely on my own, but the heavy lifting is done by these incredible new AI tools," said Stelzer. 

With an academic background in cognitive neuroscience, Stelzer started a machine learning design startup called EyeQuant fresh out of college, which served clients like Google, Canon, Spotify and eBay.

The making of ‘Salt’

Over the past year, Stelzer stumbled upon text-to-image generation tools. "The first time I used Midjourney, my chin dropped to the floor – it was the most powerful technology I had witnessed since putting up a computer for the first time," he told Analytics India Magazine. 

Stelzer believes these tools are absolute game changers, "almost as big as the invention of photography”, and promise a new era in art, films, and other creative fields. "I wanted to hop on to this bandwagon as soon as possible; this is basically the premise over which my movie, ‘Salt’, was conceived, developed and maintained," he said. He calls it a fully AI-generated multi-plot interactive film experience.

It began with generating images using Midjourney, with the idea of weaving them together to build a trailer. "I don't come from a filmmaking background and used simple Da Vinci Resolve to cut it together," he said. However, the outcome turned out to be a truly experimental and innovative short story plot which he shared on Twitter and received rave reviews.

"I realised that filmmaking is a costly affair. Since my movie is completely made using AI, my first thought was that, effectively, the cost of production is zero or approaching zero. I did not want to settle down with just one plot but have multiple of them," he said. 

Stelzer would allow his followers to select from a range of prompts. The most popular ones would then be used to develop an alternative plot. He primarily used Midjourney to create Salt, along with Synthesia, but is now experimenting with Stable Diffusion and Avatarify for character animation.

"So the vision is to have an extremely bottom-up decentralised film universe where everyone is able to render their own episodes, plots, and stories," he said, and added that currently, his project is following the 'choose your own adventure' formula, where he receives inputs/prompts from his followers. He calls it the sandbox-style film. 

The final goal, however, is to decentralise in a way that anyone with a great idea can design their own plot and become a co-creator. Stelzer is an optimist and truly believes that AI will see rapid growth, and within four years, we may have text-to-video generation tools. The barrier right now is fixing an embedding in the latent space and recreating it for different contexts.

‘Battleprompts’

Another major project that Stelzer has worked on is ‘Battleprompts’. 

"The main thought behind this project was whether an AI native game would run on Twitter," he said. Inspired by the popular 80s game Cobra, ‘Battleprompts’ accepts text prompts to generate characters. "Player 1 can give a prompt like, “Kermit the Frog with four hands and laser eyes wearing Samurai swords”. Player 2's prompt could be “a space hamster that can change time and bend Physics”. With these prompt inputs the GPT-3 can generate a fight sequence between the avatars. It is like AI taking a roll on the dice," explained Stelzer. He used the GPT-3 Da Vinci model for this.

Stelzer believes that with sufficient technical support and advancement, we may have an AR version of this game. "You will have these strange, funny avatars fighting it out on your kitchen table," he said and added, "It's like those fun experiments, which become the predecessors to more serious projects in AI native entertainment media. This is what I'm interested in."

Ethics and morality

Right at the beginning of his conversation with us, Stelzer made a disclaimer that he does not think text-to-image generation tools will make the job of an artist or illustrator redundant. "When photography came around, people thought this would be the end of the art of painting. But even today, we have paintings, some even worth more than what they were 100 years ago. I believe going forth, we will see entirely new genres based on new synthetic media tools," he said,

When quizzed about the growing noise around the copyright issues associated with AI-based image generations, Stelzer said there was a misconception that AI tools are like search engines for images where it just pulls together art from online and recombines them in new ways. "It is not a search engine for images, it creates new images," he said.

"All art is inspired. Nobody can claim to really create completely original art. Everyone's standing on the shoulders of giants. So with respect to AI art, I would like to basically use the same principles – moral and legal principles – that apply to humans as well. There is a clear case of theft, and there is inspired art," Stelzer signed off.