Meet The Sketchy Cousin of Stable Diffusion, ‘Scribble Diffusion’

What more than you need if AI can convert your rough scribbling into a refined image, and make it look like a masterpiece? An engineer from Replicate, who goes by the GitHub name zeke, created just that. Scribble Diffusion, is an open-source app that can convert your hand drawn art, along with a text-prompt, into an art piece.

The model is built on ControlNet, a neural network structure developed by Lvmin Zhang and Maneesh Agrawala for controlling large diffusion models with additional input conditions, which is human scribbling in this case. The model conditions Stable Diffusion with other inputs like edge detection, depth maps, and pose estimation.

Check out the app here.

ControlNet was released only three weeks ago and the developers are already building apps with it. Another developer built an app that generates DreamBooth images by matching the pose of the input stick figure. 

Click here to check out the GitHub repository and build your own app.

Stable Diffusion Cousins

Since the release of open-source Stable Diffusion, developers and ML enthusiasts have been experimenting with it to create different types of models. 

Firstly, it does not take much time for a new technology to be released and users start trying to make NSFW content with it. That is what happened with Unstable Diffusion. Stability AI has been under a lot of scrutiny since the release of this model.

Other models include, Riffusion, for generating real-time AI music by converting images of audio into audio. The approach involves fine-tuning Stable Diffusion for creating spectrograms, which are visualisations of audio. Another model, Amazing AI, was built on top of Stable Diffusion 1.5 that runs natively on macOS, without the need of the internet.