Article complete

Hong Kong Researchers Introduce Diffusion Model-Based Editing Tool

MagicQuill’s strength lies in real-time intent recognition via Multimodal Large Language Models (MLLMs).

The text "MagicQuil" in a stylized font on a black background.

Key takeaways

  • Introduce MagicQuill, a user-friendly image editing tool leveraging diffusion models and MLLMs.
  • Enhance image editing precision while reducing complexity for users of all skill levels.
  • Challenge existing tools like SmartEdit and Adobe's Palette by offering real-time intent recognition.
  • Incorporate specialized modules such as inpainting and control branches for advanced editing capabilities.
  • Collaborate with researchers from leading institutions to improve image editing technology.

Takeaways drafted with AI from the article text.

A group of researchers from the Hong Kong University of Sciences and Technology (HKUST) introduced MagicQuill, an advanced and interactive image editing system based on diffusion models. It is designed to provide a user-friendly experience by combining AI-driven suggestions with precise local editing capabilities.

The system builds on Stable Diffusion v1.5, enhanced with specialised modules such as Dual-branch architecture, inpainting and control branches, and Fine-tuned LLaVA-1.5, a multimodal large language model.  

The paper challenges existing tools such as SmartEdit for instruction-based editing, BrushNet for mask and prompt-guided inpainting, and GAN-based tools like SketchEdit. Even Adobe’s Palette is a competitor in this space.

As compared to platforms like Dragdiffusion or Zone, similar advanced systems for image editing, MagicQuill strength lies in real-time intent recognition via Multimodal Large Language Models (MLLMs). Besides, it’s meant for general-purpose streamlined editing as against precision oriented editing offered by other systems. 

Researchers from Ant Group, Zhejiang University (ZJU), and The University of Hong Kong (HKU), have also contributed to this paper. 

What does MagicQuill solve?

According to their research paper, MagicQuill addresses limitations in existing tools by improving precision, reducing complexity, and making sophisticated image editing accessible to users of all skill levels.

MagicQuill’s Painting Assistor employs a multimodal large language model (MLLM) to predict editing prompts in real time from brushstrokes, eliminating the need for manual input. The user-friendly Idea Collector interface streamlines interaction, allowing iterative edits, stroke management, and result previews. 

However, the research has limitations, including quality degradation when sketches deviate from prompts, loss of fine details during colour adjustments, and misinterpretation of simple or ambiguous sketches. It requires high-end hardware (15 GB VRAM) and lacks advanced features like reference-based editing, planned for future updates. 

Published Updated

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

diffusion model · 5 stories
  1. Google Just Stepped Up the Game for Text-to-Image AI
  2. NVIDIA is Late to Party but Solves Key Issues with Diffusion Models
  3. Google Releases a New Text-Based Image Editing Model Called ‘Imagic’
  4. This New AI tool Could Mark the Beginning of the End for TikTok and Instagram Influencers
  5. You are here: Hong Kong Researchers Introduce Diffusion Model-Based Editing Tool
About the author

Contributor

I hold a degree in political science, and am interested in how AI and online culture intersect. I am at & x.com/aditisuresh12

All stories
Belamy · Daily

The day’s AI news, before your first meeting.