Microsoft Unveils Visual ChatGPT, A ChatGPT for Images

This connection enables sending messages through chat and receiving images during the chat, while also injecting a series of visual model prompts, to edit the images as well.
Microsoft Unveils Visual ChatGPT, A ChatGPT for Images
Listen to this story

Microsoft recently unveiled a new model called ‘Visual ChatGPT‘, which incorporates different types of Visual Foundation Models (VFMs) including Transformers, ControlNet, and Stable Diffusion with ChatGPT. The system enables interaction with ChatGPT beyond language

This connection enables sending messages through chat and receiving images during the chat, while also injecting a series of visual model prompts to edit the images as well.

Click here to check out the GitHub repository.

AIM Daily XO

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

The paper titled, ‘Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models‘, highlights how all the visual transformer models are only experts of specific tasks with fixed inputs and outputs, and the same is the case with ChatGPT as it is only trained on text. Combining all of them makes image generation and manipulation limitless.


Download our Mobile App

In order to bridge the gap between ChatGPT and VFMs, the paper proposes the use of a Prompt Manager that includes the following features: 

  • Explicitly inform ChatGPT about the capabilities of each VFM and specify the necessary input-output formats. 
  • Convert various types of visual information—such as png images, depth images, and mask matrices—into language format to aid ChatGPT’s understanding. 
  • Manage the histories, priorities, and conflicts of different VFMs. 

By utilising the Prompt Manager, ChatGPT can effectively leverage VFMs and receive their feedback in an iterative manner until the users’ requirements are met or a concluding condition is reached.
This enables users to interact with ChatGPT using images as well, more than only text. Moreover, users can also ask complex image questions or visual editing by the collaboration of different AI models in multi-steps. Users can also ask for corrections and feedback on results.

Sign up for The Deep Learning Podcast

by Vijayalakshmi Anandan

The Deep Learning Curve is a technology-based podcast hosted by Vijayalakshmi Anandan - Video Presenter and Podcaster at Analytics India Magazine. This podcast is the narrator's journey of curiosity and discovery in the world of technology.

Mohit Pandey
Mohit dives deep into the AI world to bring out information in simple, explainable, and sometimes funny words. He also holds a keen interest in photography, filmmaking, and the gaming industry.

Our Upcoming Events

24th Mar, 2023 | Webinar
Women-in-Tech: Are you ready for the Techade

27-28th Apr, 2023 I Bangalore
Data Engineering Summit (DES) 2023

23 Jun, 2023 | Bangalore
MachineCon India 2023 [AI100 Awards]

21 Jul, 2023 | New York
MachineCon USA 2023 [AI100 Awards]

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

Council Post: The Rise of Generative AI and Living Content

In this era of content, the use of technology, such as AI and data analytics, is becoming increasingly important as it can help content creators personalise their content, improve its quality, and reach their target audience with greater efficacy. AI writing has arrived and is here to stay. Once we overcome the initial need to cling to our conventional methods, we can begin to be more receptive to the tremendous opportunities that these technologies present.