|
Listen to this story
|
Yes, it’s possible. With the rise of AI agents that use LLMs to autonomously run tasks, the next step of evolution involves the integration of multiple agents that work together to accomplish tasks. With MetaGPT already serving the same purpose, it looks like more such agents are coming to the forefront — the recent one being ChatDev, a virtual chat-powered company that aids software development. The question is, what uniqueness does this agent bring to the table?
Communicative Agents
A team of 12 researchers from Dalian University of Technology, Beijing University and Brown University have built a multi-agent team ChatDev that will help build a software within minutes. ChatDev follows a structured approach similar to the waterfall model, a linear, sequential approach for software development.
It breaks down the development process into four clear phases: design, coding, testing, and documentation. Each phase involves a team of agents, including programmers, code reviewers, and test engineers, promoting teamwork and ensuring a smooth workflow.
Representation of ChatDev Functioning. Source: ChatDev
On receiving an assignment/task such as creating ‘a gomoku game’ as explained in the paper, the ChatDev agents actively engage in effective communication and mutual verification through collaborative chatting. This process enables them to automatically craft comprehensive software solutions that encompass source codes, environment dependencies, and user manuals.
A chat chain serves as a mediator, dividing each stage into smaller, individual tasks. This dual role allows for the suggestion and confirmation of solutions through context-aware communication, ultimately leading to the effective completion of specific subtasks.
What about MetaGPT?
When MetaGPT was introduced, the multi-agent framework was trending on GitHub with 20,000 stars. Similar to ChatDev, MetaGPT connects different AI agents that have been assigned various roles such as product managers, architects, project managers, and engineers, to function together. Though similar in implementing multiple agents, the purpose and approach taken by both are different.
Development vs Solution-based
ChatDev, a chat-powered company, is specifically focused on software development, whereas, MetaGPT is designed to enhance the capabilities of existing multi-agent systems that will specifically address the limitation in solving complex tasks.
MetaGPT achieves it by encoding Standardised Operating Procedures (SOPs) into prompts to improve structured coordination among agents. It also mandates modular outputs, empowering agents with domain expertise to validate results and reduce errors. Instead of relying solely on the language model’s inherent knowledge, specific guidelines and procedures are provided to guide the agents in their interactions. ChatDev, on the other hand, follows a waterfall method, a project management and development methodology, dividing the work into multiple stages such as designing, coding, etc, which is particularly catered for software development. It uses a chat chain to facilitate communication and task breakdown.
Large Language Model
ChatDev has been experimented on the gpt3.5-turbo-16k version of ChatGPT. On the other hand, MetaGPT employs GPT4- 32k, and is said to have surpassed GPT-4 in percentage of pass rates on MBPP and HumanEval. ChatDev has not been compared with other LLMs.
Costing
ChatDev paper mentions the astounding efficacy in software generation. It claims that the entire software development process took under seven minutes at a cost of less than $1. For a project using the MetaGPT framework, it takes 516 seconds on an average, and $1.12, with a maximum cost of $1.35.
Minimising Hallucinations
Creating software systems directly with LLMs can also produce code-related hallucinations. These issues might manifest as incomplete implementations, absent dependencies, and undetected bugs. Such hallucinations can arise due to task vagueness and a lack of cross-checking in the decision-making process. However, this is largely addressed in ChatDev by introducing thought instruction mechanisms into each autonomous chat process during code completion, reviewing and testing stage. By performing a ‘role flip’, an instructor injects specific thoughts for code modifications into instructions.
MetaGPT framework incorporates efficient human workflows as a meta programming approach into a LLM-based multi-agent collaboration, and looks to address hallucinations through it. However, no further details on how it will achieve it is given in the paper.
With ChatDev, the process of multiple teams and people to accomplish various tasks can be eliminated. Building an entire software within minutes is no easy feat, and ChatDev effortlessly accomplishes it saving time, cost and resources. If put to use, ChatDev-type models can probably revolutionise software development workflow.
