Steadily treading its way and building a dominance in different work domains is ChatGPT. The chatbot is being heavily implemented across fields such as HR, Legal, Data Science, and politics to name a few. Companies are even working on their own LLM to stay relevant. To prove its prominence, ChatGPT is now being tested as a fund manager: a precarious step in stock management.

A recent report by finance website Finder showed that ChatGPT picked a theoretical stock portfolio of 38 companies that outperformed the S&P 500. For a duration of eight weeks, from March to April, the ChatGPT-selected funds climbed 4.93%, whereas a group of 10 popular funds in the UK averaged a loss of 0.78%. The chatbot’s funds outperformed real funds in 34 out of 39 market days. Promising results, for sure. But, does this experiment qualify ChatGPT as a reliable fund manager? 

Probably, an overstatement at this stage. 

Decoding ChatGPT Prediction

Last month, there was a research paper released on whether ChatGPT and other large language models can forecast stock price movements and return predictability. 

Predictability tools work on the principle of sentiment analysis. In the study, ChatGPT was used to indicate whether a business news headline is good, bad, or irrelevant for determining a firm’s stock price. Numerical scores were then computed and a positive correlation between these ChatGPT scores and daily stock market returns were documented. It was noted that ChatGPT was able to outperform traditional sentiment analysis methods, and other basic models such as GPT-1, GPT-2, and BERT were not able to accurately forecast returns, indicating that complex models were able to predict better. 

In a study by Finder, the ChatGPT fund was monitored over two months. The chatbot was instructed to create a stock portfolio from high-quality businesses with criteria such as minimal debts, sustained past growth, and assets that provide a competitive edge. Some of the companies that ChatGPT picked were Meta, Microsoft, Intel Corporation, Alphabet, NVIDIA, Visa and others. FMCG and beverage companies such as Johnson & Johnson, Coca Cola and PepsiCo Inc. were also picked. Meta, Microsoft and Intel were the best performing stocks with a  maximum increase of 30%, 20% and 18% respectively. 

Source: Finder

Not Conclusive

However, the experiment alone cannot be conclusive to prove the chatbot’s ability to predict stocks. The experiment was anecdotal and the time period of two months during which the activity was done is too less to determine anything conclusively. In addition, comparing it with actively managed funds may not necessarily be a proper benchmark. In fact, over a 20-year period, 95% of large-cap actively actively managed funds have underperformed their benchmark. 

With ChatGPT being trained on data up to September 2021, the performance of company stocks for the last two years is not tested, which is probably another setback. 

Future of AI in Investment Prediction

The CEO of Finder, Jon Ostler is apprehensive of the idea of using ChatGPT for investing research. He said that though big funds have been adopting AI for years, it’s not a good idea for the public to rely on a “rudimentary AI platform” that has claimed its data to be ‘patchy’ since September 2021, and does not have the knowledge of market psychology. 

He even mentioned that as per a survey conducted in 2021, investors use social media for investing advice. When compared to an unqualified TikTok star, he believes that AI would be the better choice, however, one should ideally not use any of it. He believes that researching through known primary sources or a qualified advisor would be a recommended method. 

Source:Finder

In the above study done by Finder to gauge people’s reaction, 8% of them have already used ChatGPT for financial advice. 19% of them would consider using it for financial data, whereas 35% would not consider it for financial advice. 

To add to the unpredictable nature of ChatGPT, there have been multiple problems of hallucinations and incorrect information that the chatbot produces. From false allegations to fake news, the bot is infamous for making wrong decisions. In addition, considering how input datasets can be maligned through activities such as data poisoning and prompt injections, the output of the chatbot can be compromised. In stock predictions, as sentiment analysis happens from input news headlines, if the training data is skewed, the predictions will be faulty as well. 

So, if you are planning to trust ChatGPT with managing your money, think again.