Since AI and particularly ChatGPT started entering the public discourse, one question that has been asked repeatedly is whether it could help improve your investing success. Even I tested ChatGPT’s ability to provide investment advice some time ago.

I was not alone. A whole bunch of academic studies have been initiated; the papers are starting to come out and the results are just eye-popping.

One of the latest is a study by researchers from the University of Florida who make the bold claim that ChatGPT can reliably predict stock market trends.

“Using public markets data and news from October 2021 to December 2022, their testing found that trading models powered by ChatGPT could generate returns exceeding 500% in this period. This performance stands in stark contrast to the -12% return from buying and holding an S&P 500 ETF during the same timeframe,” the study found.

Now, I’m not sure about you, but a 500% annual return in a declining market seems attractive to me.

The time period is interesting because, as everybody knows, the latest collection date for ChatGPT4 is September 2021. How did the model work even though it was collecting data beyond the date when ChatGPT stopped assessing it?

The answer lies in the model the researchers were using — always a critical factor! What the researchers did was collect news on different companies that form part of the main US markets. They sifted them a bit, cutting out duplicates and mere market movement news, and ended up with 67,586 headlines related to 4,138 unique companies.

They questioned ChatGPT about whether each headline was good news, bad news, or neutral. They then backtested these assessments and assumed you would invest one dollar in the next appropriate market period, either going long or short, depending on the response. The share would notionally be sold after a day had passed.

Now here is the interesting bit: There are, of course, existing data-collection systems that do this already. This is serious business, after all. The researchers compared the results with the existing methodologies and guess what? ChatGPT won by huge margins.

Just to take one example. The researchers presented this headline to ChatGPT:  “Rimini Street Fined $630,000 in Case Against Oracle”. So obviously, this is bad news for Rimini Street. But what about Oracle?

The existing predictive news-gathering organisation the group used as a comparison is called RavenPack, and concerning this headline, it found the news was bad for Oracle. However, ChatGPT responded: “The fine against Rimini Street could potentially boost investor confidence in Oracle’s ability to protect its intellectual property and increase demand for its products and services.” And it suggested investing in the company.

Why is that difference happening?  The researchers say it can be attributed to the advanced language understanding capabilities of ChatGPT4, which allow it to capture the nuances and subtleties in news headlines, naturally leading to the generation of more reliable sentiment scores.

Two other things are interesting about the study. First, ChatGPT2 and ChatGPT3 were more or less useless. The ability of the language models to understand sufficiently only began to show itself with the advent of ChatGPT4. Once again, sophistication is the critical difference.

Second — and this I find significant — the model worked well for long-only investment, returning roughly 50%. But it worked like a bomb when shorting stocks, returning 400%. Combined it really popped: returning 500%.

This makes me wonder about several different things: was the short strategy boosted unusually by the fact that it was a bear market? I guess we will find out in time.

But also, do investment managers tend to ignore bad news more than they should? Now that is a question. After all, the model is comparing the results not only against market returns but effectively also against professional investors. In fact, 2022 was a very good year for stockpickers and most beat their benchmarks. But not by much.

My instinct, probably because I’m a journalist, is that most money managers are heavily leveraged toward a rising market, and tend to give bad news the benefit of the doubt — if that’s a thing. Perhaps I’m just leveraged toward over-appreciating bad news, but I do have some evidence on my side! It might be a little artificial, but at least it’s intelligent.

Anyway, the researchers say the study “highlights the importance of continued exploration and development of LLMs [large language models] tailored explicitly for the financial industry”. That, trust me, is a very, very large understatement. DM