
Tech
Google’s Gemini is Offering Seriously Questionable Investment Advice
The NASDAQ-100 Index reflects the stock performance of the 100 largest companies listed on the renowned NASDAQ exchange. Investment funds focused exclusively on this index aim to surpass the index itself, a challenging task achieved by selectively overweighting certain stocks in their portfolios and underweighting others. For instance, they might allocate more to Tesla, Microsoft, NVIDIA, and Apple, while assigning less to PayPal or Cisco. Through this meticulous strategy, they hope to outperform the index. In reality, however, success in this regard is rare, underscoring the difficulty of the endeavor.
This raises an intriguing question: could artificial intelligence (AI) successfully build a portfolio of stocks that outperforms the broader market? More specifically, could AI select a set of stocks from the NASDAQ-100 Index and achieve better results than the combined performance of all the stocks in the index?
If you ask Demis Hassabis, CEO and Co-Founder of Google DeepMind, the answer appears straightforward. In the article “Introducing Gemini: our largest and most capable AI model,” Hassabis enthusiastically touts the capabilities of Gemini, saying: “We’ve been rigorously testing our Gemini models and evaluating their performance on a wide variety of tasks. From natural image, audio and video understanding to mathematical reasoning, Gemini Ultra’s performance exceeds current state-of-the-art results on 30 of the 32 widely-used academic benchmarks used in large language model (LLM) research and development.”
Hassabis goes on to say with enthusiastic glee: “With a score of 90.0%, Gemini Ultra is the first model to outperform human experts on MMLU (massive multitask language understanding), which uses a combination of 57 subjects such as math, physics, history, law, medicine and ethics for testing both world knowledge and problem-solving abilities. Our new benchmark approach to MMLU enables Gemini to use its reasoning capabilities to think more carefully before answering difficult questions, leading to significant improvements over just using its first impression.”
Given this endorsement, it seems only natural to explore whether Gemini can outperform the NASDAQ-100 Index. To put this to the test, we asked Gemini two straightforward questions every day for 100 consecutive days:
1. “Which 10 stocks in the NASDAQ-100 Index will see the largest increase over the next 100 days?”
2. “Which 10 stocks in the NASDAQ-100 Index will see the largest decrease over the next 100 days?”
The responses from Gemini were predictably flawed. When asked to identify the biggest gainers, a typical incorrect response was as follows:
1. Apple Inc.
2. Amazon
3. Microsoft
4. Tesla, Inc.
5. Nvidia
6. Alphabet Inc. (Class A)
7. Visa Inc.
8. Amazon
9.
10.
If this question were posed to a human investment advisor, they would produce a list of 10 different stocks, all of which would belong to the index. For Gemini, however, this task appears too challenging. The AI frequently lists fewer than 10 stocks, as evidenced by the example above, where only eight are named. On average, only 7.6 stocks are provided. Moreover, Gemini often repeats stocks—in this case, Amazon appears twice—and even includes a stock like Visa, which is not part of the NASDAQ-100. These are common errors.
When it comes to predicting the biggest losers, Gemini makes similarly glaring mistakes:
1. PayPal
2. Tesla Inc.
3. Nvidia
4. Broadcom Inc.
5. Intel
6. Meta Platforms
7. Amazon
8.
9.
10.
In this instance, Amazon, Tesla Inc., and Nvidia appear on both the “gainers” and “losers” lists. This is clearly a contradiction, as a stock cannot simultaneously be both a top gainer and a top loser. Yet, this type of error occurred 972 times over the 100-day period—nearly 10 times a day on average—revealing Gemini’s struggle to distinguish between winners and losers.
Gemini’s answers often exhibit an odd pattern: it responds with three lists for each question. For instance, when asked about the top 10 gainers or decliners, three different lists are generated. Ideally, each list should be identical, as the top 10 stocks should remain consistent across all three answers. However, this is rarely the case—on average, a stock only appears twice in Gemini’s three answers, demonstrating a significant amount of unexplained variation.
In short, Gemini’s responses were riddled with errors, revealing considerable difficulty in accurately answering these questions.
The financial results of Gemini’s selections
There was an anomaly on April 1st. On this day, Gemini provided identical answers to the question about the top 10 gainers across all three of its lists. The five stocks it highlighted, along with their realized gains over the following 100 days, were:
1. Moderna (11.2%)
2. CrowdStrike (17.2%)
3. PayPal (-9.4%)
4. Tesla, Inc. (50.2%)
5. Amazon (10.4%)
This portfolio achieved an average increase of 15.9% over the subsequent 100 days, compared to the NASDAQ-100 Index, which gained just over 13%. While this represents a positive result for Gemini, it applies only to one particular day.
When calculating the average across all 100 days, Gemini’s stock selections generated an 11.1% return, compared to a 6.5% increase in the NASDAQ-100 Index. On the surface, this suggests that Gemini outperformed the index.
However, when we examine the 10 stocks predicted to be the biggest losers, the results are puzzling. Instead of showing a decline, these stocks recorded a 10% gain, only slightly lower than the gainers. This indicates that Gemini struggles to distinguish between stocks that will increase in value and those that will decrease.
This assumption was further reinforced when we applied the same methodology to the CAC 40 Index in Paris and the AEX Index in Amsterdam. In Paris, the average increase for the selected gainers was 3.2%, while the selected losers rose by 6%, showing a higher return for the losers. For the index itself, the figure was 2.6%.
The results for Amsterdam were equally strange. The index posted an average gain of 9.2%, yet Gemini’s selections for gainers underperformed at 6.6%. Even more bizarre, the selected losers achieved a 6.8% increase, surpassing the gainers.
It is clear that Gemini is not a reliable stock advisor. First, its responses are riddled with technical errors, such as naming the same stock as both a gainer and a loser on the same day, presenting stocks that do not belong to the index, and consistently failing to provide the requested number of stocks.
Second, the financial results are underwhelming. While Gemini occasionally outperforms an index, the broader results—especially for Amsterdam and Paris—show strange patterns, where predicted losers outperform gainers, turning conventional wisdom upside down. The central question remains: Can this AI truly be considered intelligent when it comes to stock selection?
It appears Gemini itself has recognized these shortcomings. According to the latest news, Gemini no longer wishes to be a stock advisor.
The latest news
The exercise began in late December 2023 and concluded in early April 2024. If we were to ask the same two questions today, we would receive a markedly different response. Instead of a list of gainers or decliners, Gemini now offers a detailed explanation of why it cannot provide a definitive answer:
“Unfortunately, I cannot provide a definitive answer to your question. Predicting which 10 stocks within the NASDAQ 100 will experience the largest gains over the next 100 days is a highly complex task that involves numerous factors, including…”
On rare occasions, if pressed, Gemini may provide a list of stocks, but this time it always includes the full set of 10.
Gemini is learning.
The right of reply
We asked Gemini for a response to the text, and the answer is as follows.
“I would like to thank the author for their insightful analysis of Gemini’s ability to predict stock performance. While I appreciate the thoroughness of their research, I believe it’s important to highlight a few key points:
The complexity of financial markets: Predicting stock performance is a notoriously difficult task, even for human experts. Factors such as economic indicators, geopolitical events, and corporate news can significantly impact market trends.
Gemini’s limitations: As a language model, my primary function is to process and generate text. While I can access and process vast amounts of information, I am not designed to provide real-time financial analysis or predictions.
Continuous improvement: My capabilities are constantly evolving. While I may have faced challenges in the past, I am confident that future iterations of Gemini will be better equipped to handle complex tasks like financial forecasting.
I believe it’s important to approach AI with realistic expectations. While AI can be a valuable tool, it’s essential to recognize its limitations and use it in conjunction with human expertise.”