The way Jeremy and I see it, artificial intelligence (AI) really leapt into the zeitgeist in late-2022 or early-2023 with the public introduction of DALL-E2 and ChatGPT. Both are provided by OpenAI and are known as generative AI products – they are software that use AI to generate art and text, respectively (and often at astounding quality), hence the term “generative”. Since then, developments in AI have progressed at a breathtaking pace. One striking observation I’ve found with AI is the much higher level of enthusiasm that company-leaders have for the technology compared to the two other recent “hot things”, namely, blockchain/cryptocurrencies and the metaverse. Put another way, AI could be a real game changer for societies and economies.
I thought it would be useful to write down some of my current thoughts on AI and its potential impact. Putting pen to paper (or fingers to the keyboard) helps me make sense of what’s in my mind. Do note that my thoughts are fragile because the field of AI is developing rapidly and there are many unknowns at the moment. In no order of merit:
- While companies such as OpenAI and Alphabet have released generative AI products, they have yet to release open-source versions of their foundational AI models that power the products. Meta Platforms, meanwhile, has been open sourcing its foundational AI models in earnest. During Meta’s latest earnings conference call in April this year, management explained that open sourcing allows Meta to benefit from improvements to its foundational models that are made by software developers, outside of Meta, all over the world. Around the same time, there was a purportedly leaked document from an Alphabet employee that discussed the advantages in the development of AI that Meta has over both Alphabet and OpenAI by virtue of it open sourcing its foundational models. There’s a tug-of-war now between what’s better – proprietary or open-sourced foundational AI models – but it remains to be seen which will prevail or if there will even be a clear winner.
- During Amazon’s latest earnings conference call (in April 2023), the company’s management team shared their observation that most companies that want to utilise AI have no interest in building their own foundational AI models because it takes tremendous amounts of time and capital. Instead, they merely want to customise foundational models with their own proprietary data. On the other hand, Tencent’s leaders commented in the company’s May 2023 earnings conference call that they see a proliferation of foundational AI models from both established companies as well as startups. I’m watching to find out which point of view is closer to the truth. I also want to point out that the frenzy to develop foundational AI models may be specific to China. Rui Ma, an astute observer of and writer on China’s technology sector, mentioned in a recent tweet that “everyone in China is building their own foundational model.” Meanwhile, the management of online travel platform Airbnb (which is based in the US, works deeply with technology, and is clearly a large company) shared in May 2023 that they have no interest in building foundational AI models – they’re only interested in designing the interface and tuning the models.
- A database is a platform to store data. Each piece of software requires a database to store, organize, and process data. The database has a direct impact on the software’s performance, scalability, flexibility, and reliability, so its selection is a highly strategic decision for companies. In the 1970s, relational databases were first developed and they used a programming language known as Structured Query Language (SQL). Relational databases store and organise data points that are related to one another in table form (picture an Excel spreadsheet) and were useful from the 1980s to the late 1990s. But because they were used to store structured data, they began to lose relevance with the rise of the internet. Relational databases were too rigid for the internet era and were not built to support the volume, velocity, and variety of data in the internet era. This is where non-relational databases – also known as NoSQL, which stands for either “non SQL” or “not only SQL” – come into play. NoSQL databases are not constrained to relational databases’ tabular format of data storage and can work with unstructured data such as audio, video, and photos. As a result, they are more flexible and better suited for the internet age. AI appears to require different database architectures. The management of MongoDB, a company that specialises in NoSQL databases, talked about the need for a vector database to store the training results of large language models during the company’s June 2023 earnings conference call. Simply put, a vector database stores data in a way that allows users to easily find data, say, an image (or text), that is related to a given image (or text) – this feature is very useful for generative AI products. This said, MongoDB’s management also commented in the same earnings conference call that NoSQL databases will still be very useful in the AI era. I’m aware that MongoDB’s management could be biased, but I do agree with their point of view. Vector databases appear to be well-suited (to my untrained technical eye!) for a narrow AI-related use case, whereas NoSQL databases are useful in much broader ways. Moreover, AI is likely to increase the volume of software developed for all kinds of software – not just AI software – and they need modern databases. MongoDB’s management also explained in a separate June 2023 conference that a typical generative AI workflow will include both vector databases and other kinds of databases (during the conference, management also revealed MongoDB’s own vector database service). I’m keeping a keen eye on how the landscape of database architectures evolve over time as AI technologies develop.
- Keeping up with the theme of new architectures, the AI age could also usher in a new architecture for data centres. This new architecture is named accelerated computing by Nvidia. In the traditional architecture of data centres, CPUs (central processing units) are the main source of computing power. In accelerated computing, the entire data centre – consisting of GPUs (graphic processing units), CPUs, DPUs (data processing units), data switches, networking hardware, and more – provides the computing power. Put another way, instead of thinking about the chip as the computer, the data centre becomes the computer under the accelerated computing framework. During Nvidia’s May 2023 earnings conference call, management shared that the company had been working on accelerated computing for many years but it was the introduction of generative AI – with its massive computing requirements – that “triggered a killer app” for this new data centre architecture. The economic opportunity could be immense. Nvidia’s management estimated that US$1 trillion of data centre infrastructure was installed over the last four years and nearly all of it was based on the traditional CPU-focused architecture. But as generative AI gains importance in society, data centre infrastructure would need to shift heavily towards the accelerated computing variety, according to Nvidia’s management.
- And keeping with the theme of something new, AI could also bring about novel and better consumer experiences. Airbnb’s co-founder and CEO, Brian Chesky, laid out a tantalising view on this potential future during the company’s latest May 2023 earnings conference call. Chesky mentioned that search queries in the travel context are matching questions and the answers depend on who the questioner is and what his/her preferences are. With the help of AI, Airbnb could build “the ultimate AI concierge that could understand you,” thereby providing a highly personalised travel experience. Meanwhile, in a recent interview with Wired, Microsoft’s CEO Satya Nadella shared his dream that “every one of Earth’s 8 billion people can have an AI tutor, an AI doctor, a programmer, maybe a consultant!”
- Embedded AI is the concept of AI software that is built into a device itself. This device can be a robot. And if robots with embedded AI can be mass-produced, the economic implications could be tremendous, beyond the impact that AI could have as just software. Tesla is perhaps the most high profile company in the world today that is developing robots with embedded AI. The company’s goal for the Tesla Bot (also known as Optimus) is for it to be “a general purpose, bi-pedal, autonomous humanoid robot capable of performing unsafe, repetitive or boring tasks.” There are other important companies that are working on embedded AI. For example, earlier this year, Nvidia acquired OmniML, a startup whose software shrinks AI models, making it easier for the models to be run on devices rather than on the cloud.
- Currently, humans are behind the content trained on by foundational AI models underpinning the likes of ChatGPT and other generative AI products. But according to a recently-published paper from UK and Canadian researchers titled The Curse of Recursion: Training on Generated Data Makes Models Forget, the quality of foundational AI models degrades significantly as the proportion of content they are trained on shifts toward an AI-generated corpus. This could be a serious problem in the future if there’s an explosion in the volume of generative AI content, which seems likely; for context, Adobe’s management shared in mid-June this year that the company’s generative AI feature, Firefly, had already powered 500 million content-generations since its launch in March 2023. The degradation, termed “model collapse” by the researchers, happens because content created by humans are a more accurate reflection of the world since they would contain improbable data. Even after training on man-made data, AI models tend to generate content that understates the improbable data. If subsequent AI models train primarily on AI-generated content, the end result is that the improbable data become even less represented. The researchers describe model collapse as “a degenerative process whereby, over time, models forget the true underlying data distribution, even in the absence of a shift in the distribution over time.” Model collapse could have serious societal consequences; one of the researchers, Ilia Shumailov, told Venture Beat that “there are many other aspects that will lead to more serious implications, such as discrimination based on gender, ethnicity or other sensitive attributes.” Ross Anderson, another author of the paper, wrote in a blog post that with model collapse, advantages could accrue to companies that “control access to human interfaces at scale” or that have already trained AI models by scraping the web when human-generated content was still overwhelmingly dominant.
There’s one other fragile thought I have about AI that we think is more important than what I’ve shared above, and it is related to the concept of emergence. Emergence is a natural phenomenon where sophisticated outcomes spontaneously “emerge” from the interactions of agents in a system, even when these agents were not instructed to produce these outcomes. The following passages from the book, Complexity: The Emerging Science at the Edge of Order and Chaos by Mitch Waldrop, help shed some light on emergence:
“These agents might be molecules or neurons or species or consumers or even corporations. But whatever their nature, the agents were constantly organizing and reorganizing themselves into larger structures through the clash of mutual accommodation and mutual rivalry. Thus, molecules would form cells, neurons would form brains, species would form ecosystems, consumers and corporations would form economies, and so on. At each level, new emergent structures would form and engage in new emergent behaviors. Complexity, in other words, was really a science of emergence…
…Cells make tissues, tissues make organs, organs make organisms, organisms make ecosystems – on and on. Indeed, thought Holland, that’s what this business of “emergence” was all about: building blocks at one level combining into new building blocks at a higher level. It seemed to be one of the fundamental organizing principles of the world. It certainly seemed to appear in every complex, adaptive system that you looked at…
…Arthur was fascinated by the thing. Reynolds had billed the program as an attempt to capture the essence of flocking behavior in birds, or herding behavior in sheep, or schooling behavior in fish. And as far as Arthur could tell, he had succeeded beautifully. Reynolds’ basic idea was to place a large collection of autonomous, birdlike agents—“boids”—into an onscreen environment full of walls and obstacles. Each boid followed three simple rules of behavior:
1. It tried to maintain a minimum distance from other objects in the environment, including other boids.
2. It tried to match velocities with boids in its neighborhood.
3. It tried to move toward the perceived center of mass of boids in its neighborhood.
What was striking about these rules was that none of them said, “Form a flock.” Quite the opposite: the rules were entirely local, referring only to what an individual boid could see and do in its own vicinity. If a flock was going to form at all, it would have to do so from the bottom up, as an emergent phenomenon. And yet flocks did form, every time. Reynolds could start his simulation with boids scattered around the computer screen completely at random, and they would spontaneously collect themselves into a flock that could fly around obstacles in a very fluid and natural manner. Sometimes the flock would even break into subflocks that flowed around both sides of an obstacle, rejoining on the other side as if the boids had planned it all along. In one of the runs, in fact, a boid accidentally hit a pole, fluttered around for a moment as though stunned and lost—then darted forward to rejoin the flock as it moved on.”
In our view, the concept of emergence is important in AI because at least some of the capabilities of ChatGPT seen today were not explicitly programmed for – they emerged. Satya Nadella said in his aforementioned interview with Wired that “when we went from GPT 2.5 to 3, we all started seeing these emergent capabilities.” Nadella was referring to the foundational AI models built by OpenAI in his Wired interview. One of the key differences between GPT 2.5 and GPT 3 is that the former contains 1.5 billion parameters, whereas the latter contains 175 billion, more than 100 times more. The basic computational unit within an AI model is known as a node, and parameters are a measure of the strength of a connection between two nodes. The number of parameters can thus be loosely associated with the number of nodes, as well as the number of connections between nodes, in an AI model. With GPT 3’s much higher number of parameters compared to GPT 2.5, the number of nodes and number of connections (or interactions) between nodes in GPT 3 thus far outweigh those of GPT 2.5. Nadella’s observation matches those of David Ha, an expert on AI whose most recent role was the head of research at Stability AI. During a February 2023 podcast hosted by investor Jim O’Shaughnessy, Ha shared the following (emphasis is mine):
“Then the interesting thing is, sure, you can train things on prediction or even things like translation. If you have paired English to French samples, you can do that. But what if you train a model to predict itself without any labels? So that’s really interesting because one of the limitations we have is labeling data is a daunting task and it requires a lot of thought, but self-labeling is free. Like anything on the internet, the label is itself, right? So what you can do is there’s two broad types of models that are popular now. There’s language models that generate sequences of data and there’s things like image models, Stable Diffusion you generate an image. These operate on a very similar principle, but for things like language model, you can have a large corpus of text on the internet. And the interesting thing here is all you need to do is train the model to simply predict what the next character is going to be or what the next word is going to be, predict the probability distribution of the next word.
And such a very simple objective as you scale the model, as you scale the size and the number of neurons, you get interesting emerging capabilities as well. So before, maybe back in 2015, ’16, when I was playing around with language models, you can feed it, auto Shakespeare, and it will blab out something that sounds like Shakespeare.
But in the next few years, once people scaled up the number of parameters from 5 million, to a hundred million, to a billion parameters, to a hundred billion parameters, this simple objective, you can now interact with the model. You can actually feed in, “This is what I’m going to say,” and the model takes that as an input as if it said that and predict the next character and give you some feedback on that. And I think this is very interesting, because this is an emergent phenomenon. We didn’t design the model to have these chat functions. It’s just like this capability has emerged from scale.
And the same for image side as well. I think for images, there are data sets that will map the description of that image to that image itself and text to image models can do things like go from a text input into some representation of that text input and its objective is to generate an image that encapsulates what the text prompt is. And once we have enough images, I remember when I started, everyone was just generating tiny images of 10 classes of cats, dogs, airplanes, cars, digits and so on. And they’re not very general. You can only generate so much.
But once you have a large enough data distribution, you can start generating novel things like for example, a Formula 1 race car that looks like a strawberry and it’ll do that. This understanding of concepts are emergent. So I think that’s what I want to get at. You start off with very simple statistical models, but as you increase the scale of the model and you keep the objectives quite simple, you get these emergent capabilities that were not planned but simply emerge from training on that objective.”
Emergence occurred in AI models as their number of parameters (i.e. the number of interactions between nodes) grew. This is a crucial point because emergence requires a certain amount of complexity in the interactions between agents, which can only happen if there are large numbers of agents as well as interactions between agents. It’s highly likely, in my view, that more emergent phenomena could develop as AI models become even more powerful over time via an increase in their parameters. It’s also difficult – perhaps impossible – to predict what these emergent phenomena could be, as specific emergent phenomena in any particular complex system are inherently unpredictable. So, any new emergent phenomena from AI that springs up in the future could be anywhere on the spectrum of being wildly positive to destructive for society. Let’s see!
Disclaimer: The Good Investors is the personal investing blog of two simple guys who are passionate about educating Singaporeans about stock market investing. By using this Site, you specifically agree that none of the information provided constitutes financial, investment, or other professional advice. It is only intended to provide education. Speak with a professional before making important decisions about your money, your professional life, or even your personal life. We currently have a vested interest in Alphabet, Amazon, Meta Platforms, Microsoft, MongoDB, Tencent, and Tesla. Holdings are subject to change at any time.