Bill Gates feels Generative AI has plateaued, says GPT-5 will not be any better::The billionaire philanthropist in an interview with German newspaper Handelsblatt, shared his thoughts on Artificial general intelligence, climate change, and the scope of AI in the future.
I’m not sure I’d say it’s plateaued today but I definitely think machine learning is going to hit a wall soon. Some tech keeps improving until physical limits stop progress but I see generative AI as being more like self-driving cars where the “easy” parts end up solved but the last 10% is insanely hard.
There’s also the economic reality of scaling. Maybe the “hard” problems could, in theory, be easily solved with enough compute power. We’ll eventually solve those problems but it’s going to be on Nvidia’s timeline, not OpenAI’s.
Generative ai is a bit different from self driving cars in the sense that they’re tolerant to failures. This may give more room for improvements when compared to other applications.
deleted by creator
Yes, especially when you consider that the human brain runs on 15W of power!
Let me save you a click: he doesn’t say anything interesting about why he thinks this.
What does Bill Gates know about GenAI? Is he an expert on the subject?
He’s not. He might pay experts, but he isn’t one.
Cool, Bill Gates has opinions. I think he’s being hasty and speaking out of turn and only partially correct. From my understanding, the “big innovation” of GPT-4 was adding more parameters and scaling up compute. The core algorithms are generally agreed to be mostly the same from earlier versions (not that we know for sure since OpenAI has only released a technical report). Based on that, the real limit on this technology is compute and number of parameters (as boring as that is), and so he’s right that the algorithm design may have plateaued. However, we really don’t know what will happen if truly monster rigs with tens-of-trillions of parameters are used when trained on the entirety of human written knowledge (morality of that notwithstanding), and that’s where he’s wrong.
You got it the wrong way around. We already have a ton of compute and what this kind of AI can do is pretty cool.
But adding more compute power and parameters won’t solve the inherent problems.
No matter what you do, it’s still just a text generator guessing the next best word. It doesn’t do real math or logic, it gets basic things wrong and hallucinates new fake facts.
Sure, it will get slightly better still, but not much. You can throw a million times the power at it and it will still fuck up in just the same ways.
it’s still just a text generator guessing the next best word. It doesn’t do real math or logic, it gets basic things wrong and hallucinates new fake facts.
If humans are any kind of yardstick here, I’d say all this is true of us too on many levels. The brain is a shortcut engine, not a brute force computer. It’s not solving equations to help you predict where that tennis ball will bounce next. It’s making guesses based on its corpus of past experience. Good enough guesses are frankly our brains’ bread and butter and most of us get through most days on little more than this.
It’s true that we can do more. Some of us, anyway. How many people actually exercise math and logic though? Sometimes it seems like… not a lot. And how many people hallucinate fake facts? A lot.
It’s much like evaluating self-driving cars. We may be tempted to say they’re just bloody awful, but so are human drivers.
I’d say the majority of humans know what 2 + 2 is. Chat GPT doesn’t. As it found the answer in some texts it will tell you 4, but all it takes is you telling it that’s wrong and suddenly it’s 5. So even for the most simple math problem it’s extremely easy to throw the whole thing off. Which also means for any prompt you put in it can go in wildly wrong directions at times.
And this is all with good input data, there’s plenty of trolls online and the data will only get worse (it already did, the original data up to 2021 was okayish, in the last year tons of crap was put out on top, some of it by Chat GPT itself. So the new model might input the crap it produced before, getting worse over time). The problem on top of that is that you don’t know the sources it used. If you ask about a recent event you might receive an insane answer it picked up from a right wing conspiracy site, you simply don’t know. There is no fact checking in place.
It’s a stunningly good text generator, but that’s all it is and it ever will be, at least until they do much more than just add more compute power to it.
Hehe. I’m imagining sitting 100 human test subjects down in a lab setting and asking them what 2+2 is, and then telling them they’re wrong when they answer 4. I don’t know how many of them would guess again but I know it’s not zero. Meanwhile, GPT can probably give a better answer to any advanced math or science query than the majority of humans.
I’m a writer and a language nerd and I watch people all the time use words incorrectly because they think they know what they mean, but they really don’t. They’re just regurgitating them in what they think is the same situation they heard them. They don’t “understand” the word and are just guessing and churning out crap.
I don’t have a dog in this race but I think it’s interesting how people judge artificial intelligence with too much credit given to what goes on with human intelligence. Most people who say it’s “just a next word predictor” read that phrase somewhere and are regurgitating it, not at all dissimilarly to what LLMs do. They use phrases like “it doesn’t actually understand” without being able to define, with any clarity or precision, and without resorting to examples, what would actually impress them as real intelligence.
GPT can probably give a better answer to any advanced math or science query than the majority of humans
Only if that answer is already out there and in the model. So pretty much a Google search away.
GPT isn’t coming up with new math or science facts (at least not real ones).
It literally is a word predictor, an insanely complex one, it’s the best way to describe it. If you start with layers, parameters and so on most people lose interest. But there’s some really good explanations around.
Generic AI (real AI) has internal logic, can learn and improve itself and can do self motivated actions. Chat GPT can tell you exactly how to create an account and order something from Amazon, but despite being able to put that text out it will never be able to actually follow them itself.
Only if that answer is already out there
Again, pretty similar to the vast majority of humans. How many times in your science education did you learn ann equation that you’d already figured out on your own previously?
And to be fair, GPT doesn’t have hands and the ability to conduct experiments. So we have to, in a sense, judge its success on an apples to apples basis of what it, and we, do with the corpus of written knowledge.
In contrast to humans, GPT has at least read it all ;) (I say this in jest - I know it doesn’t have access to everything, but humans are too lazy to read, for the most part, even things they have access to).
Only if that answer is already out there and in the model.
That’s not true. I wanted a vba script for Excel. I don’t know vba or excel so I spent hours searching Google for help. There were explanations of functions but no working code. I tried GPT for the fun of it and it spit back working code. Code that was nowhere on the Internet.
It was able to put together functions into working code based on the definition of functions, not simply cutting and pasting what somebody else had already written.
Chat GPT can tell you exactly how to create an account and order something from Amazon, but despite being able to put that text out it will never be able to actually follow them itself.
This is a really good ELI5 explanation of its limit.
That would still give it too much credit in that case. It’s purely an input output system. You put text in (the prompt), you get text out (the result). If there is no input from you, there is no output. It doesn’t have any intrinsic functionality that runs on its own.
Maybe a bit too much for an ELI5.
How is it a definitional limit on its intelligence that it can’t use interfaces designed for people with hands? You also cannot send an http request with your lips no matter how you try - that’s just not an interface made for you.
Bots can 100% operate websites and take online actions, conduct quality tests, write fake reviews. That doesn’t mean they are intelligent. I just can’t see how it has any bearing either way whether ChatGPT can place an Amazon order.
This is short-sighted.
The jump to GPT 3.5 was preceded by the same general misunderstanding (we’ve reached the limit of what generative pre-trained transformers can do, we’ve reached diminishing returns, ECT.) and then a relatively small change (AFAIK it was a couple additional layers of transforms and a refinement of the training protocol) and suddenly it was displaying behaviors none of the experts expected.
Small changes will compound when factored over billions of nodes, that’s just how it goes. It’s just that nobody knows which changes will have that scale of impact, and what emergent qualities happen as a result.
It’s ok to say “we don’t know why this works” and also “there’s no reason to expect anything more from this methodology”. But I wouldn’t dismiss further improvements as a forgone possibility.
Another way to think of this is feedback from humans will refine results. If enough people tell it that Toronto is not the capital of Canada it will start biasing toward Ottawa, for example. I have a feeling this is behind the search engine roll out.
ChatGPT doesn’t learn like that though, does it? I thought it was “static” with its training data.
I was speculating about how you can overcome hallucinations, etc., by supplying additional training data. Not specific to ChatGPT or even LLMs…
You can finetune LLMs using smaller datasets, or with RLHF (reinforcement learning from human feedback) wherein people can give ratings to responses and the model can be either “rewarded” or “penalized” based off of the ratings for a given output. This retrains the LLM to produce outputs that people prefer.
Active Learning Models. Though public exposure can eaily fuck it up, without adult supervision. With proper supervision though, there’s promise.
So it will always have the biases of the supervisors
Toronto is Canadian New York. It wants to be the capital and probably should be but it doesn’t speak enough French.
This is exactly it. And it’s funny you’re getting downvoted.
We don’t truly know the depth of ML yet and how these general models could potential change when a few vectors in the equation change, and that’s the big unknown with it. I agree with you here that Gates’ opinion is just that and isn’t particularly well informed. Especially in comparison to what some of the industry and ML experts are saying about how far we can go with the models, how they will evolve as we change parameters/vectors/dependencies and the impact of that evolution on potential applications. It’s just too early.
I kinda get why I’m getting downvoted, honestly. The ChatGPT fanboys definitely give off an “NFT-grindset” kind of vibe, and they can be loud and overzealous with their prognosticating. It feels cathartic to make fun of the thing they’ve adopted as a centerpiece of their personality
None of that changes what is objectively the very real and very unexpected improvement these models are displaying, and we’re still not sure what it is they’re doing behind the curtain. “Predicting the next most likely word” is simply not a sufficient explanation for how these models seem to correctly interpret intent and apply factual knowledge stored in its dataset in abstract ways.
People want to squabble over anthropomorphic word choices and debate ‘consiousness’, and fair enough, its an interesting question. But that doesn’t really come close to what’s really interesting about the models gaining functionality when by all accounts they should only be ‘guessing the next most likely word’.
I’m not really interested in debating people who are performatively unimpressed by these products, but it bothers me that those people continue rolling their eyes when significant advancements are made. Like sure, it’s not new that ML algorithms can decode keystrokes from an audio recording, but it’s a big deal when those models can be run on consumer grade hardware and not just a super computer run by a three letter agency.
I mean, that’s more-or-less what I said. We don’t know the theoretical limits of how good that text generation is when throwing more compute at it and adding parameters for the context window. Can it generate a whole book that is fairly convincing, write legal briefs off of the sum of human legal knowledge, etc.? Ultimately, the algorithm is the same, so like you said, the same problems persist, and the definition of “better” is wishy-washy.
It will obviously get even better, but you’ll never be able to rely on it. Sure, 99.9% of that generated legal document will look perfect, till you overlook one sentence where the AI hallucinated. There is no fact checking in there, that’s the issue.
Yeah and I think he may be scaling to like true AGI. Very possible LLMs just don’t become AGI, you need some extra juice we haven’t come up with yet, in addition to computational power no one can afford yet.
Except that scaling alone won’t lead to AGI. It may generate better, more convincing text, but the core algorithm is the same. That “special juice” is almost certainly going to come from algorithmic development rather than just throwing more compute at the problem.
See my reply to the person you replied to. I think you’re right that there will need to be more algorithmic development (like some awareness of its own confidence so that the network can say IDK instead of hallucinating its best guess). Fundamentally though, llm’s don’t have the same dimensions of awareness that a person does, and I think that that’s the main bottleneck of human-like understanding.
My hypothesis is that that “extra juice” is going to be some kind of body. More senses than text-input, and more ways to manipulate itself and the environment than text-output. Basically, right now llm’s can kind of understand things in terms of text descriptions, but will never be able to understand it the way a human can until it has all of the senses (and arguably physical capabilities) that a human does. Thought experiment: Presumably you “understand” your dog - can you describe your dog without sensory details, directly or indirectly? Behavior had to be observed somehow. Time is a sense too. EDIT: before someone says it, as for feelings I’m not really sure, I’m not a biology guy. But my guess is we sense our own hormones as well
First, they do have senses. For example, many LLMs can “see” images. Second, they’re actually pretty good at describing things. What they’re really bad at is analysis and logic, which is not related to senses at all.
I’m not so convinced that logic is completely unrelated to the senses. How did you learn to count, add, and subtract mentally? You used your fingers. I don’t know about you, but even though I don’t count my fingers anymore I still tend to “visualize” math operations. Would I be capable of that if I were born blind? Maybe I’d figure out how to do the same thing in a different dimension of awareness, but I have no doubt that being able to conceptualize visually helps my own logic. As for more complicated math, I can’t do that mentally either, I need a calculator and/or scratch paper. Maybe analogues to those can be implemented into the model? Maybe someone should just train a model on khan academy videos, and it’ll pick this stuff up emergently? I’m not saying that the ability to visualize is the only roadblock though, I’m sure that improvements could be made to the models themselves, but I bet that it’ll be key to human-like reasoning
That’s a good point.
The problem is that between gpt 3 and 4 there is massive increase in number of parameters, but not massive increase in its abilities
I’ll listen to his opinions more than some, but unfortunately this article doesn’t say anything interesting about why he has this opinion. I guess the author supposes we will simply regard him as an oracle on name recognition alone.
I hope so. Theyve already got scary implications for creative parts of the economy.
That said, we’re in the Cambrian explosion of the tech. As it plateaus, the next step will be enhanced tooling and convenience around it. Better inputs than just text, better, more applications in new spaces, etc.
bill is a wanker. dont be like bill.
Now now. He only hired assholes and monsters to execute immoral MS mob style tactics, while he played the great innocent altruist.
“GPT-4 should be enough for anyone.” -Bill Gates
Not a single comment yet stating how Gates is a great human being because of his foundation, and how all you haters should fuck the fuck off? sigh, let me the first one.
Just to make things extremely clear, the above comment has been sarcastic. He’s an awful person.
They’re upset he insulted their AI girlfriends.
You mean his tax haven?
I mean they’ve done some good things, but the capitalist system that gave him his wealth is the same one that causes poverty and his foundation isn’t working to change that.
But we have more areas to apply this to. I still can’t ask my PC to do some work, like Unistall OneDrive or change a setting in the OS. Send a message on Teams. Where is Jarvis?
Having Generative AI make API calls on your behalf is a work in progress across pretty much every industry. It’ll make complex tasks across multiple services a lot easier but it’s definitely going to cause weird unpredictable behavior too.
I wouldn’t trust it do everything yet, but it sure as hell would be useful to retrieve information. Wish I could just ask it “Hey, is the door locked?” and get an answer.
Maybe it could suggest actions, but I wouldn’t want to have it do anything without manual human confirmation, it’s too unreliable.
That is why you have to confirm the action before execting
On the one hand, I don’t really know enough about AI to comment. What I do remember is that, Bill Gates said the Internet was just a fad in the 90s. This comment caused myself and others problems promoting the Internet in workplaces because those in charge for some reason put some weight to his words. :p
I’ve been saying this for years!
Bill Gates views on AI are about as insightful as Gordon Ramsey’s on orbital mechanics.
I think he could be right about generative AI, but that’s not a serious problem given we’re moving beyond generative AI and into virtual intelligence territory.
Generative ai right now requires someone (or something) to initiate it with a prompt, but according to some of the latest research papers in OpenAI as well as the drama that happened recently surrounding the leadership, it appears that we’re moving beyond the ‘generative’ phase into the ‘virtual intelligence’ phase.
It’s not going to be ‘smart’ it will be knowledgeable (and accurate, hopefully). That is to say VI’s will be useful as data retrieval or organization but not necessarily data creation (although IIRC the way to get around this would be to develop a VI that specifically only works on creating ideas but we’d be moving into AGI territory and I don’t expect we’ll have serious contenders for AGI for another decade at least).
The rumours abound surrounding the OpenAI drama, the key one being the potential for accidentally developing AGI internally (I doubt this heavily). The more likely reason is that the board of directors had a financial stake in Nvidia and when they found out altman was working on chips specifically for AI that were faster, lower cost, and lower power consumption than current nvidia trash (by literally tens of thousands of dollars), they fired him to try and force the company onto their preferred track (and profit in the process, which IMO, kind of ironic that a non-profit board of directors has so many ‘closed door’ discussions with nvidia staff…)
This is just the thoughts of a comp-sci student with a focus on artificial intelligence systems.
If interested in further reading:
https://www.ibm.com/blog/understanding-the-different-types-of-artificial-intelligence/
https://digitalreality.ieee.org/publications/virtual-intelligence-vs-artificial-intelligence
Keep in mind that because it’s still early days in this field that a lot of terms haven’t reached an established consensus across academia yet, so you’ll notice variations in how each organization explains what “x” type of intelligence is.
Maybe, but I am sure the tools the AIs can use will improve making the AIs jobs easier and thus the AI more efficient. I hope he is right tbh.
Eww, as a long time Linux user I need to take a shower now. I feel dirty.
The next big steps coming right now are AI trained on generative data and agents that act more automatically (rather than waiting for a prompt, take an action like searching the web and act on that to better complete the goal for example), and better indexed data so generated data can be informed by and cite sources in the moment.
AI trained on generative data
This has already been shown to degrade the output very quickly.
I think the wall that generative AI is hitting is the lack of more training data. All the web has been scraped to get it to where it is today, more and more content on the web is itself generated by AI and therefore not only useless but harmful if used as training data.Orca 2 is an example of an opensource model that was built to better collect and build on synthetic data: https://www.microsoft.com/en-us/research/publication/orca-progressive-learning-from-complex-explanation-traces-of-gpt-4/
The case I think being made is that building LFMs on the Internet gets you closer to an average internet users level of our put, using reinforcement learning you can further curate the outputs, then finally using these techniques you can generate even tighter high quality models.
It’s interesting stuff for sure.
deleted by creator
