Luck notwithstanding, people are now buying the chips as fast as Nvidia can make them. “GPUs at this point are considerably harder to get than drugs,” Elon Musk told a conference in May.
While rival companies make GPUs, one technology investor describes Nvidia as “the only game in town”. And for those that want the best, only the H100 will do.
The state of the art model that powers ChatGPT, for example, was developed using an estimated 25,000 A100s, the chip’s predecessor.
The new version is an order of magnitude more powerful: accomplishing in 20 hours what would take the A100 a week.
For all their ascribed “intelligence”, the best AI systems are really a product of the best training, and that is really a matter of having the best computer.
The first version of the model behind ChatGPT had around 100 million variables known as parameters; the latest is believed to have trillions.
No surprise, then, that tracking down the best chips has become an act of desperation. “There’s one thing that can provide competition to H100 GPUs: Taylor Swift concert tickets,” Aravind Srinivas, an AI entrepreneur, tweeted.
One online commentator refers to it as the “Opec of GPUs”.
Silicon Valley bosses trade stories of their efforts to obtain the fabled chips: some tap up their well connected venture capital investors for contacts, others beg friends at universities for a chance to plug in to their research labs.
There is only one guaranteed way to obtain a steady supply, however: hundreds of millions of dollars. That means there are now broadly two camps of customers that can buy chips in the quantities that matter: tech giants, and nation states.
The global race for AI supremacy
Within a few years, AI has moved from a promising technology to a critical national priority.
In 2017, China outlined a “new generation AI plan” vowing to make the country the “primary” centre for innovation by 2030. According to the US-based Center for Security and Emerging Technology, Beijing is seeking to push towards “artificial general intelligence”, a system capable of matching human intelligence.
The plan has been met with a volley of US sanctions designed to cripple Chinese efforts in AI.
Last year the White House barred Nvidia from selling its H100 and A100 chips to China without a licence. The company has rushed out new versions of the chips to evade the controls, and the Biden administration is now expected to respond with a new set of rules.
Saudi Arabia and the United Arab Emirates are both investing heavily in chips, developing their own “large language models” similar to ChatGPT.
According to the Financial Times, the Saudi software, being developed at the oil-rich state’s Kaust research institution, is largely being created by Chinese researchers whose education bars them from working in America.
In the US, the Biden administration is spending tens of billions to encourage microchip companies to set up factories in America, a plan that has been described as “one of the biggest government interventions since the Second World War”.
Nvidia may be an American company, but its highest-powered chips are exclusively made in Taiwan, making the entire AI industry dangerously exposed to the threat of Chinese invasion.