Grok3 is garnering a lot of attention.
As I write this, it is less than 24 hours since the release. Throughout the day, many tested it against the established benchmarks. So far, it appears that Grok3 is near the top in the major categories.
Of course, when it comes to the AI world, things can change quickly.
What is different this time is the fact this is the first time Grok, which is developed by xAI, reached the top of the charts. It was a latecomer to the game, something it battled since the first version went live.
The question now is whether anyone can catch Grok in the future? Is it destined to hold the top spot and hold it?
Grok3: Can It Be Caught?
Elon Musk has once again put a company at the forefront of an emerging industry. xAI only had its first model 17 months ago. xAI is not even 3 years old yet, compared to 9 years for competitor OpenAI.
So what allowed the gap to be closed?
Grok3 is proving one of Musk's theories? He felt that scaling laws would hold, something that he bet billions of dollars on. This was done via the purchasing of massive amounts of compute. It was believed that a larger system would lead to a more intelligent one.
For years, this was disputed. It appears that we are now dealing with it as a reality.
Compute is crucial. This, naturally, is coupled with data. Reports are that Grok3 used 10 times the amount of data as the second version.
This was processed through 100K H100. This is where xAI starts to separate itself.
The company was able to set up a supercluster of 100K in 122 days. This was undertaken after checking with companies who offered a timeline of years. It was a feat that Jensen Huang, the CEO of Nvidia, mentioned on a few occasions.
Another step forward was the connectinf in a way that had 100K clusters operating as one. This was something that many questioned was even possible, with 30K beling the perceived cap.
News is emerging the second set of 100K GPUs were added in 92 days. This means the team is accelerating their implementation.
How much of a lead does this provide xAI?
Racing Ahead
If xAI stopped here, the window could close quickly. However, this appears not to be the case.
During the presentation, it was mentioned the cluster would expand again. No details were given on if it would be another 100K set or more will be added. There was also no timeline provided.
In the past, Musk mentioned the potential of having a supercluster of 1 million. There is no reason to believe this is still not on the roadmap. If that is achieved, it will be 20 times the chips and 40 times the compute used to train Grok3.
We also can reason that Grok4 is already being pre-trained on the 200K cluster.
According to futurist Brian Wang, the super cluster race is favoring xAI.
Previous reports were that xAI order over $1 billion worth of B200s, Nvidia's latest GPU. Deliveries were slated to start in January. The same 90 day timeline would align with Wang's April 2025 install date.
This is part of $9 billion that xAI committed to spending.
Looking at the chart, we can see that Grok5 could be targeted for early next year, training on a 500K cluster. This means another 300K of H100 equivalents are needed.
Google could be in this game if they can get enough chips. They do not utilize Nvidia, opting for their own source. Here is where manufacturing scaling enters.
Either way, Huang knows that xAI will put the chips received online faster than anyone else. This means more orders coming in, something every company wants.