An acquisition worth 4 trillion NVIDIA Corporation (NVDA.US)

After the release of the financial report for the second quarter of this year, the spotlight was mainly on whether the revenue of this chip manufacturer could continue to prove the rationality of its rapidly growing market value. But behind all the headlines, there is a particular business segment that is particularly eye-catching, and that is the networking business. Analysts believe that this will be the low-key engine driving the company's transformation into a $4 trillion market cap giant. Related data shows that the contribution of the business labeled as "networking" to the overall revenue of NVIDIA Corporation (NVDA.US) may be far higher than 16.1%. Revenue skyrocketed by 46% compared to the previous period, almost doubling year-on-year, reaching $7.25 billion in just the second quarter. In other words, in just the previous quarter, the research and development center established by the acquisition of Mellanox for NVIDIA Corporation created revenue that exceeded the acquisition cost. This has led to the annual operational turnover of this department reaching $25-30 billion, an extraordinary figure for a department that was once considered a supporting role to NVIDIA Corporation's flagship graphics processor. Behind this achievement is the credit to the $6.9 billion acquisition of Mellanox they made that year. NVIDIA Corporation's Unsung Heroes In recent years, the challenge to NVIDIA Corporation has not only been seen from a computational standpoint, but software and networking have also been frequently mentioned. For example, the recent UAlink was established as an organizational alliance to break through the barriers of NVIDIA Corporation. The reason behind this is that a single chip or a single rack is not enough to meet the increasing demand for AI computational power, pressing for Scale Up and Scale Out approaches. NVIDIA Corporation stated that due to physical constraints such as energy supply and chip density, data centers today are reaching the limits of what a single facility can provide. The new platform, Spectrum-XGS, has addressed obstacles such as long latency, issues that have hindered the operation of independent facilities as a unified system until now. NVIDIA Corporation's CEO, Jensen Huang, also emphasized in a previous earnings call, "We have Spectrum-XGS, which has a scale of gigabits, connecting multiple data centers and AI factories into a super factory, a huge system. This is the reason why NVIDIA Corporation has invested so much effort in the networking field. As we mentioned earlier, Spectrum-X is now a quite considerable business, and it was only established about 1.5 years ago. Therefore, Spectrum-X is a home run." Earlier, a breakthrough in technology made by the Israeli subsidiary established by NVIDIA Corporation through the acquisition of Mellanox allowed geographically distant data centers to operate as if they were in one place, effectively creating "AI factories" and significantly increasing the maximum computing capacity available in the industry. The company stated in a press release, "With advanced distance-congestion control, precise delay management, and end-to-end telemetry technology, Spectrum-XGS Ethernet almost doubles the performance of NVIDIA's Collective Communication Library (CCL), accelerating multi-GPU and multi-node communication, providing predictable performance in geographically distributed AI clusters. As a result, multiple data centers can operate as one AI super factory and have been fully optimized for long-distance connectivity." As Jensen Huang said, "This is why NVIDIA Corporation acquired Mellanox 5.5 years ago." Mellanox, founded by Eyal Waldman in 1999, is a pioneer in InfiniBand interconnect technology. At the time of the acquisition by NVIDIA Corporation, this technology and its high-speed Ethernet products were already applied in more than half of the world's fastest supercomputers and many leading hyperscale data centers. Mellanox went public in 2007 and reached its first billion-dollar annual sales in 2018. In 2018, the company's GAAP net income was $134.3 million, also setting a record high, maintaining profitability for 10 years out of the 13 years before the acquisition, and consistently maintaining positive free cash flow since 2005. Between Mellanox and NVIDIA, there has been a long history of collaboration and co-innovation. NVIDIA opened a design center in Israel as early as 2016 and established an artificial intelligence research center in 2018. The company had previously promised to "continue investing in Israel's local top talent, as Israel is one of the most important technology centers globally." Eyal Waldman previously stated in a podcast, "I believe that the synergy between processors (the brain) and network connections will elevate NVIDIA Corporation from a $93 billion company to the $4 trillion giant it is today." He further pointed out that without Mellanox's InfiniBand, there would be no ChatGPT: "OpenAI has always purchased the most advanced products from us. If this connection did not exist, they would not achieve the data processing speed needed for artificial intelligence." Eyal Waldman mentioned. "This is the most important acquisition in the industry's history," Eyal Waldman emphasized. Unprecedented Importance of Networking Connections NVIDIA's Senior Vice President of Networking, Gilad Shainer, recalled in an interview with HPCwire that Mellanox was not building network components at the time. The company primarily built end-to-end comprehensive infrastructure focused on InfiniBand, equipped with network cards and switches, and the connections between them, as well as all the software based on it, making it a complete platform. "It is a comprehensive infrastructure designed for distributed computing applications. Therefore, it has been widely used in the HPC and scientific computing fields. All large cluster simulations use InfiniBand, as it is designed for decomposable computing and has extremely low latency. InfiniBand ensures that all nodes have effective bandwidth. Jitter is one factor that everyone wants to minimize." Gilad Shainer added. As he described, for HPC, this is a great technology, and when AI emerged, it became another case of distributed computing. For example, you may think that the sensitivity to latency is higher or lower because there are differences between AI workloads and scientific computing workloads. Scientific computing workloads may have a higher sensitivity to latency than early AI training; However, the sensitivity diminished to some extent at that time. "Nanosecond-scale latency is not as important for training, but a significant effective bandwidth is still required." Gilad Shainer pointed out. He stated that now we view inference as a major component of artificial intelligence. Inference relies on latency, as you need low latency. Therefore, artificial intelligence and high-performance computing (HPC) essentially have the same requirements. This is where infrastructure becomes more critical. Gilad Shainer mentioned that when comparing HPC and AI, an interesting phenomenon is observed in high-performance computing; computational power improves generation after generation. However, the scale of the data centers remains the same. Typically, data centers have thousands of nodes, and you can obtain telemetry data from each node, but the scale remains constant. Moving to artificial intelligence, the scale goes even higher. This is not just about the computational power of each server, but also the computational power of each new GPU, significantly increasing the scale of infrastructure. A few years ago, people were discussing 16,000 GPUs, even 30,000 GPUs. This is similar to the comparison with high-performance computing (HPC), where there are large infrastructures. Nowadays, a solution with 16,000 GPUs has been put on a back burner. Large infrastructures typically contain several hundred thousand GPUs now, with discussions among cloud providers talking about moving to millions of GPUs in a few years. This is not just a computer issue, but also an infrastructure scale issue. To achieve this scale, appropriate networking and scalable infrastructure is needed. Data centers are now the standard for measuring computational power. It is not just a box; it is a complete data center. "The data center is the network. The network defines how GPUs work together as a unified computing element. Otherwise, it will be just a cluster of GPU servers, which is why NVIDIA acquired Mellanox. This is where infrastructures are becoming increasingly important." Gilad Shainer said. In the eyes of NVIDIA Corporation, they are maintaining a rhythm of launching new data centers every year: introducing new GPUs, new computing engines, new switches, and new infrastructures every year. New data centers are put into operation each year to provide more powerful functionalities for artificial intelligence applications, whether that is training or inference. These new systems are spawning a plethora of artificial intelligence frameworks and applications worldwide. CPO: A Trend As everyone says, now, infrastructure consists of multiple domains required by data centers. In addition to scale-out (connecting servers), there is also a need to build or expand GPUs, that is, to combine GPUs and form larger virtual GPUs. To achieve this larger virtual GPU, significant bandwidth is needed between each GPU. If you want to make it appear as a whole, that is where NVlink comes into play. This feature is in the realm of system networking scale-up. NVlink requires supporting vast bandwidth, nine to ten times the scale-out. It needs very low latency. Hence, the Mellanox team introduces Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) into NVlink to achieve reduction, making the rack a unit, and they are trying to install more and more GPUs in that rack. In the future, NVIDIA Corporation plans to deploy 576 GPUs in a single rack. This entails a massive computational load, requiring scaling up the infrastructure within the rack. The company is also striving to keep it within the rack to maximize copper cabling utilization. From NVIDIA Corporation's perspective, once you have the huge bandwidth needed to transfer between components, you need to build it in the most cost-effective way, and copper cabling is the most efficient way to connect. But not stopping there, as now you need to connect these racks together, referring to making tens of thousands of GPUs work as one unit or making 200,000 GPUs work as one unit. Some customers may want 500,000 or even 1 million GPUs. Now, because of the long distances, we need to build a fiber-optic lateral expansion infrastructure, which must have the same characteristics as the OFED layer, including effective bandwidth and determinism. In the view of NVIDIA Corporation, InfiniBand is still considered the golden standard for lateral expansion infrastructure. Anything that you intend to build that is not based on InfiniBand can be compared to InfiniBand because InfiniBand is the golden standard for performance. To Gilad Shainer, scaling up systems is one aspect of artificial intelligence. Every year, the scale of data centers is growing significantly. This means greater inter-rack bandwidth and greater computation in the lines. Hence, there is more bandwidth needed in the lines. Gilad Shainer also pointed out that now, more fiber connections need to be deployed, and this suddenly introduces a power budget. "In artificial intelligence data centers, the limiting factor is not space or budget but how much power you can bring in," he mentioned. As Gilad Shainer stated, fiber connections between racks consume a lot of power. This results in a reduction in the number of GPUs that can fit within a rack. Consequently, fiber networks are beginning to consume close to 10% of the computational power, a significant number. So, in this situation, one factor to consider is whether there is a way to reduce the power consumption of fiber networks. This is not just because as data center scale increases, more components need to be built GPUs need to be installed, network cards need to be installed, cables need to be laid, transceivers and switches need to be set up, and all the necessary configurations need to be done, with the component growing the fastest being the number of optical transceivers. Because each GPU has about 6 optical transceivers. If I have 100,000 GPUs, then I need 600,000 transceivers. As you know, these transceivers are sensitive to dust and may need to be replaced by administrators if they fail. This situation could lead to an increase in the replacement of these components in data centers, because now there are more components. Therefore, in NVIDIA Corporation's view, the next significant move in data center infrastructure is to improve and elevate the fiber connections, perhaps integrating the fiber connections currently used as external transceivers into switches to bring them to a new level. "If I put them in a package, I dont need to transmit electrical signals through the switch. This means I can reduce power, drive the optical signals through the switch with less power. In this case, I could lower power consumption by almost four times. Now, in the same network, I could actually accommodate three times more GPUs." Thus, NVIDIA Corporation is pushing to integrate silicon photonics engines or optical engines into switches, eliminating the need for those external transceivers. As Gilad Shainer mentioned, the co-packaged optics (CPO) is not a new concept. Some have attempted to do this in the market. Today, various devices can be seen, and some switch systems are trying to adopt CPO, but they have not achieved full-scale production and reached high yield successfully yet for cost-effective scalability. There are many reasons behind this. One of the reasons is that this technology has not been validated, resulting in low yield. Earlier optical engines were manufactured using the technology for building large optical engines. If I have a large radix switch, due to its size, I cannot fit all these optical engines onto the same switch. A new packaging technology or even new laser technology may be needed. The achievements made are closely related to NVIDIA Corporation's acquisition of Mellanox back then. In Conclusion Eyal Waldman described the negotiations for the sale of Mellanox as a "war" between Intel Corporation, NVIDIA Corporation, and other companies in a podcast interview. "Ultimately, the connection with Jensen Huang (CEO of NVIDIA Corporation) was a perfect fit from the start." "We knew that was the direction from the beginning. In 2019, Intel Corporation's market cap far surpassed that of NVIDIA Corporation, but just a year later, NVIDIA Corporation overtook it. Since then, due to the right bet on artificial intelligence, its stock price has been soaring," Eyal Waldman emphasized. Following the acquisition of Mellanox, NVIDIA Corporation established a research and development team in Israel, second only to the United States. Reports show that this chip giant has over 5,000 employees in seven research and development centers in Israel. The company is also developing CPUs for data centers, Siasun Robot&Automation, SoCs for cars, and algorithms for autonomous driving locally. In this light, this was an unprecedentedly crucial transaction for NVIDIA Corporation. This article was originally published on the "Observations on the Semiconductor Industry" WeChat official account, edited by GMTEight: Song Zhiying.