Nvidia announced that its AI inference platform consistently scored higher than the competition and set performance records in multiple categories in the latest MLPerf benchmark testing program. Although, it also was clear that some companies in the sector didn't prioritize MLPerf participation to the same degree as Nvidia.
The results were achieved in the MLPerf 1.0 program, and Nvidia’s results in many cases showed vast performance improvements in the six months since the previous MLPerf testing, according to Paresh Kharya, senior director of product management and marketing at Nvidia.
MLPerf is an industry benchmark program started in 2018 by ML Commons and the MLPerf Consortium to measure AI performance across seven different applications--image classification, medical imaging, recommendation systems, speech recognition, natural language processing and both high-resolution and low-resolution image detection. The tests involve AI inference in data center environments in both offline and server query-driven scenarios and in edge servers and devices in offline, single-stream and multi-stream scenarios (the latter being cases in which the device processors may be processing streams from multiple sensors.)
Along with Nvidia, Intel, AMD, Xilinx and Qualcomm each also participated in some of the tests. During a press and analyst briefing to announce the MLPerf results, Kharya presented data showing that Nvidia’s A100 high-performance data center processing product beat submissions from those four companies in the image classification category in both offline and server-driven benchmark scenarios, which was the only category in which submissions from all five companies participated.
The A100 was the highest performer across all categories, and Nvidia’s recently-unveiled A10 and A30 low-power GPUs for more mainstream server applications also performed well in the same tests, Kharya said. Nvidia's only defeats that Kharya acknowledged came in the new, separate MLPerf benchmarking for energy efficiency, in which it was narrowly bested by Qualcomm's AI 100 in two of six energy efficiency test categories on the basis of performance per watt.
Among other MLPerf notes, Nvidia’s Triton Inference Server was used in some of the MLPerf tests, and Kharya said the company also broke new ground by showcasing an A100 using the Nvidia Ampere architecture’s Multi-Instance GPU (MIG) capability running all seven MLPerf Offline tests on a single GPU at the same time. The configuration showed nearly identical performance--98%--compared with a single MIG instance running alone. Kharya explained the significance: “Customers now have a choice if they have a lot of compute work because there are a lot of concurrent users that are pinging the server constantly for queries. They can run it on the entire GPU… or choose to run it on a single MIG.”
Where's the competition?
While Nvidia treated MLPerf as a technology showcase, Kharya acknowledged that Nvidia was the only company to submit results for every test in the data center and edge categories. He was not the only participant in the briefing to highlight the discrepancy. During a Q&A session, Karl Freund, founder and principal analyst at Cambrian-AI Research, likened the MLPerf tests to going to the movies every few months, but the movie being shown never changes.
“Every time MLPerf comes out, Nvidia dominates, but it dominates no one,” Freund said. “Nobody else is showing up.”
In fact, in addition to Nvidia’s own MLPerf submissions, Nvidia partners Alibaba Cloud, DellEMC, Fujitsu, Gigabyte, HPE, Inspur, Lenovo and Supermicro submitted a total of over 360 results using Nvidia GPUs, but other processor vendors both big and small were notably absent from many tests.
Regarding the lack of competition, Kharya suggested it reflects the relative immaturity of the AI and machine learning markets. “AI is still in its infancy. It’s still in Chapter One of a really evolving story. MLPref has provided a great yardstick for measuring product performance. Nvidia’s ecosystem shows up because our customers want to see the results.”
After the briefing, Freund concurred with Kharya’s view in comments emailed to Fierce Electronics, saying it may be too early in the development of AI and machine learning to expect more MLPerf participation, and that many companies are spending as much time as possible perfecting their products and engaging with their prospective customers.
“I think participation will increase as these companies mature,” Freund said. “Some companies will fail to deliver good results, and these companies will likely fade away. But most are simply not yet ready.”
He added that MLPerf has strong support from vendors across the industry (MLCommons.org lists more than 40 companies as members.) "These benchmarks are representative of real workloads and will help buyers sort through all the claims, which today can be misleading,” Freund said.
RELATED: Nvidia makes strides in training conversational AI