SAN FRANCISCO, April 3, 2025 – MLCommons, an artificial intelligence (AI) industry group, has introduced two new benchmarks designed to evaluate the performance of cutting-edge hardware and software in running AI applications. This development arrives as chip companies increasingly focus on producing hardware capable of efficiently supporting the code behind popular AI tools, a shift spurred by the widespread adoption of platforms like OpenAI's ChatGPT.

The new benchmarks, updates to the MLPerf suite, are designed to assess the speed at which systems can handle the growing demands of AI applications such as chatbots and search engines. One benchmark leverages Meta's Llama 3.1, a 405-billion-parameter AI model, to test general question answering capabilities, as well as math and code generation. This test format emphasizes a system's capacity to process extensive queries and synthesize information from diverse sources.

Nvidia and system builders like Dell Technologies have submitted their chips for evaluation using the new benchmarks. Notably, Nvidia's latest AI server generation, Grace Blackwell, which incorporates 72 Nvidia GPUs, demonstrated performance improvements of 2.8 to 3.4 times compared to the previous generation. These results were observed even when comparing configurations with only eight GPUs in the newer server, ensuring a direct comparison. Nvidia has focused on enhancing the interconnectivity of chips within its servers, a crucial factor for AI tasks that require chatbots to operate across multiple chips simultaneously.

The second benchmark is also based on an open-source AI model developed by Meta, and its design aims to replicate the performance expectations of consumer AI applications like ChatGPT. The primary objective is to minimize response times, striving for near-instantaneous results.

In related news, President Trump's recent imposition of import tariffs on major manufacturing countries, effective April 9, is expected to significantly impact Apple's global supply chain. These reciprocal tariffs may hinder Apple's efforts to diversify its supply chain away from China. Furthermore, countries targeted by these tariffs may retaliate against U.S. service sectors, including finance, travel, engineering, and medical services, potentially impacting the United States' trade surplus in services.