AMD Rips ‘Biased’ And ‘Unreliable' Intel-Optimized SYSmark Benchmark
AMD is making a stink about SYSmark, a popular benchmarking program that's been around for many years, and one the chip designer says is not reliable. Rather than provide meaningful results and information, AMD claims SYSmark unfairly favors Intel products and puts too much emphasis on strict CPU performance above all else.
John Hampton, director of AMD's client computing products, explained in a four-and-a-half-minute video posted to YouTube why SYSmark itself is an unreliable metric of performance. He even brought up the "recent debacle" involving Volkswagen as proof that "information provided by even the most established organizations can be misleading." It wasn't an outright claim that Intel and SYSmark developer BAPCo are in cahoots in a cheating scandal, but certainly AMD feels there's some serious bias taking place.
AMD's beef with SYSmark is that the results aren't indicative of a system's real-world performance. To prove its point, AMD ran the latest version of SYSmark on a pair of PCs, one powered by an Intel Core i5 processor and the other by an AMD FX chip, and compared the performance delta with that of what Futuremark's PCMark 8 showed.
The tests took place in AMD's lab with Tony Salinas, an engineering manager and technical marketing lead at AMD, explaining the results. In SYSmark, the Core i5 system scored 987. A "comparable AMD platform" running an FX processor scored 659.
"That's a delta of 50 percent. Quite astonishing and not realistic in what real life performance is like," Salinas said.
In PCMark 8's Work Accelerated test, the performance delta narrowed to 7 percent with the same Core i5 system scoring 4,199 and the AMD setup posting 3,908, which was "quite different than the story SYSmark" provided.
Why are the benchmarks so far apart? Salinas says SYSmark's focus on the CPU is so "excessive" that it's really only evaluating the processor, not the system as a whole. In comparison, PCMark 8 probes not only the CPU, but graphics and subsystems as well.
In an attempt to drive the point home, AMD ran a set of custom scripts it developed based on Microsoft Office and timed how long it took each system to complete them. The Intel system took 61 seconds to finish the benchmark versus 64 seconds for the AMD platform, a difference of about 6-7 percent and in line with what PCMark 8 indicated.
As far as AMD is concerned, SYSmark is an anomaly and doesn't use realistic, everyday workloads, nor is it transparent. Those concerns prompted AMD to resign from the BAPCo consortium back in 2011. AMD also encourages users to read what the FTC had to say (PDF) on the matter, which if you don't want to click is that SYSmark's tests "may have been optimized for performance only on Intel microprocessors."
At the time that AMD turned its back on BAPCo, the company released a statement explaining that each member gets one vote on any proposals made by member companies, and that "AMD voited in support of over 80 percent of the SYSmark 2012 development milestones, and were supported by BAPCo in 100 percent of the SYSmark 2012 proposals they put forward to the consortium."
The latest build of BAPCO's benchmark is SYSmark 2014 version 1.5.
John Hampton, director of AMD's client computing products, explained in a four-and-a-half-minute video posted to YouTube why SYSmark itself is an unreliable metric of performance. He even brought up the "recent debacle" involving Volkswagen as proof that "information provided by even the most established organizations can be misleading." It wasn't an outright claim that Intel and SYSmark developer BAPCo are in cahoots in a cheating scandal, but certainly AMD feels there's some serious bias taking place.
AMD's beef with SYSmark is that the results aren't indicative of a system's real-world performance. To prove its point, AMD ran the latest version of SYSmark on a pair of PCs, one powered by an Intel Core i5 processor and the other by an AMD FX chip, and compared the performance delta with that of what Futuremark's PCMark 8 showed.
The tests took place in AMD's lab with Tony Salinas, an engineering manager and technical marketing lead at AMD, explaining the results. In SYSmark, the Core i5 system scored 987. A "comparable AMD platform" running an FX processor scored 659.
"That's a delta of 50 percent. Quite astonishing and not realistic in what real life performance is like," Salinas said.
In PCMark 8's Work Accelerated test, the performance delta narrowed to 7 percent with the same Core i5 system scoring 4,199 and the AMD setup posting 3,908, which was "quite different than the story SYSmark" provided.
Why are the benchmarks so far apart? Salinas says SYSmark's focus on the CPU is so "excessive" that it's really only evaluating the processor, not the system as a whole. In comparison, PCMark 8 probes not only the CPU, but graphics and subsystems as well.
In an attempt to drive the point home, AMD ran a set of custom scripts it developed based on Microsoft Office and timed how long it took each system to complete them. The Intel system took 61 seconds to finish the benchmark versus 64 seconds for the AMD platform, a difference of about 6-7 percent and in line with what PCMark 8 indicated.
As far as AMD is concerned, SYSmark is an anomaly and doesn't use realistic, everyday workloads, nor is it transparent. Those concerns prompted AMD to resign from the BAPCo consortium back in 2011. AMD also encourages users to read what the FTC had to say (PDF) on the matter, which if you don't want to click is that SYSmark's tests "may have been optimized for performance only on Intel microprocessors."
At the time that AMD turned its back on BAPCo, the company released a statement explaining that each member gets one vote on any proposals made by member companies, and that "AMD voited in support of over 80 percent of the SYSmark 2012 development milestones, and were supported by BAPCo in 100 percent of the SYSmark 2012 proposals they put forward to the consortium."
The latest build of BAPCO's benchmark is SYSmark 2014 version 1.5.
