Last but not least BenchMate captures the result automatically before it is displayed by the benchmark and stores it in a secure location. The benchmark executable, its executed machine code and the application's logic remains untouched throughout the whole process. We also validate all files and platform parameters for the benchmark and its workloads automatically to avoid tampering of executables, libraries and resource files. Every run is measured with up to three independent timers allowing a maximum deviance of only 0.01% and less than 10 milliseconds. Benchmarks are guarded by a combination of a custom driver, two background services, the BenchMate launcher and client to protect against access with malicious intent as well as unknowingly skewed results.īenchMate monitors the benchmark’s process closely to enable precise time measurement provided by low-level access to the system’s timer hardware.

While automating the upload of results improves the ranking, it still fails to provide an automatically applied and more importantly equal result standard for all benchmarks.Ī well-considered minimum of security and timer reliability applicable to all benchmarks, though much more advanced than anything currently available in this industry. The implementation depends heavily on the knowledge and maintenance of the benchmark developer and therefore varies in security, reliability and in the end quality.

Due to undetectable time measurement bugs and easily abusable vulnerabilities of benchmarks, results are very hard to judge and validate simply because there is not enough data available for the task.Įven encrypted submission files provided by a few benchmarks (like our own GPUPI) are sadly flawed by design.

Well, you shouldn't.Įven today many benchmark results have to be entered manually, which is known to be error-prone, inconsistent, time consuming and hard to learn while still being far from trustworthy. You probably have taken the validity of these results for granted. If you have seen a PC benchmark world record, advertized performance numbers or casual benchmark results on the internet, chances are you just saw a single number, a screenshot or at best an uploaded result to a ranking platform like HWBOT, the world’s biggest result platform. Benchmarking is in a horrible spot right now.

Y cruncher download