Linux Benchmarking HOWTO

1. Introduction

"What we cannot speak about we must pass over in silence."

Ludwig Wittgenstein (1889-1951), Austrian philosopher

Benchmarking means measuring the speed with which a computer system will execute a computing task, in a way that will allow comparison between different hard/software combinations. It does not involve user-friendliness, aesthetic or ergonomic considerations or any other subjective judgment.

Benchmarking is a tedious, repetitive task, and takes attention to details. Very often the results are not what one would expect, and subject to interpretation (which actually may be the most important part of a benchmarking procedure).

Finally, benchmarking deals with facts and figures, not opinion or approximation.

1.1 Why is benchmarking so important ?

Apart from the reasons pointed out in the BogoMips Mini-HOWTO (section 7, paragraph 2), one occasionally is confronted with a limited budget and/or minimum performance requirements while putting together a Linux box. In other words, when confronted with the following questions:

• How do I maximize performance within a given budget ?
• How do I minimize costs for a required minimum performance level ?
• How do I obtain the best performance/cost ratio (within a given budget or given performance requirements)?

one will have to examine, compare and/or produce benchmarks. Minimizing costs with no performance requirements usually involves putting together a machine with leftover parts (that old 386SX-16 box lying around in the garage will do fine) and does not require benchmarks, and maximizing performance with no cost ceiling is not a realistic situation (unless one is willing to put a Cray box in his/her living room - the leather-covered power supplies around it look nice, don't they ?).

Benchmarking per se is senseless, a waste of time and money; it is only meaningful as part of a decision process, i.e. if one has to make a choice between two or more alternatives.

Usually another parameter in the decision process is cost, but it could be availability, service, reliability, strategic considerations or any other rational, measurable characteristic of a computer system. When comparing the performance of different Linux kernel versions, for example, stability is almost always more important than speed.

1.2 Invalid benchmarking considerations

Very often read in newsgroups and mailing lists, unfortunately:

1. Reputation of manufacturer (unmeasurable and meaningless).
2. Market share of manufacturer (meaningless and irrelevant).
3. Irrational parameters (for example, superstition or prejudice: would you buy a processor labeled 131313ZAP and painted pink ?)
4. Perceived value (meaningless, unmeasurable and irrational).
5. Amount of marketing hype: this one is the worst, I guess. I personally am fed up with the "XXX inside" or "kkkkkws compatible" logos (now the "aaaaaPowered" has joined the band - what next ?). IMHO, the billions of dollars spent on such campaigns would be better used by research teams on the design of new, faster, (cheaper :-) bug-free processors. No amount of marketing hype will remove a floating-point bug in the FPU of the brand-new processor you just plugged in your motherboard, but an exchange against a redesigned processor will.
6. "You get what you pay for" opinions are just that: opinions. Give me the facts, please.