Click here to see detected mean changes (possibly regressions) instead of response times.
Currently, the automated benchmarking system is not maintained. These pages thus show performance numbers for 4 years of history of the Mono project (August 2004 to October 2008).
The purpose of this project is detection of performance regressions in daily development versions of MONO. Selected benchmarks are compiled and run for each daily version. The daily versions are compiled MONO daily packages. Selected plots from the benchmarks follow. Description of the system used for benchmarking is here. The benchmarks are run on a dedicated machine, with disabled network interface and with no system services running.
The plots are updated daily. If some mono version is not included in
some of the plots, it can be due to a compilation problem or a bug in a
specific MONO version, see the
status page
for more info. You may want to check
diffs of mono sources to track down causes of
performance changes suggested by the plots.
The following table shows a brief summary of performance changes in
the last 7 Mono versions for all supported benchmarks and platforms. The
red background (positive values) signifies performance regressions, the
green background (negative values) signifies performance improvements. The
equal sign means that there was no significant performance change in the
specific version. The N/A values are used whenever the specific Mono version
failed on a given platform and/or benchmark. Large absolute values (i.e. 5
and more) of the changes mean that the change is very unlikely to be a false
alarm. Please note that the SciMark (not "FFT Scimark") benchmark does not
have proper warmup, and thus performance changes detected are not to be
taken as reliable as changes detected by other benchmarks. More detailed
info on the benchmarks and performance changes is provided
below the table.
| 2008-10-07 | 2008-10-06 | 2008-10-05 | 2008-10-04 | 2008-10-03 | 2008-10-02 | 2008-10-01 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| P4 | IA64 | P4 | IA64 | P4 | IA64 | P4 | IA64 | P4 | IA64 | P4 | IA64 | P4 | IA64 | |
| HTTP Ping | | | | | | | | | | | | | | |
| HTTP Ping (all JIT optimizations) | | | | | | | | | | | | | | |
| TCP Ping | | | | | | | | | | | | | | |
| TCP Ping (all JIT optimizations) | | | | | | | | | | | | | | |
| FFT SciMark | | | | | | | | | | | | | | |
| FFT SciMark (all JIT optimizations) | | | | | | | | | | | | | | |
| Rijndael | | | | | | | | | | | | | | |
| Rijndael (all JIT optimizations) | | | | | | | | | | | | | | |
| FFT SciMark (no allocation) | | | | | | | | | | | | | | |
| FFT SciMark (no allocation, all JIT optimizations) | | | | | | | | | | | | | | |
These results are for Pentium 4/Linux platform.
Detailed platform specification is here. Click here to see results
for Itanium/Linux platform.
These results are with default options. Click here to get results with all JIT optimizations (-O=all).
Click here to see detected mean changes (possibly regressions) instead of response times.
| HTTP Ping Benchmark (p4) |
The following plot shows HTTP ping response times for consecutive versions of
MONO. Description of the benchmark and more plots are available here.
Click here to see detected mean changes (possibly regressions) instead of response times.
| TCP Ping Benchmark (p4) |
The following plot shows TCP ping response times for consecutive versions of
MONO. It differs from the HTTP Ping
benchmark only in that uses TCP channels instead of HTTP channels.
Description of the benchmark and more plots are available here.
Click here to see detected mean changes (possibly regressions) instead of response times.
| SciMark Benchmark (p4) |
The following plot shows SciMark normalized MFLOPs for consecutive versions
of MONO. Description of the benchmark and more plots are available
here
| FFT-SciMark Benchmark (p4) |
The following plot shows FFT-Scimark computation times for consecutive versions of
MONO.
Description of the benchmark and more plots are available here.
Click here to see detected mean changes (possibly regressions) instead of response times.
| FFT-SciMark Benchmark (no allocation) (p4) |
The following plot shows changes in mean FFT-Scimark computation
times for consecutive versions of MONO. The benchmark differs from the
previous one in that it uses pre-allocated memory.
Description of the benchmark and more plots are available here.
Click here to see detected mean changes (possibly regressions) instead of response times.
| Rijndael Benchmark (p4) |
The following plot shows Rijndael encryption/decryption times for consecutive versions of
MONO.
Description of the benchmark and more plots are available here.
Click here to see detected mean changes (possibly regressions) instead of response times.
| Platform Description (p4) |
All mono versions and benchmarks are compiled and run on DELL Precision 340
with Fedora Core 2.
The hardware configuration is Pentium 4 (Northwood) CPU, 2.2GHz, 512KB L2
cache, 512MB RAM.
The software configuration is 2.6.8-1.521 FC2 kernel compiled for i686, gcc
3.3.3, glibc 2.3.3.
| Benchmarking Software Download (p4) |
You can download our software we use for the measurement and results evaluation via this link. The software was designed for benchmarking of Mono on Linux only, if you want a more generic yet less mature regression benchmarking software, you may want to have a look at the BEEN project.
|
|
|
Tomas Kalibera
Distributed Systems Research Group |