Distributed Systems Research Group, Charles University, Prague

PEOPLE	PROJECTS	PUBLICATIONS	LINKS	TEACHING	CONTACT
F. Plasil	Components and services	By year	Conferences	Seminar	Address & phones
P. Tuma	Behavior specification	By project	Journals	C# Language & .NET Framework	Map
T. Bures	Performance evaluation	By author	Related work	Commercial Workshops	Home
T. Kalibera	Middleware	By category	Internal matters	Java
L. Bulej	Summer jobs		Ph.D. studies	Object & Component Systems
P. Hnetynka			Task suggestions	Operating Systems
V. Holub			Other links	Principles of Computers & Operating Systems
J. Kofron				Program Analysis & Code Verification
P. Parizek				Software Development & Modeling Tools
Ph.D. students
					Visits / Hits: 4064723 / 4888238 (since 1999)

CORBA Benchmarking

The project continues as a part of Middleware Benchmarking Project.

User benchmarking suite

User benchmark suite (C++) includes benchmark sources, binaries and a database of results. Java version of the user suite is currently being prepared.

Vendor benchmarking suite

We provide the debug results from vendor benchmarking suite. Currently available results are:

Project history

2
0
0
1

In May 2001, the "Open CORBA Benchmark Suite" was released. Currently, it consists of full sources prepared for MICO 2.3.x, omniORB 3.0.x, omniORB 4.0.x, ORBacus 4.x, ORBacus/E 1.x, Orbix 2000 1.2.x, Orbix/E 2.0.x, TAO 1.1.x runnig on RedHat Linux, Solaris 7.0, Windows NT 4.0, Windows 2000. Benchmark results can be submitted to our web site and then searched based on various criteria. For details please see

http://dsrg.mff.cuni.cz/~bench/

2
0
0
0

In March 2000, we have done the benchmarking project for IONA Technologies, PLC, Ireland. The project report is not publicly available.

In June 2000, the "Automated CORBA Benchmark Suite" was released at http://dsrg.mff.cuni.cz/~bench/. It has included omniORB 3.0.0 B2, ORBacus 4.0.1 and TAO 1.1.3. It is deprecated now by a newer, completely rewritten version.

1
9
9
9

In June 1999, an extension of the CORBA Comparison Project was concluded. The extension covers three more ORBs, namely omniORB 2.7.1, ORBacus 3.1.2 and Orbix 3.0, and also slightly extends the set of applied benchmarks.

The final report containing the results of the measurements, structured as an appendix to the original project report
(release: August 16, 1999)
- PostScript (3.6MB)
- GZipped PS (1MB)
- PDF color (1MB), PDF b/w (0.3MB)

1
9
9
8

In February 1998, our group launched a CORBA Comparison Project for MLC Systeme GmbH. The purpose of the project is to devise a sensible set of criteria for comparing ORB implementations and to apply this criteria to compare three ORBs selected by MLC (omniORB, Orbix, VisiBroker).

The final report containing the results of the measurements
- PostScript (11MB)
- Zipped PS (4MB)
- PDF (500kB)
Throughput test results and sources.

The experience gained during this project lead us to sent a response to OMG's Benchmarking RFI. Our response is available here or is available as a technical report.

Related publications:

Refereed (journals/proceedings)
PDF PDF WWW Kalibera, T., Bulej, L., Tuma, P.: Automated Detection of Performance Regressions: The Mono Experience,
in proceedings of the 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2005), Atlanta, GA, USA, Copyright (C) IEEE, Piscataway, New Jersey, USA, ISBN 0-7695-2458-3, ISSN 1526-7539, pp. 183-190, Sep 2005

PDF WWW Kalibera, T., Bulej, L., Tuma, P.: Quality Assurance in Performance: Evaluating Mono Benchmark Results,
in proceedings of the Second International Workshop on Software Quality (SOQUA 2005), Erfurt, Germany, Copyright (C) Springer-Verlag, Berlin, LNCS3712, ISBN 3-540-29033-8, ISSN 0302-9743, pp. 271-288, Sep 2005

PDF PDF Kalibera, T., Bulej, L., Tuma, P.: Benchmark Precision and Random Initial State,
in Proceedings of the 2005 International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS 2005), Cherry Hill, NJ, USA, Copyright (C) SCS, San Diego, CA, USA, pp. 853-862, ISBN 1-56555-300-4, Jul 2005

PDF WWW Bulej, L., Kalibera, T., Tuma, P.: Repeated Results Analysis for Middleware Regression Benchmarking,
in Performance Evaluation: An International Journal, Performance Modeling and Evaluation of High-Performance Parallel and Distributed Systems, vol. 60, pg. 345-358, Elsevier B.V., ISSN 0166-5316, May 2005

PDF PDF WWW Kalibera, T., Bulej, L., Tuma, P.: Generic Environment for Full Automation of Benchmarking,
in proceedings of Net.ObjectDays 2004, First International Workshop on Software Quality (SOQUA 2004), Erfurt, Germany, Copyright (C) 2004 tranSIT GmbH Ilmenau, Germany, pp. 35-41, ISBN 3-9808628-3-6 ; also in Testing of Component-Based Systems and Software Quality, Proceedings of SOQUA 2004 and TECOS 2004, LNI 58, GI 2004, pp. 125-132, ISBN 3-88579-387-3, ISSN 1617-5468, Sep 2004

PDF PDF WWW Bulej, L., Kalibera, T., Tuma, P.: Regression Benchmarking with Simple Middleware Benchmarks,
in proceedings of the 2004 International Performance Computing and Communications Conference (IPCCC 2004), International Workshop on Middleware Performance, Phoenix, AZ, USA, Copyright (C) 2004 IEEE, Piscataway, New Jersey, USA, pp. 771-776, ISBN 0-7803-8396-6, ISSN 1097-2641, Apr 2004

PDF PDF Buble, A., Bulej, L., Tuma, P.: CORBA Benchmarking: A Course With Hidden Obstacles,
in proceedings of the 2003 Internation Parallel & Distributed Processing Symposium (IPDPS 2003), Workshop on Performance Modeling, Evaluation and Optimization of Parallel and Distributed Systems (PMEOPDS 2003), Nice, France, Copyright (C) 2003 IEEE, Piscataway, New Jersey, USA, ISBN 0-7695-1926-1, ISSN 1530-2075, pp. 279, CDROM DATA/W18_PMEO_11.PDF, Apr 2003

PDF Tuma, P., Buble, A.: Overview of the CORBA Performance,
Proceedings of the 2002 EurOpen.CZ Conference, Znojmo, Czech Republic, Sep 2002

PDF Tuma, P., Buble, A.: Open CORBA Benchmarking,
Proceedings of the 2001 International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS 2001), Orlando, FL, Copyright (C) SCS, San Diego, CA, USA, Jul 2001

Nonrefereed (proceedings/slides)
PDF PDF WWW Kalibera, T., Bulej, L., Tuma, P.: Mono Regression Benchmarking,
in proceedings of WDS'05, MatfyzPress, Charles University, Prague, Czech Republic, pp. 19-24, ISBN 80-86732-59-2, Sep 2005

PDF Kalibera, T.: Regression Benchmarking Framework Architecture,
Presented at ObjectWeb Benchmarking Workshop, Prague, Czech Republic, Jul 2004

PDF Bulej, L.: Writing System Probes on Various Platforms,
Presented at ObjectWeb Benchmarking Workshop, Prague, Czech Republic, Jul 2004

PDF PDF Kalibera, T.: Regression Benchmarking Environment,
in proceedings of WDS'04, MatfyzPress, Charles University, Prague, Czech Republic, pp. 174-178, ISBN 80-86732-32-0, Jun 2004

PDF DOC Bulej, L., Kalibera, T., Tuma, P.: Regression Benchmarking,
Presented at Department Meeting Kokorin04, Jun 2004

PDF Kalibera, T.: Middleware Benchmarking Project - Regression Benchmarking,
Presented at ObjectWeb Benchmarking Workshop, Grenoble, France, Apr 2004

PDF Tuma, P., Bulej, L.: Current Trends in Middleware Benchmarking,
ObjectWeb Architecture Meeting, LIP6, Paris, France (slides), Jul 2003

PDF PS Bulej, L., Tuma, P.: Current Trends in Middleware Benchmarking,
Proceedings of the Week of Doctoral Students 2003 conference (WDS 2003), Matfyzpress, Charles University, Prague, Czech Republic, pp.232-237, Jun 2003

PDF Buble, A., Tuma, P.: On Benchmarking Object-oriented Communication Middleware,
Proceedings of the Week of Doctoral Students 2000 conference (WDS 2000), Faculty of Mathematics and Physics, Charles University, Prague, Jun 2000

FTP Plasil, F., Tuma, P., Buble, A.: Charles University Response to the Benchmark RFI,
OMG document bench/98-10-04, Oct 1998

Technical reports
PDF Tuma, P., Buble, A.: Technical Report on Open CORBA Benchmarking,
Tech. Report No. 2001/1, Dep. of SW Engineering, Charles University, Prague (slightly modified form appears in the SPECTS '01 conference proceedings), Mar 2001

PDF Distributed Systems Research Group: CORBA Comparison Project,
Project Extension Final Report, Aug 1999

PDF Plasil, F., Tuma, P., Buble, A.: CORBA Benchmarking,
Tech. Report No. 98/7, Dep. of SW Engineering, Charles University, Prague (in a slightly modified form submitted to OMG as the Charles University Response to ORBOS Benchmark RFI as the OMG document bench/98-10-04), Oct 1998

PDF Distributed Systems Research Group: CORBA Comparison Project,
Final Project Report, Jun 1998

Master theses (selected)
PDF WWW Kuhn, J.: IIOP Engine For Testing CORBA Implementation Robustness,
Master Thesis, advisor: Petr Tuma, Sep 2002

PDF WWW Strnad, P.: IDL Source Generator and IDL Compiler Testing Tool,
Master Thesis, advisor: Petr Tuma, Apr 2002

PDF PDF Buble, A.: Comparing CORBA Implementations,
Master Thesis, advisor: Petr Tuma, Aug 1999

Last modified December 4, 2006

Refereed (journals/proceedings)
PDF PDF WWW	Kalibera, T., Bulej, L., Tuma, P.: Automated Detection of Performance Regressions: The Mono Experience, in proceedings of the 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2005), Atlanta, GA, USA, Copyright (C) IEEE, Piscataway, New Jersey, USA, ISBN 0-7695-2458-3, ISSN 1526-7539, pp. 183-190, Sep 2005
PDF WWW	Kalibera, T., Bulej, L., Tuma, P.: Quality Assurance in Performance: Evaluating Mono Benchmark Results, in proceedings of the Second International Workshop on Software Quality (SOQUA 2005), Erfurt, Germany, Copyright (C) Springer-Verlag, Berlin, LNCS3712, ISBN 3-540-29033-8, ISSN 0302-9743, pp. 271-288, Sep 2005
PDF PDF	Kalibera, T., Bulej, L., Tuma, P.: Benchmark Precision and Random Initial State, in Proceedings of the 2005 International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS 2005), Cherry Hill, NJ, USA, Copyright (C) SCS, San Diego, CA, USA, pp. 853-862, ISBN 1-56555-300-4, Jul 2005
PDF WWW	Bulej, L., Kalibera, T., Tuma, P.: Repeated Results Analysis for Middleware Regression Benchmarking, in Performance Evaluation: An International Journal, Performance Modeling and Evaluation of High-Performance Parallel and Distributed Systems, vol. 60, pg. 345-358, Elsevier B.V., ISSN 0166-5316, May 2005
PDF PDF WWW	Kalibera, T., Bulej, L., Tuma, P.: Generic Environment for Full Automation of Benchmarking, in proceedings of Net.ObjectDays 2004, First International Workshop on Software Quality (SOQUA 2004), Erfurt, Germany, Copyright (C) 2004 tranSIT GmbH Ilmenau, Germany, pp. 35-41, ISBN 3-9808628-3-6 ; also in Testing of Component-Based Systems and Software Quality, Proceedings of SOQUA 2004 and TECOS 2004, LNI 58, GI 2004, pp. 125-132, ISBN 3-88579-387-3, ISSN 1617-5468, Sep 2004
PDF PDF WWW	Bulej, L., Kalibera, T., Tuma, P.: Regression Benchmarking with Simple Middleware Benchmarks, in proceedings of the 2004 International Performance Computing and Communications Conference (IPCCC 2004), International Workshop on Middleware Performance, Phoenix, AZ, USA, Copyright (C) 2004 IEEE, Piscataway, New Jersey, USA, pp. 771-776, ISBN 0-7803-8396-6, ISSN 1097-2641, Apr 2004
PDF PDF	Buble, A., Bulej, L., Tuma, P.: CORBA Benchmarking: A Course With Hidden Obstacles, in proceedings of the 2003 Internation Parallel & Distributed Processing Symposium (IPDPS 2003), Workshop on Performance Modeling, Evaluation and Optimization of Parallel and Distributed Systems (PMEOPDS 2003), Nice, France, Copyright (C) 2003 IEEE, Piscataway, New Jersey, USA, ISBN 0-7695-1926-1, ISSN 1530-2075, pp. 279, CDROM DATA/W18_PMEO_11.PDF, Apr 2003
PDF	Tuma, P., Buble, A.: Overview of the CORBA Performance, Proceedings of the 2002 EurOpen.CZ Conference, Znojmo, Czech Republic, Sep 2002
PDF	Tuma, P., Buble, A.: Open CORBA Benchmarking, Proceedings of the 2001 International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS 2001), Orlando, FL, Copyright (C) SCS, San Diego, CA, USA, Jul 2001
Nonrefereed (proceedings/slides)
PDF PDF WWW	Kalibera, T., Bulej, L., Tuma, P.: Mono Regression Benchmarking, in proceedings of WDS'05, MatfyzPress, Charles University, Prague, Czech Republic, pp. 19-24, ISBN 80-86732-59-2, Sep 2005
PDF	Kalibera, T.: Regression Benchmarking Framework Architecture, Presented at ObjectWeb Benchmarking Workshop, Prague, Czech Republic, Jul 2004
PDF	Bulej, L.: Writing System Probes on Various Platforms, Presented at ObjectWeb Benchmarking Workshop, Prague, Czech Republic, Jul 2004
PDF PDF	Kalibera, T.: Regression Benchmarking Environment, in proceedings of WDS'04, MatfyzPress, Charles University, Prague, Czech Republic, pp. 174-178, ISBN 80-86732-32-0, Jun 2004
PDF DOC	Bulej, L., Kalibera, T., Tuma, P.: Regression Benchmarking, Presented at Department Meeting Kokorin04, Jun 2004
PDF	Kalibera, T.: Middleware Benchmarking Project - Regression Benchmarking, Presented at ObjectWeb Benchmarking Workshop, Grenoble, France, Apr 2004
PDF	Tuma, P., Bulej, L.: Current Trends in Middleware Benchmarking, ObjectWeb Architecture Meeting, LIP6, Paris, France (slides), Jul 2003
PDF PS	Bulej, L., Tuma, P.: Current Trends in Middleware Benchmarking, Proceedings of the Week of Doctoral Students 2003 conference (WDS 2003), Matfyzpress, Charles University, Prague, Czech Republic, pp.232-237, Jun 2003
PDF	Buble, A., Tuma, P.: On Benchmarking Object-oriented Communication Middleware, Proceedings of the Week of Doctoral Students 2000 conference (WDS 2000), Faculty of Mathematics and Physics, Charles University, Prague, Jun 2000
FTP	Plasil, F., Tuma, P., Buble, A.: Charles University Response to the Benchmark RFI, OMG document bench/98-10-04, Oct 1998
Technical reports
PDF	Tuma, P., Buble, A.: Technical Report on Open CORBA Benchmarking, Tech. Report No. 2001/1, Dep. of SW Engineering, Charles University, Prague (slightly modified form appears in the SPECTS '01 conference proceedings), Mar 2001
PDF	Distributed Systems Research Group: CORBA Comparison Project, Project Extension Final Report, Aug 1999
PDF	Plasil, F., Tuma, P., Buble, A.: CORBA Benchmarking, Tech. Report No. 98/7, Dep. of SW Engineering, Charles University, Prague (in a slightly modified form submitted to OMG as the Charles University Response to ORBOS Benchmark RFI as the OMG document bench/98-10-04), Oct 1998
PDF	Distributed Systems Research Group: CORBA Comparison Project, Final Project Report, Jun 1998
Master theses (selected)
PDF WWW	Kuhn, J.: IIOP Engine For Testing CORBA Implementation Robustness, Master Thesis, advisor: Petr Tuma, Sep 2002
PDF WWW	Strnad, P.: IDL Source Generator and IDL Compiler Testing Tool, Master Thesis, advisor: Petr Tuma, Apr 2002
PDF PDF	Buble, A.: Comparing CORBA Implementations, Master Thesis, advisor: Petr Tuma, Aug 1999