Component And Middleware Performance
	OOPSLA 2004 Workshop October 25, 2004 Vancouver Convention & Exhibition Centre Vancouver, British Columbia, Canada
The workshop call for papers has been issued Published in special issue of Studia Informatica journal

The goal of the workshop is to help advance the current practice of gathering performance characteristics of component systems and middleware implementations through benchmarking. Positions are solicited especially from people with previous or impending benchmarking experience.

The participants of the workshop will identify requirements and obstacles of component and middleware benchmarking and form a position on issues such as designing a framework that would allow the design, running and evaluating a diverse range of benchmarks over a diverse range of platforms, designing benchmark criteria that would allow for a meaningful comparison of results collected over different platforms, designing a framework suitable for regular regression testing, or providing means to verify the benchmarking results and their applicability to specific usage situations.

Schedule

Please note that the schedule is meant to be very flexible. At no time will discussion be stiffled for the mere purpose of observing the set presentation times.

Submissions

The attendees are invited to submit a statement of interest, a position paper or an experience report. The submitted materials should be reasonably concise. The organizing committee will disseminate all submissions to the workshop attendees prior to the workshop and select position papers and experience reports to be presented at the beginning of the workshop. The position papers and experience reports selected for presentation will be published in a special issue of the Studia Informatica journal on components and performance, scheduled for publication in Spring 2005. A joint position of the attendees will be considered for publication.

The topics are envisioned to include, but not be limited, to:

Current practice:

• Experience with existing benchmarks and benchmarking frameworks (ECperf, JMeter, Load Runner, OpenCB, OpenSTA, RUBiS and others).
• Existing component and middleware benchmarking initiatives and standardization efforts (JCP, SPEC, TSS and others).

Issues concerning benchmarks

• Impact and characteristics of various object oriented designs in benchmarks.
• Characterizing requirements on platform performance.
• Characterizing workload and background load.
• Platform independent benchmark specification.

Issues concerning benchmarking frameworks

• Designing frameworks for a diverse range of benchmarks over a diverse range of platforms.
• Designing frameworks for standalone benchmarking of components.
• Designing framework for regular regression testing.

Issues concerning benchmarking applications

• Use of benchmarking results for performance modeling.
• Meaningful comparison of results collected over different platforms.
• Guidelines for carrying out performance and scalability evaluation studies.
• Use of benchmarking results for predictable assembly of components and middleware.
• Means of verifying the benchmarking results and their applicability to specific usage situations.

The statements of interest, position papers and experience reports should be mailed to the organizing committee at oopsla-workshop@nenya.ms.mff.cuni.cz. Any of the DOC, PDF, PS or RTF formats are accepted, for formatting instructions please see the Studia Informatica journal.

Important dates

Organizing committee

• Paul Brebner, University College London, United Kingdom
• Emmanuel Cecchet, ObjectWeb, INRIA Rhone-Alpes, France
• Petr Tuma, Charles University, Czech Republic

Program committee

• Paul Brebner, University College London, United Kingdom
• Emmanuel Cecchet, ObjectWeb, INRIA Rhone-Alpes, France
• Patrick Lardieri, Lockheed Martin Advanced Technology Labs, United States
• John Murphy, Performance Engineering Laboratory, Dublin City University, Ireland
• Balachandran Natarajan, Vanderbilt University, United States
• Gautam G. Thaker, Lockheed Martin Advanced Technology Labs, United States
• Petr Tuma, Charles University, Czech Republic

Contact us at oopsla-workshop@nenya.ms.mff.cuni.cz.
For the 2003 workshop page, click here.