The Volano Report

What is VolanoMark?

For background information and download locations, please see the Java Benchmarks page on Volano's Web site. Since the server side of the VolanoMark benchmark is our VolanoChat server, these test results help our customers choose the best Java virtual machine for their environment.

VolanoMark is a pure Java server benchmark characterized by long-lasting network connections and high thread counts. The VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the clients to take turns broadcasting their messages to the group. At the end of the test, it reports a score as the average number of messages transferred by the server per second.

Bug Id 4075058, "java.io: Add support for non-blocking I/O," explains why you need lots of threads in order to handle dedicated connections to a pure Java server. JSR 51, "New I/O APIs for the Java Platform," explains how Sun plans to change that.

What's new?

Well, just about everything is new -- the network, the hardware, the operating systems, and the Java virtual machines. This time I restricted testing to the Java 2 Standard Edition and excluded IBM OS/2 Warp Server and Novell NetWare, but that still left 12 Java platforms on 4 operating systems.

This is the first time that Tower's TowerJ did not come out on top in the performance test. TowerJ's problems in the network scalability test, though, are most likely due to my running the test on Red Hat Linux 6.2, which Tower has not yet tested. In the network scalability test, only Appeal's JRockit 1.1.2 on Windows and Blackdown's J2SE 1.2.2-FCS on Linux made it all the way to 5,000 concurrent connections.

On Linux, the latest "FCS" version of Blackdown's port solved problems that I hit trying to run Blackdown's earlier "RC4" version on Red Hat 6.2 (Linux 2.2.14, Glibc 2.1.3), even though "RC4" ran fine on Red Hat 6.0 (Linux 2.2.5, Glibc 2.1.1). In these tests and in the previous Volano Report back in March, the Blackdown releases have been consistently more stable at high connection counts than Sun's own releases of the same version.

On Solaris Intel, Sun's older ExactVM (1.2.2) outperformed the new HotSpot Server VM (1.3.0), although results may be quite different for Solaris SPARC. Both virtual machines do very well in the network scalability test, failing only when there are more than 4,000 concurrent connection. Our customers have tested successfully up to 10,000 concurrent connections with Java on Solaris SPARC, so these problems may be limited to Java on the Solaris Intel Edition.

Without a native binary release of Java 2 Standard Edition for FreeBSD, I tested the Linux Java VMs with FreeBSD's Linux emulation package. The earlier Blackdown "RC4" release and Sun's 1.3.0 Classic VM worked quite well up to about 980 connections on FreeBSD, but the latest "FCS" Blackdown release failed to run at all. In addition, neither Sun's nor Blackdown's native thread Java VMs, including HotSpot, worked under the FreeBSD Linux emulation.

All of the Windows Java virtual machines did well, but only Appeal's JRockit gets around the Windows stack-based thread limit by mapping the Java threads onto its own smaller set of native threads.

The tests

The tests presented in this report look at the performance and network scalability of 12 Java virtual machines on 4 operating systems using a common Intel hardware platform running VolanoMark 2.1. You can find links to VolanoMark test results on other (non-Intel) hardware platforms on our Java Benchmarks page. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of dedicated socket connections:

  1. Is it fast?
  2. Does it scale?

See the Details section for details on the hardware platform, operating systems, and Java virtual machine environments.

Is it fast?

Run rules

The performance test was executed with the following commands on a local loopback connection, using the heap size options shown below where possible:

Server
java -Xms8m -Xmx64m COM.volano.Main
Client
java -Xms8m -Xmx64m COM.volano.Mark -count 100

See the COM.volano.Mark command synopsis for a complete description of all options. The operating system was rebooted before each set of tests for a particular Java virtual machine. The server side was not restarted before each run of the client benchmark. The client benchmark ran four times, with the final score as the average of the last three results.

Results

Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the Details section for details on the hardware platform, operating systems, and Java virtual machine environments.

Table 1. VolanoMark 2.1.2 performance test, measuring throughput in messages per second with 200 concurrent local loopback connections.

Does it scale?

Run rules

The network scalability test was executed with the following commands over a 100-Mbps Ethernet cross-over cable connection, using the heap and stack size options shown below where possible:

Server
java -Xms8m -Xmx256m -Xss32k COM.volano.Main
Client
java -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 50
java -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 100
java -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 150
java -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 200
java -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 250

See the COM.volano.Mark command synopsis for a complete description of all options. The client test driver was executed under the Sun J2SE 1.3.0 Release Candidate HotSpot Server VM using Solaris 8 (6/00) on a 500-MHz Intel Pentium III with 384 megabytes of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs.

Results

Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the Details section for details on the hardware platform, operating systems, and Java virtual machine environments.

Table 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second with 1000 through 5000 concurrent network connections.

Details

All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Hardware

Server

The server machine under test is a Dell OptiPlex GXpro with a 200-MHz Intel Pentium Pro processor, 256 kilobytes L2 cache, 320 megabytes ECC EDO RAM, a 3Com EtherLink XL 10/100 PCI TX NIC (3C905B-TX), and Phoenix ROM BIOS PLUS Version 1.10 A10.

Client

The client side driving the network tests is a Dell OptiPlex GX1 with a 500-MHz Intel Pentium III processor, 512 kilobytes L2 cache, 384 megabytes ECC SDRAM, a 3Com EtherLink XL 10/100 PCI TX NIC (3C905B-TX), and Dell OptiPlex GX1 500L+ BIOS Revision A08.

Software

Test scripts

Loopback performance tests

The following Unix shell scripts and Windows batch files were used for the loopback performance tests with 200 concurrent connections.

The server side used:

The client side used:

For a sleep function, the Windows batch files used sleep.java.

Network scalability tests

The following Unix shell scripts and Windows batch files were used for the network scalability tests with 1000 through 5000 concurrent connections.

The server side used:

The client side used:

FreeBSD 4.0-RELEASE

Operating system version:

FreeBSD ppro.sf.volano.net 4.0-RELEASE FreeBSD 4.0-RELEASE #0:
Mon Mar 20 22:50:22 GMT 2000
root@monster.cdrom.com:/usr/src/sys/compile/GENERIC  i386

Contents of /etc/sysctl.conf:

# Increase file descriptor limits from their defaults of 1064.
kern.maxfiles=16384             # System-wide limit
kern.maxfilesperproc=8192       # Per-process limit
Blackdown/Inprise J2SE 1.2.2 FreeBSD

Downloaded from:

http://www.blackdown.org/java-linux/ports.html

Ran using the Linux emulation package (Linux 2.2.12, Glibc 2.1.2) and the Inprise just-in-time compiler. Changed the calls to expr in jre1.2.2/bin/.java_wrapper as follows:

expr -> /usr/compat/linux/usr/bin/expr

Java version:

java version "1.2.2"
Classic VM (build Linux_JDK_1.2.2_FCS, green threads, javacomp)
Sun/Inprise J2SE 1.3.0 FreeBSD

Downloaded from:

http://java.sun.com/products/jdk/1.3/

Ran using the Linux emulation package (Linux 2.2.12, Glibc 2.1.2) and the Inprise just-in-time compiler. Changed the calls to /usr/bin/expr in jre1.3/bin/.java_wrapper as follows:

/usr/bin/expr -> /usr/compat/linux/usr/bin/expr

Java version:

java version "1.3.0beta_refresh"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0beta_refresh-b09)
Classic VM (build 1.3.0beta_refresh-b09, green threads, javacomp)

Microsoft Windows 2000 Professional

Operating system version:

Microsoft Windows 2000 Professional
Version 5.0.2195 Service Pack 1 Build 2195

Optimized performance for background services (rather than applications) under Start, Settings, Control Panel, System, Advanced, Performance Options, Application response. This change specifies that all programs receive equal amounts of processor resources.

Appeal JRockit 1.1.2 Windows

Downloaded from:

http://www.jrockit.com/

Windows 2000 Service Pack 1 broke the stack size options for JRockit, so neither the Xss32k option nor the Xexpandstacks option were used.

Java version:

java version "1.3"
JRockit(tm) Virtual Machine (version 1.1.2, build sarek130, Thin Threads)
Java Home: C:\Program Files\JavaSoft\JRE\1.3
Using bootclasspath
  C:\Program Files\JavaSoft\JRE\1.3\lib\rt.jar;
  C:\Program Files\JavaSoft\JRE\1.3\lib\i18n.jar;
  C:\Program Files\JavaSoft\JRE\1.3\classes
IBM J2SE 1.3.0 Windows

Downloaded from:

http://www.ibm.com/java/jdk/download/index.html

Java version:

java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0)
Classic VM (build 1.3.0, J2RE 1.3.0 IBM build cn130-20000622 (JIT enabled: jitc))
Sun J2SE 1.3.0 HotSpot Server Windows

Downloaded from:

http://java.sun.com/products/jdk/1.3/

Java version:

java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Server VM (build 2.0fcs-E, mixed mode)

Red Hat Linux 6.2

Operating system version (Linux 2.2.14, Glibc 2.1.3):

Red Hat Linux release 6.2 (Zoot)
Linux ppro.sf.volano.net 2.2.14-5.0 #1
  Tue Mar 7 21:07:39 EST 2000 i686 unknown

Added to /etc/rc.d/rc.local:

# Increase system-wide file descriptor limit.
echo  32768 > /proc/sys/fs/file-max
echo 131072 > /proc/sys/fs/inode-max

Added to /etc/pam.d/login:

session    required     /lib/security/pam_limits.so

Added to /etc/security/limits.conf:

*       soft    nofile  1024
*       hard    nofile  8192
Blackdown/Inprise J2SE 1.2.2 Linux

Downloaded from:

http://www.blackdown.org/java-linux/ports.html

Ran using the Inprise just-in-time compiler.

Java version:

java version "1.2.2"
Classic VM (build Linux_JDK_1.2.2_FCS, green threads, javacomp)
IBM J2SE 1.3.0 Linux

Downloaded from:

http://www.ibm.com/java/jdk/download/index.html

Java version:

java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0)
Classic VM (build 1.3.0, J2RE 1.3.0 IBM build cx130-20000623 (JIT enabled: jitc))
Sun J2SE 1.3.0 HotSpot Server Linux

Downloaded from:

http://java.sun.com/products/jdk/1.3/

Java version:

java version "1.3.0beta_refresh"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0beta_refresh-b09)
Java HotSpot(TM) Server VM (build 1.3.0beta-b07, mixed mode)
Sun/Inprise J2SE 1.3.0 Linux

Downloaded from:

http://java.sun.com/products/jdk/1.3/

Ran using the Inprise just-in-time compiler.

Java version:

java version "1.3.0beta_refresh"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0beta_refresh-b09)
Classic VM (build 1.3.0beta_refresh-b09, green threads, javacomp)
Tower TowerJ 3.5.3 Linux

Downloaded from:

http://www.towerj.com/

Used the TowerJ project files Main.tj and Mark.tj. Built against Blackdown JDK 1.2.2 RC4 with the environment variables:

export TOWERJ=/usr/local/TowerJ
export TOWERJ_JAVA_HOME=/usr/local/jdk1.2.2/jre
export LD_LIBRARY_PATH=$TOWERJ/lib/x86-linux

Java version:

TowerJ Compiler (version 3.5.3.0 x86-linux)
Copyright (c) 2000 Tower Technology Corporation. All rights reserved.

Sun Solaris 8 (6/00)

Operating system version:

Solaris 8 6/00 s28x_u1wos_08 INTEL
SunOS ppro 5.8 Generic_108529-01 i86pc i386 i86pc

Added patch 108941-07, "Motif 2.1.1_x86: Runtime library patch for Solaris 8_x86," as required by Sun J2SE 1.3.0-RC.

Added to /etc/system:

set rlim_fd_max = 8192
set rlim_fd_cur = 1024
Sun J2SE 1.2.2 Solaris

Downloaded from:

http://www.sun.com/software/solaris/java/download.html

Java version:

java version "1.2.2"
Solaris VM (build Solaris_JDK_1.2.2_05a, native threads, sunwjit)
Sun J2SE 1.3.0 HotSpot Server Solaris

Downloaded from:

http://www.sun.com/software/solaris/java/download.html

Java version:

java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-RC)
Java HotSpot(TM) Server VM (build 1.3.0-RC, mixed mode)

Valid HTML 4.0! Valid CSS! Copyright © 2000 Volano LLC. All rights reserved.
Contact John Neffenger with questions or comments.