The Volano Report

Last updated on December 27, 2001, to include Excelsior JET 2.1.

For background information on VolanoMark^™ and its download locations, please see our Benchmarks page. Since the server side of the VolanoMark benchmark is our VolanoChat server, these test results help our customers choose the best Java virtual machine for their environment. VolanoMark is a pure Java server benchmark characterized by long-lasting network connections and high thread counts. The VolanoMark benchmark creates client connections in groups of 20 and measures the time required by the clients to take turns broadcasting a set of messages to the group. At the end of the test, it reports a score as the average number of messages transferred by the server per second. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Bug Id 4075058, "java.io: Add support for non-blocking I/O," explains why you need lots of threads to handle dedicated connections to a pure Java server. JSR 51, "New I/O APIs for the Java Platform," explains how Sun is addressing the problem starting with Java version 1.4.

Performance

The performance test was executed with the following commands on a local loopback connection, using the heap size options shown below where possible. The operating system was rebooted before each set of tests for a particular Java virtual machine. The client benchmark ran four times, with the final score as the average of the last three results. The server side was not restarted before each run of the client benchmark.

Server: java -Xms8m -Xmx64m COM.volano.Main
Client: java -Xms8m -Xmx64m COM.volano.Mark -count 100

Figure 1. VolanoMark 2.1.2 performance test showing the throughput in messages per second with 200 concurrent local loopback connections. Bigger numbers are faster. Click here for details.

Network Scalability

The network scalability test was executed with the following commands over a 100-Mbps Ethernet cross-over cable connection, using the heap and stack size options shown below where possible. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs.

Server: java -Xms8m -Xmx256m -Xss32k COM.volano.Main
Client: java -Xms64m -Xmx256m -Xss64k COM.volano.Mark -host xxx.yyy.zzz -rooms 50
java -Xms64m -Xmx256m -Xss64k COM.volano.Mark -host xxx.yyy.zzz -rooms 100
...
java -Xms64m -Xmx256m -Xss64k COM.volano.Mark -host xxx.yyy.zzz -rooms 300

Figure 2. VolanoMark 2.1.2 network scalability test showing the maximum number of concurrent network connections achieved during the test. Bigger numbers indicate greater network scalability. The order of the Java virtual machines is preserved from Figure 1. Click here for details.

I ran the TowerJ network test with the default maximum heap size of 4 gigabytes, since TowerJ allocates the 32-kilobyte stacks onto the Java heap in addition to the other Java objects. TowerJ needs an extra 375 megabytes on the Java heap due to this design. The other Java VMs require this storage as well, but allocate it outside the heap.

Environment

Hardware

Server

The server machine under test is a Dell OptiPlex GXpro with a 200-MHz Intel Pentium Pro processor, 256 kilobytes L2 cache, 320 megabytes ECC EDO RAM, a 3Com EtherLink XL 10/100 PCI TX NIC (3C905B-TX), and Phoenix ROM BIOS PLUS Version 1.10 A10.

Note that this is not a hardware test but rather a test of the relative performance and network scalability of these Java platforms running VolanoMark. We continue to use a 200 MHz Intel Pentium Pro so that you can compare these results with our previous Volano Reports and so that we can drive the system with a much more powerful processor on the client side of the network scalability tests.

With the exception of Sun 1.3.1 on Solaris, the maximum number of concurrent connections shown in Figure 2 is due to architectural constraints rather than processor speed or the actual amount of memory on the system.

Client

The client side driving the network tests is a Dell OptiPlex GX1 with a 500-MHz Intel Pentium III processor, 512 kilobytes L2 cache, 384 megabytes ECC SDRAM, a 3Com EtherLink XL 10/100 PCI TX NIC (3C905B-TX), and Dell OptiPlex GX1 500L+ BIOS Revision A08. The client test driver was executed under the Sun J2SE 1.3.0_02 HotSpot Server VM using Solaris 8 (6/00).

Note that the client side must be a much more powerful system than the server side since it simulates up to 6,000 clients on just one machine and must drive the server side at its maximum speed.

Software

Test Scripts

Loopback performance tests

The following Unix shell scripts and Windows batch files were used for the loopback performance tests with 200 concurrent connections. For a sleep function, the Windows batch files used sleep.java.

server.sh: UNIX server side
client.sh: UNIX client side
server.bat: Windows server side
client.bat: Windows client side

Network Scalability Tests

The following Unix shell scripts and Windows batch files were used for the network scalability tests with 1,000 through 6,000 concurrent connections.

netserver.sh: UNIX server side
netserver.bat: Windows server side
netclient.sh: Client side on Solaris 8 (6/00)

Operating Systems

FreeBSD 4.2-RELEASE

FreeBSD ppro 4.2-RELEASE FreeBSD 4.2-RELEASE #0: Mon Nov 20 13:02:55 GMT 2000
  jkh@bento.FreeBSD.org:/usr/src/sys/compile/GENERIC  i386

Contents of /etc/sysctl.conf:

# Increase file descriptor limits from their defaults of 1064.
kern.maxfiles=16384             # System-wide limit
kern.maxfilesperproc=8192       # Per-process limit

Microsoft Windows 2000 Professional

Microsoft Windows 2000 Service Pack 2 [Version 5.00.2195]

Optimized performance for background services (rather than applications) under Start, Settings, Control Panel, System, Advanced, Performance Options, Application response. This change specifies that all programs receive equal amounts of processor resources.

Red Hat Linux 7.1

Applied all updates as of May 28, 2001, to get the following Linux kernel and Glibc versions for all but the TowerJ and JRockit tests:

Red Hat Linux release 7.1 (Seawolf)
Linux ppro.sf.volano.net 2.4.2-2 #1 Sun Apr 8 20:41:30 EDT 2001 i686 unknown
glibc-2.2.2-10

For the TowerJ 3.8.1 and JRockit 3.0.0 tests, applied all updates as of December 10, 2001, to get the following Linux kernel and Glibc versions:

Red Hat Linux release 7.1 (Seawolf)
Linux ppro.sf.volano.net 2.4.9-12 #1 Tue Oct 30 18:33:49 EST 2001 i686 unknown
glibc-2.2.4-19

Added to /etc/pam.d/login:

session    required     /lib/security/pam_limits.so

Added to /etc/security/limits.conf:

*               soft    nofile          1024
*               hard    nofile          8192

Sun Solaris 8 (6/00)

Added latest recommended patch cluster dated May 23, 2001.

Solaris 8 6/00 s28x_u1wos_08 INTEL
SunOS ppro 5.8 Generic_108529-08 i86pc i386 i86pc

Added to /etc/system:

set rlim_fd_max = 8192
set rlim_fd_cur = 1024

Java Virtual Machines

IBM 1.3.0 Linux

java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0)
Classic VM (build 1.3.0, J2RE 1.3.0 IBM build cx130-20010502 (JIT enabled: jitc))

IBM 1.3.0 Windows

java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0)
Classic VM (build 1.3.0, J2RE 1.3.0 IBM build cn130-20010329 (JIT enabled: jitc))

Tower TowerJ 3.8.1 Linux

TowerJ Compiler (version 3.8.1.0 (Tower JRE 1.3.1) x86-linux)
Java version "1.3.1-internal"

Used the TowerJ project files Main.tj and Mark.tj.

Excelsior JET 2.1 Windows

Excelsior JET v2.1 Evaluation Version (c) Excelsior 1997,2001
Java vendor:      Sun Microsystems Inc.
Java vendor URL:  http://www.excelsior-usa.com/
Java version:     1.3.1_01

Used the JET project files Main.prj and Mark.prj.

Appeal JRockit 3.0.0 Linux

java version 1.3.1_dax.appeal.se-20011120-1857
Java(TM) 2 Runtime Environment, Standard Edition
  (build "1.3.1_dax.appeal.se-20011120-1857")
JRockit Virtual Machine (build 3.0.0-dax.appeal.se-20011120-1610)

Microsoft 1.1.4 Windows

Microsoft (R) Command-line Loader for Java  Version 5.00.3802
Copyright (C) Microsoft Corp 1996-2000. All rights reserved.

Appeal JRockit 2.0.7 Windows

JRockit build 2.0.7-excelsior80, Thin Threads,
  Generational Stop & Copy Garbage Collector, started.

java version "1.3.1"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1-b24)
JRockit(tm) Virtual Machine (build 2.0.7-excelsior80, Thin Threads)

Sun 1.3.1 Windows (HotSpot Server VM)

java version "1.3.1"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1-b24)
Java HotSpot(TM) Server VM (build 1.3.1-b24, mixed mode)

Sun 1.3.1 Linux (HotSpot Server VM)

java version "1.3.1"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1-b24)
Java HotSpot(TM) Server VM (build 1.3.1-b24, mixed mode)

Sun 1.2.2 Solaris (Exact VM)

java version "1.2.2"
Solaris VM (build Solaris_JDK_1.2.2_08, native threads, sunwjit)

Blackdown 1.3.0 Linux (green threads, OpenJIT)

java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build Blackdown-1.3.0-FCS)
Classic VM (build Blackdown-1.3.0-FCS, green threads, OpenJIT)

Blackdown 1.3.0 FreeBSD (green threads, OpenJIT)

java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build Blackdown-1.3.0-FCS)
Classic VM (build Blackdown-1.3.0-FCS, green threads, OpenJIT)

Ran using the Linux emulation package:

Linux ppro 2.2.12 FreeBSD 4.2-RELEASE #0:
  Mon Nov 20 13:02:55 GMT 2000 jkh@ben i386 unknown
Glibc 2.1.2

Modified /usr/local/j2sdk1.3.0/bin/.java_wrapper as follows:

$ diff .java_wrapper.old .java_wrapper
36,37c36,37
<     link=`expr "$ls" : '.*-> \(.*\)$'`
<     if expr "${link}" : '/' > /dev/null; then
---
>     link=`/usr/compat/linux/usr/bin/expr "$ls" : '.*-> \(.*\)$'`
>     if /usr/compat/linux/usr/bin/expr "${link}" : '/' > /dev/null; then

Sun 1.3.1 Solaris (HotSpot Server VM)

java version "1.3.1"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1-b24)
Java HotSpot(TM) Server VM (build 1.3.1-b24, mixed mode)