The Volano Report

The tests presented in this report look at the performance and network scalability of 16 Java virtual machines on 7 operating systems using a common Intel hardware platform running VolanoMark™ 2.1. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of dedicated socket connections:

  1. Is it fast?
  2. Does it scale?

When run at high connection counts, VolanoMark essentially measures the consequences of omitting non-blocking read and write operations from the Java platform, as summarized by this Java Developer Connection request for enhancement (free registration required):

Note that I have been unsuccessful running VolanoMark on the following Java platforms:

For background information and download locations, please see the Java Benchmarks page on Volano's Web site. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments, along with a table showing the compiler and threading model of each Java platform.

Reality check

Volano has been testing Java virtual machines for over three years both in our lab with the VolanoMark benchmark and live on the Internet with our VolanoChat server. We have found that performance is no longer the limiting factor in pure Java network server applications running on relatively new hardware. Other factors limit the capacity and throughput of the server long before the processor usage reaches 100 percent -- factors such as the stability of the platform, the memory requirements, and architectural limits on the number of sockets and threads per server process. The results of our attempts to test those limits are shown in Figure 2 and Table 2 below.

Based on our experience and these test results, we recommend the following Java platforms for running a pure Java server application:

Is it fast?

Run rules

The performance test was executed with the following commands on a local loop-back connection, using the heap size options shown below where possible:

Server
java -ms8m -mx64m COM.volano.Main
Client
java -ms8m -mx64m COM.volano.Mark -count 100
See the COM.volano.Mark command synopsis for a complete description of all options.

The operating system was rebooted before each set of tests for a particular Java virtual machine. The server side was not restarted before each run of the client benchmark. The client benchmark ran four times, with the final score as the average of the last three results.

Results

Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

Java Platform Score
Tower TowerJ 3.1.4 Linux
IBM JDK 1.1.8 Windows NT
IBM JDK 1.1.8 OS/2
Microsoft VM 3229 Windows 2000
IBM JDK 1.1.8 Linux
Sun HotSpot 1.0.1 Windows NT
Sun JDK 1.3 Beta Windows NT
Sun JDK 1.2.2 Windows NT
Microsoft VM 3186 Windows NT
Sun JDK 1.2.1_04 Solaris
Novell JDK 1.1.7 NetWare
Sun JDK 1.2.1_03 Solaris
Sun JDK 1.2 Linux
Transvirtual Kaffe 1.0b4 Linux
Blackdown JDK 1.1.7 Linux
JDK 1.1.8 FreeBSD
2309 2309
2279 2279
2136 2136
1970 1970
1770 1770
1601 1601
1550 1550
1485 1485
1478 1478
1358 1358
1229 1229
1051 1051
915 915
389 389
285 285
173 173
Figure 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second.

Java Platform Operating System Results Score
Tower TowerJ 3.1.4 Linuxa Red Hat Linux 6.0 Intel 2303, 2313, 2313, 2300 2309
IBM JDK 1.1.8 Windows NT Windows NT Workstation 4.0 2282, 2276, 2280, 2282 2279
IBM JDK 1.1.8 OS/2 OS/2 Warp Server for e-business 2145, 2141, 2118, 2150 2136
Microsoft VM 3229 Windows 2000 Windows 2000 Server Release Candidate 2 1948, 1965, 1976, 1968 1970
IBM JDK 1.1.8 Linux Red Hat Linux 6.0 Intel 1242, 1772, 1787, 1750 1770
Sun HotSpot 1.0.1 Windows NT Windows NT Workstation 4.0 1558, 1611, 1604, 1589 1601
Sun JDK 1.3 Beta Windows NT Windows NT Workstation 4.0 1538, 1550, 1556, 1545 1550
Sun JDK 1.2.2 Windows NT Windows NT Workstation 4.0 1496, 1481, 1489, 1485 1485
Microsoft VM 3186 Windows NT Windows NT Workstation 4.0 1469, 1484, 1478, 1473 1478
Sun JDK 1.2.1_04 Solaris Solaris 7 Desktop Intel Platform Edition 1371, 1344, 1374, 1355 1358
Novell JDK 1.1.7 NetWareb NetWare 5 1227, 1232, 1229, 1226 1229
Sun JDK 1.2.1_03 Solaris Solaris 7 Desktop Intel Platform Edition 1118, 1055, 1050, 1049 1051
Sun JDK 1.2 Linux Red Hat Linux 6.0 Intel 915, 914, 912, 919 915
Transvirtual Kaffe 1.0b4 Linuxc Red Hat Linux 6.0 Intel 397, 391, 390, 386 389
Blackdown JDK 1.1.7 Linux Red Hat Linux 6.0 Intel 284, 285, 286, 283 285
JDK 1.1.8 FreeBSD FreeBSD 3.2-RELEASE 169, 173, 173, 172 173
Table 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second. The final score is the average of the last three results. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Notes:

  1. TowerJ is the only Java platform listed here which is not available for free.
  2. Received "java.net.SocketException: Software caused connection abort" once and "java.net.SocketException: Socket closed" 42 times when the server side was killed on NetWare.
  3. The Kaffe OpenVM is the only Java platform listed here which is still unable to run our VolanoChat product due to Kaffe Bug io/78, "java.io.StreamCorruptedException".

Does it scale?

Run rules

The network scalability test was executed with the following commands over an isolated 10-Mbps Ethernet connection, using the heap and stack size options shown below where possible:

Server
java -ms8m -mx128m -ss32k COM.volano.Main
Client
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 50
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 100
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 150
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 200

See the COM.volano.Mark command synopsis for a complete description of all options.

The client test driver was executed under the Sun JDK 1.2.1_03 Production Release for Solaris using Solaris 7 on a dual 200-MHz Sun UltraSPARC with 256 megabytes of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs.

Results

Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

2486 2623 1521 1408 1474 1468 1875 1575 1794 612 165 37 1357 1826 708 856 770 1040 1058 739 1081 406 523 659 613 421 354
1000 2000 3000 4000
Tower TowerJ 3.1.4 Linux
IBM JDK 1.1.8 Windows NT
IBM JDK 1.1.8 OS/2
Microsoft VM 3229 Windows 2000
IBM JDK 1.1.8 Linux
Sun HotSpot 1.0.1 Windows NT
Sun JDK 1.3 Beta Windows NT
Sun JDK 1.2.2 Windows NT
Microsoft VM 3186 Windows NT
Sun JDK 1.2.1_04 Solaris
Novell JDK 1.1.7 NetWare
Sun JDK 1.2.1_03 Solaris
Sun JDK 1.2 Linux
Transvirtual Kaffe 1.0b4 Linux
Blackdown JDK 1.1.7 Linux
JDK 1.1.8 FreeBSD
Figure 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections.

Java Platform 1000 2000 3000 4000 Notes
Tower TowerJ 3.1.4 Linux 2486 1357 739 421 No errors!
IBM JDK 1.1.8 Windows NT 2623 1826 1081 ---- "java.net.SocketException: Socket read failed: 10107" at 3553/4000 connections.
IBM JDK 1.1.8 OS/2 1521 ---- ---- ---- "java.lang.OutOfMemoryError" at 1098/2000 connections.
Microsoft VM 3229 Windows 2000 1408 708 406 ---- "java.lang.OutOfMemoryError" at 3785/4000 connections.
IBM JDK 1.1.8 Linux ---- ---- ---- ---- Process hangs in hard run at 500/1000 connections and dumps core when killed.
Sun HotSpot 1.0.1 Windows NT ---- ---- ---- ---- "java.lang.OutOfMemoryError" at 902/1000 connections.
Sun JDK 1.3 Beta Windows NT 1474 856 523 ---- "java.lang.OutOfMemoryError" at 3272/4000 connections.
Sun JDK 1.2.2 Windows NT ---- ---- ---- ---- "java.lang.OutOfMemoryError" at 926/1000 connections.
Microsoft VM 3186 Windows NT 1468 770 ---- ---- "java.net.SocketException: No available buffer space" on 3000-connection test.
Sun JDK 1.2.1_04 Solaris 1875 1040 659 ---- "Bus error - core dumped" on 4000-connection test.
Novell JDK 1.1.7 NetWare 1575 ---- ---- ---- Process hangs with disk thrashing at 1960/2000 connections.
Sun JDK 1.2.1_03 Solaris 1794 1058 613 354 No errors!
Sun JDK 1.2 Linux ---- ---- ---- ---- Tests fail with "java.net.SocketException: Interrupted system call".
Transvirtual Kaffe 1.0b4 Linux 612 ---- ---- ---- Dumps core at just over 1000 connections.
Blackdown JDK 1.1.7 Linux 165 ---- ---- ---- Dumps core at just over 1000 connections.
JDK 1.1.8 FreeBSD 37 ---- ---- ---- Client begins failing at 1017 connections with "java.net.NoRouteToHostException: Connection timed out".
Table 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Environments

All tests ran identical copies of VolanoMark 2.1.2 on identical hardware -- a 200-MHz Intel Pentium Pro processor with a 256-kilobyte L2 cache and 256 megabytes of RAM on an Intel VS440FX motherboard running American Megatrends AMIBIOS Version 1.00.18.CS1 dated "08/28/98-13:59". The network card is a 3Com Etherlink III PCI Bus-Master Adapter Model 3C590.

Tower TowerJ 3.1.4 Linux
IBM JDK 1.1.8 Windows NT
IBM JDK 1.1.8 OS/2
Microsoft VM 3229 Windows 2000
IBM JDK 1.1.8 Linux
Sun HotSpot 1.0.1 Windows NT
Sun JDK 1.3 Beta Windows NT
Sun JDK 1.2.2 Windows NT
Microsoft VM 3186 Windows NT
Sun JDK 1.2.1_04 Solaris
Novell JDK 1.1.7 NetWare
Sun JDK 1.2.1_03 Solaris
Sun JDK 1.2 Linux
Transvirtual Kaffe 1.0b4 Linux
Blackdown JDK 1.1.7 Linux
JDK 1.1.8 FreeBSD

Java Platform Operating System Compiler Thread Model
Tower TowerJ 3.1.4 Linux Red Hat Linux 6.0 Intel static many-to-one
IBM JDK 1.1.8 Windows NT Windows NT Workstation 4.0 mixed mode one-to-one
IBM JDK 1.1.8 OS/2 OS/2 Warp Server for e-business mixed mode one-to-one
Microsoft VM 3229 Windows 2000 Windows 2000 Server Release Candidate 2 JIT one-to-one
IBM JDK 1.1.8 Linux Red Hat Linux 6.0 Intel JIT one-to-one
Sun HotSpot 1.0.1 Windows NT Windows NT Workstation 4.0 mixed mode one-to-one
Sun JDK 1.3 Beta Windows NT Windows NT Workstation 4.0 mixed mode one-to-one
Sun JDK 1.2.2 Windows NT Windows NT Workstation 4.0 JIT one-to-one
Microsoft VM 3186 Windows NT Windows NT Workstation 4.0 JIT one-to-one
Sun JDK 1.2.1_04 Solaris Solaris 7 Desktop Intel Platform Edition JIT many-to-many
Novell JDK 1.1.7 NetWare NetWare 5 JIT many-to-one
Sun JDK 1.2.1_03 Solaris Solaris 7 Desktop Intel Platform Edition JIT many-to-many
Sun JDK 1.2 Linux Red Hat Linux 6.0 Intel JIT one-to-one or many-to-one
Transvirtual Kaffe 1.0b4 Linux Red Hat Linux 6.0 Intel JIT many-to-one
Blackdown JDK 1.1.7 Linux Red Hat Linux 6.0 Intel none many-to-one
JDK 1.1.8 FreeBSD FreeBSD 3.2-RELEASE none many-to-one
Table 3. The compiler and threading model used by each Java platform.

Compiler

static
A static compiler which takes Java source files or class files and translates them into a native executable or shared library before they run.
mixed mode
A Java VM which translates some of the Java bytecodes into native instructions on the fly while the Java program runs, while letting a Java interpreter handle the other parts of the program.
JIT
A Java VM which translates the Java bytecodes into native instructions on the fly before executing them.
none
A Java VM which uses a normal Java bytecode interpreter.

Thread Model

one-to-one
Each Java thread is mapped one-to-one onto its own operating system thread.
many-to-one
All of the Java threads in the process are mapped onto one operating system process.
many-to-many
More than one Java thread is mapped to one operating system thread, and a Java thread may be mapped to different operating system threads during its lifetime.

Valid HTML 4.0! Valid CSS! Copyright © 1999-2001 Volano LLC. All rights reserved.
Contact John Neffenger with questions or comments.