The Volano Report

The tests presented in this report look at the performance and network scalability of 11 Java virtual machines on six operating systems using a common Intel hardware platform running VolanoMark 2.1. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of active socket connections:

  1. Is it fast?
  2. Does it scale?

When run at high connection counts, VolanoMark essentially measures the consequences of omitting non-blocking input/output operations from the Java platform, as summarized by this Java Developer Connection request for enhancement (free registration required):

For background information and download locations, please see the Java Benchmarks page on Volano's Web site.

Is it fast?

Run rules

The performance test was executed with the following commands on a local loopback connection, using the heap size options shown below where possible:

Server
java -ms8m -mx64m COM.volano.Main
Client
java -ms8m -mx64m COM.volano.Mark -count 100
See the COM.volano.Mark command synopsis for a complete description of all options.

The operating system was rebooted before each set of tests for a particular Java virtual machine, and the first test result was discarded. The server side was restarted before each run of the client benchmark. The final score is the average of the best two out of three subsequent results.

Results

Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

Java Platform Score
Tower TowerJ 3.0.1 Linux
IBM JDK 1.1.7 OS/2
IBM JDK 1.1.7 Windows
Sun HotSpot 1.0 Windows
Microsoft SDK 3.2 Windows
Sun JDK 1.2.1 Windows
Novell JDK 1.1.5 NetWare
Sun JDK 1.2 Solaris
Transvirtual Kaffe 1.0b4 Linux
Blackdown JDK 1.1.7 Linux
JDK 1.1.7 FreeBSD
1899 1899
1841 1841
1660 1660
1435 1435
1400 1400
1346 1346
1218 1218
883 883
379 379
264 264
172 172
Figure 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second.

Java Platform Operating System Results Score
Tower TowerJ 3.0.1 Linux Red Hat Linux 6.0 Intel 1897, 1900, 1893 1899
IBM JDK 1.1.7 OS/2 OS/2 Warp Server for e-business 1849, 1777, 1833 1841
IBM JDK 1.1.7 Windows Windows NT Workstation 4.0 1652, 1664, 1656 1660
Sun HotSpot 1.0 Windows Windows NT Workstation 4.0 1398, 1444, 1425 1435
Microsoft SDK 3.2 Windows Windows NT Workstation 4.0 1408, 1391, 1388 1400
Sun JDK 1.2.1 Windows Windows NT Workstation 4.0 1344, 1344, 1348 1346
Novell JDK 1.1.5 NetWare NetWare 5 1216, 1216, 1220a 1218
Sun JDK 1.2 Solaris Solaris 7 Desktop Intel Platform Edition 881, 885, 827 883
Transvirtual Kaffe 1.0b4 Linux Red Hat Linux 6.0 Intel 378, 380, 371 379
Blackdown JDK 1.1.7 Linux Red Hat Linux 6.0 Intel 264, 264, 262 264
JDK 1.1.7 FreeBSD FreeBSD 3.1-RELEASE 171, 172, 171 172
Table 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second. The final score is the average of the best 2 out of 3 results. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Notes:

  1. Received "java.net.SocketException: Bad file number" 36 times on the third run, but the test completed successfully.

Does it scale?

Run rules

The network scalability test was executed with the following commands over an isolated 10-Mbps Ethernet connection with a 10-Mbps hub, using the heap and stack size options shown below where possible:

Server
java -ms8m -mx128m -ss32k COM.volano.Main
Client
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 15
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 30
...
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 105

See the COM.volano.Mark command synopsis for a complete description of all options.

The client test driver was executed under the IBM JDK 1.1.7 Java virtual machine using Windows NT 4.0 on a 200-MHz Intel Pentium Pro with a 256-kilobyte L2 cache and 256 megabytes of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs except where noted.

Results

Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

2661 2157 2535 2260 2367 2404 2487 1612 897 324 296 2387 1901 2181 2081 1927 1788 2014 1513 732 172 189 2206 473 1753 0 1529 1516 1844 1448 600 149 122 0 0 1499 0 1203 0 1293 1385 0 0 0 0 0 1452 0 973 0 1505 1321 0 0 0 0 0 755 0 697 0 0 1248 0 0 0 0 0 498 0 552 0 0 1184 0 0 0
300 600 900 1200 1500 1800 2100
Tower TowerJ 3.0.1 Linux
IBM JDK 1.1.7 OS/2
IBM JDK 1.1.7 Windows
Sun HotSpot 1.0 Windows
Microsoft SDK 3.2 Windows
Sun JDK 1.2.1 Windows
Novell JDK 1.1.5 NetWare
Sun JDK 1.2 Solaris
Transvirtual Kaffe 1.0b4 Linux
Blackdown JDK 1.1.7 Linux
JDK 1.1.7 FreeBSD
Figure 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections.

Java Platform 300 600 900 1200 1500 1800 2100 Limit
Tower TowerJ 3.0.1 Linux 2661 2387 2206 ----h ---- ---- ---- 1014
IBM JDK 1.1.7 OS/2 2157 1901 473 ----a ---- ---- ---- 1102
IBM JDK 1.1.7 Windows 2535 2181 1753 1499 1452 755c 498c ~ 3600
Sun HotSpot 1.0 Windows 2260 2081 ----a ---- ---- ---- ---- 899
Microsoft SDK 3.2 Windows 2367 1927 1529 1203 973 697 552 ~ 3600
Sun JDK 1.2.1 Windows 2404 1788 1516 ----a ---- ---- ---- 922
Novell JDK 1.1.5 NetWare 2487 2014 1844 1293 1505 ----d ---- ~ 1500
Sun JDK 1.2 Solaris 1612 1513 1448 1385 1321 1248 1184 ~ 4000
Transvirtual Kaffe 1.0b4 Linux 897 732 600 ----e ---- ---- ---- 1016
Blackdown JDK 1.1.7 Linux 324 172 149 ----f ---- ---- ---- 1016
JDK 1.1.7 FreeBSD 296 189 122 ----g ---- ---- ---- 1023
Table 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware. The Limit is the maximum number of simultaneous VolanoMark connections possible (shown as an estimate when preceded by "~").

Notes:

  1. Failed with "java.lang.OutOfMemoryError".
  2. Failed with "java.net.SocketException: accept: Invalid argument".
  3. Failed on the first try at 1800 and 2100 connections with "java.net.SocketException: Socket read failed: 10055" and "java.net.SocketException: Connection shutdown". Restarted the server and it worked on the second try.
  4. Took too long to complete at 1800 connections (thrashing).
  5. Failed with "java.io.IOException: Too many open files".
  6. Failed with "java.net.SocketException: Too many open files".
  7. Failed with "java.net.SocketException: Too many open files in system".
  8. Failed with "java.net.SocketException: accept: Too many open files".

Solaris Intel vs. SPARC

The following tests compare VolanoMark 2.1.2, JDK 1.2, and Solaris 7 on two different hardware architectures:

Java Platform Score
Sun JDK 1.2 Solaris SPARC
Sun JDK 1.2 Solaris Intel
1653 1653
883 883
Figure 4. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second.

Java Platform Operating System Results Score
Sun JDK 1.2 Solaris SPARC Solaris 7 Desktop SPARC Platform Edition 1473, 1613, 1693 1653
Sun JDK 1.2 Solaris Intel Solaris 7 Desktop Intel Platform Edition 881, 885, 827 883
Table 4. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second. The final score is the average of the best 2 out of 3 results.

2631 1612 2311 1513 2130 1448 1894 1385 1780 1321 1628 1248 1488 1184
300 600 900 1200 1500 1800 2100
Sun JDK 1.2 Solaris SPARC
Sun JDK 1.2 Solaris Intel
Figure 5. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections.

Java Platform 300 600 900 1200 1500 1800 2100 Limit
Sun JDK 1.2 Solaris SPARC 2631 2311 2130 1894 1780 1628 1488 ~ 4000
Sun JDK 1.2 Solaris Intel 1612 1513 1448 1385 1321 1248 1184 ~ 4000
Table 5. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections. The Limit is the maximum number of simultaneous VolanoMark connections possible (shown as an estimate when preceded by "~").

Environments

All tests ran identical copies of VolanoMark 2.1.2 on identical hardware—a 200-MHz Intel Pentium Pro processor with a 256-kilobyte L2 cache and 256 megabytes of RAM.

Tower TowerJ 3.0.1 Linux
  • 10 May 1999—Tower Technology TowerJ Compiler 3.0.1.0
  • Red Hat Linux Intel 6.0 (Linux 2.2.5, glibc 2.1.1)
  • TowerJ Compiler "version 3.0.1.0 x86-linux"
  • Installed from TowerJ_3_0_1_0_x86_linux.class (13,116,025 bytes).
  • Uses user-level threads and static native pre-compilation.
  • Used the experimental command line option -Dtowerj.poll to increase the network scalability on Linux.
  • Built executable with TowerJ project files Main.tj and Mark.tj.
  • Did not increase the per-process file descriptor hard limit beyond its default of 1024 since it requires rebuilding the Linux 2.2 kernel.
IBM JDK 1.1.7 OS/2
  • 06 May 1999—IBM OS/2 Warp Developer Kit, Java Edition, Version 1.1.7
  • IBM OS/2 Warp Server for e-business (Internal revision 14.039F_UNI)
  • java full version "JDK 1.1.7A IBM build o117-19990506 (JIT enabled: javax)"
  • Installed from the runtime.exe update (5,540,519 bytes) on top of the Java runtime shipped with OS/2 Warp Server for e-business.
  • Uses native threads and javax just-in-time compiler.
  • Modified the CONFIG.SYS file to increase the thread limit from 1024 to 4095 (THREADS=4095) and the initial swap file size from 2 MB to 32 MB (SWAPPATH=C:\OS2\SYSTEM 2048 32768).
IBM JDK 1.1.7 Windows
  • 28 January 1999—IBM Win32 Java Virtual Machine Version 1.1.7
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
  • java full version "JDK 1.1.7 IBM build n117p-19990128 (JIT enabled: ibmjitc)"
  • Installed from ibm117gm.exe (10,132,439 bytes).
  • Uses native threads and ibmjitc just-in-time compiler.
Sun HotSpot 1.0 Windows
  • 30 April 1999—Sun HotSpot VM 1.0 FCS
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
  • Sun JDK 1.2.1 base with HotSpot VM (1.0fcs, mixed mode, build E).
  • Installed from hotspot1_0-win.zip (1,622,666 bytes).
  • Uses native threads and an adaptive compiler.
  • The heap options must be preceded by the capital letter "X", and the stack options are not available.
Microsoft SDK 3.2 Windows
  • 05 April 1999—Microsoft SDK 3.2 Windows Build 3177
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
  • jview version 5.00.3177
  • Installed from MSJavx86.exe (6,631,656 bytes) and ClassR.exe (1,593,064 bytes).
  • Uses native threads and just-in-time compiler.
  • Heap and stack command line options are not available.
Sun JDK 1.2.1 Windows
  • 27 March 1999—Sun JDK 1.2.1 Windows 95/98/NT Production Release
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
  • java full version "JDK-1.2.1-A"
  • Installed from jdk1_2_1-win.exe (20,514,654 bytes).
  • Uses native threads and the symcjit Symantec Just-in-Time compiler.
  • The heap options must be preceded by the capital letter "X", and the stack options are not available.
Novell JDK 1.1.5 NetWare
  • 25 January 1999—Novell JVM for NetWare
  • Novell NetWare Version 5.00
  • java full version "1.1.5" (JVM dated 01/08/99)
  • Installed from jvm.exe (18,107,065 bytes).
  • Uses native threads and symcjit just-in-time compiler (Symantec Java! JustInTime Compiler Version 3.00.040(x) for JDK 1.1.x).
  • Set Maximum Packet Receive Buffers = 1000 (default is 500).
  • Modified the java.cfg file to enable the just-in-time compiler (JAVA_COMPILER=symcjit).
Sun JDK 1.2 Solaris
  • 26 February 1999—Sun JDK 1.2 Solaris Production Release
  • Sun Solaris 7 Desktop Intel Platform Edition with patches 106981-04 and 107079-02.
  • java full version "Solaris_JDK_1.2_01"
  • Installed from Solaris_JDK_1.2_01_i386.tar.Z (26,165,449 bytes).
  • Uses native threads and sunwjit just-in-time compiler.
  • The heap and stack options must be preceded by the capital letter "X".
  • Increased the per-process file descriptor hard limit to 4096 by setting the rlim_fd_max variable in /etc/system.
Transvirtual Kaffe 1.0b4 Linuxbeta Java VM
  • 09 April 1999—Transvirtual Technologies Kaffe OpenVM 1.0 Beta 4
  • Red Hat Linux Intel 6.0 (Linux 2.2.5, glibc 2.1.1)
  • Kaffe Virtual Machine "Engine: Just-in-time Version: 1.0b4 Java Version: 1.1"
  • Installed from the kaffe-1.0.b4-2 package shipped with Red Hat Linux 6.0.
  • Uses jthreads user-level threads and kaffe.jit just-in-time compiler.
  • Did not increase the per-process file descriptor hard limit beyond its default of 1024 since it requires rebuilding the Linux 2.2 kernel.
Blackdown JDK 1.1.7 Linux
  • 04 November 1998—Blackdown JDK 1.1.7 Linux
  • Red Hat Linux Intel 6.0 (Linux 2.2.5, glibc 2.1.1)
  • java full version "Linux_JDK_1.1.7B_v3_green_threads"
  • Installed from jdk_1.1.7-v3-glibc-x86.tar.bz2 (10,130,082 bytes).
  • Uses user-level threads and no just-in-time compiler.
  • Did not increase the per-process file descriptor hard limit beyond its default of 1024 since it requires rebuilding the Linux 2.2 kernel.
JDK 1.1.7 FreeBSD
  • 21 December 1998—JDK 1.1.7 for FreeBSD
  • FreeBSD 3.1-RELEASE
  • java full version "jdk1.1.7-FreeBSD:1998/12/21"
  • Installed from jdk1.1.7.V98-12-21.tar.gz (12,920,606 bytes).
  • Uses user-level threads and no just-in-time compiler.
  • Did not increase the per-process file descriptor hard limit beyond its default of 1024 since the performance at high connection levels did not justify running the tests.

Valid HTML 4.0! Valid CSS! Copyright © 1999-2001 Volano LLC. All rights reserved.
Contact John Neffenger with questions or comments.

This site contains articles by John Neffenger about the Java™ platform, the VOLANO® chat server benchmark (VolanoMark), and the business of developing and publishing software. More…

John Neffenger <john@status6.com>
More…

A PROJECT OF STATUS:6®