The Volano Report

Last updated on May 31, 1999. The most recent Volano Report is available here.

The tests presented in this report look at the performance and network scalability of 11 Java virtual machines on six operating systems using a common Intel hardware platform running VolanoMark^™ 2.1. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of active socket connections:

Is it fast?
Does it scale?

When run at high connection counts, VolanoMark essentially measures the consequences of omitting non-blocking input/output operations from the Java platform, as summarized by this Java Developer Connection request for enhancement (free registration required):

Bug Id 4075058, "java.io: Add support for non-blocking I/O"

For background information and download locations, please see the Java Benchmarks page on Volano's Web site.

Is it fast?

Run rules

The performance test was executed with the following commands on a local loopback connection, using the heap size options shown below where possible:

Server: java -ms8m -mx64m COM.volano.Main
Client: java -ms8m -mx64m COM.volano.Mark -count 100

See the COM.volano.Mark command synopsis for a complete description of all options.

The operating system was rebooted before each set of tests for a particular Java virtual machine, and the first test result was discarded. The server side was restarted before each run of the client benchmark. The final score is the average of the best two out of three subsequent results.

Results

Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

Java Platform	Score
Tower TowerJ 3.0.1 Linux IBM JDK 1.1.7 OS/2 IBM JDK 1.1.7 Windows Sun HotSpot 1.0 Windows Microsoft SDK 3.2 Windows Sun JDK 1.2.1 Windows Novell JDK 1.1.5 NetWare Sun JDK 1.2 Solaris Transvirtual Kaffe 1.0b4 Linux Blackdown JDK 1.1.7 Linux JDK 1.1.7 FreeBSD	1899 1841 1660 1435 1400 1346 1218 883 379 264 172

Figure 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second.

Java Platform	Operating System	Results	Score
Tower TowerJ 3.0.1 Linux	Red Hat Linux 6.0 Intel	1897, 1900, 1893	1899
IBM JDK 1.1.7 OS/2	OS/2 Warp Server for e-business	1849, 1777, 1833	1841
IBM JDK 1.1.7 Windows	Windows NT Workstation 4.0	1652, 1664, 1656	1660
Sun HotSpot 1.0 Windows	Windows NT Workstation 4.0	1398, 1444, 1425	1435
Microsoft SDK 3.2 Windows	Windows NT Workstation 4.0	1408, 1391, 1388	1400
Sun JDK 1.2.1 Windows	Windows NT Workstation 4.0	1344, 1344, 1348	1346
Novell JDK 1.1.5 NetWare	NetWare 5	1216, 1216, 1220^a	1218
Sun JDK 1.2 Solaris	Solaris 7 Desktop Intel Platform Edition	881, 885, 827	883
Transvirtual Kaffe 1.0b4 Linux	Red Hat Linux 6.0 Intel	378, 380, 371	379
Blackdown JDK 1.1.7 Linux	Red Hat Linux 6.0 Intel	264, 264, 262	264
JDK 1.1.7 FreeBSD	FreeBSD 3.1-RELEASE	171, 172, 171	172

Table 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second. The final score is the average of the best 2 out of 3 results. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Notes:

Received "java.net.SocketException: Bad file number" 36 times on the third run, but the test completed successfully.

Does it scale?

Run rules

The network scalability test was executed with the following commands over an isolated 10-Mbps Ethernet connection with a 10-Mbps hub, using the heap and stack size options shown below where possible:

Server: java -ms8m -mx128m -ss32k COM.volano.Main
Client: java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 15
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 30
...
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 105

See the COM.volano.Mark command synopsis for a complete description of all options.

The client test driver was executed under the IBM JDK 1.1.7 Java virtual machine using Windows NT 4.0 on a 200-MHz Intel Pentium Pro with a 256-kilobyte L2 cache and 256 megabytes of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs except where noted.

Results

Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

300

600

900

1200

1500

1800

2100

Tower TowerJ 3.0.1 Linux

IBM JDK 1.1.7 OS/2

IBM JDK 1.1.7 Windows

Sun HotSpot 1.0 Windows

Microsoft SDK 3.2 Windows

Sun JDK 1.2.1 Windows

Novell JDK 1.1.5 NetWare

Sun JDK 1.2 Solaris

Transvirtual Kaffe 1.0b4 Linux

Blackdown JDK 1.1.7 Linux

JDK 1.1.7 FreeBSD

Figure 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections.

Java Platform	300	600	900	1200	1500	1800	2100	Limit
Tower TowerJ 3.0.1 Linux	2661	2387	2206	----^h	----	----	----	1014
IBM JDK 1.1.7 OS/2	2157	1901	473	----^a	----	----	----	1102
IBM JDK 1.1.7 Windows	2535	2181	1753	1499	1452	755^c	498^c	~ 3600
Sun HotSpot 1.0 Windows	2260	2081	----^a	----	----	----	----	899
Microsoft SDK 3.2 Windows	2367	1927	1529	1203	973	697	552	~ 3600
Sun JDK 1.2.1 Windows	2404	1788	1516	----^a	----	----	----	922
Novell JDK 1.1.5 NetWare	2487	2014	1844	1293	1505	----^d	----	~ 1500
Sun JDK 1.2 Solaris	1612	1513	1448	1385	1321	1248	1184	~ 4000
Transvirtual Kaffe 1.0b4 Linux	897	732	600	----^e	----	----	----	1016
Blackdown JDK 1.1.7 Linux	324	172	149	----^f	----	----	----	1016
JDK 1.1.7 FreeBSD	296	189	122	----^g	----	----	----	1023

Table 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware. The Limit is the maximum number of simultaneous VolanoMark connections possible (shown as an estimate when preceded by "~").

Notes:

Failed with "java.lang.OutOfMemoryError".
Failed with "java.net.SocketException: accept: Invalid argument".
Failed on the first try at 1800 and 2100 connections with "java.net.SocketException: Socket read failed: 10055" and "java.net.SocketException: Connection shutdown". Restarted the server and it worked on the second try.
Took too long to complete at 1800 connections (thrashing).
Failed with "java.io.IOException: Too many open files".
Failed with "java.net.SocketException: Too many open files".
Failed with "java.net.SocketException: Too many open files in system".
Failed with "java.net.SocketException: accept: Too many open files".

Solaris Intel vs. SPARC

The following tests compare VolanoMark 2.1.2, JDK 1.2, and Solaris 7 on two different hardware architectures:

a 200 MHz Intel Pentium Pro processor with 256 MB of RAM, and
a 200 MHz Sun UltraSPARC processor with 256 MB of RAM.

Java Platform	Score
Sun JDK 1.2 Solaris SPARC Sun JDK 1.2 Solaris Intel	1653 883

Figure 4. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second.

Java Platform	Operating System	Results	Score
Sun JDK 1.2 Solaris SPARC	Solaris 7 Desktop SPARC Platform Edition	1473, 1613, 1693	1653
Sun JDK 1.2 Solaris Intel	Solaris 7 Desktop Intel Platform Edition	881, 885, 827	883

Table 4. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second. The final score is the average of the best 2 out of 3 results.

300

600

900

1200

1500

1800

2100

Sun JDK 1.2 Solaris SPARC

Sun JDK 1.2 Solaris Intel

Figure 5. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections.

Java Platform	300	600	900	1200	1500	1800	2100	Limit
Sun JDK 1.2 Solaris SPARC	2631	2311	2130	1894	1780	1628	1488	~ 4000
Sun JDK 1.2 Solaris Intel	1612	1513	1448	1385	1321	1248	1184	~ 4000

Table 5. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections. The Limit is the maximum number of simultaneous VolanoMark connections possible (shown as an estimate when preceded by "~").

Environments

All tests ran identical copies of VolanoMark 2.1.2 on identical hardware—a 200-MHz Intel Pentium Pro processor with a 256-kilobyte L2 cache and 256 megabytes of RAM.

Tower TowerJ 3.0.1 Linux

10 May 1999—Tower Technology TowerJ Compiler 3.0.1.0
Red Hat Linux Intel 6.0 (Linux 2.2.5, glibc 2.1.1)
TowerJ Compiler "version 3.0.1.0 x86-linux"
Installed from TowerJ_3_0_1_0_x86_linux.class (13,116,025 bytes).
Uses user-level threads and static native pre-compilation.
Used the experimental command line option -Dtowerj.poll to increase the network scalability on Linux.
Built executable with TowerJ project files Main.tj and Mark.tj.
Did not increase the per-process file descriptor hard limit beyond its default of 1024 since it requires rebuilding the Linux 2.2 kernel.

IBM JDK 1.1.7 OS/2

06 May 1999—IBM OS/2 Warp Developer Kit, Java Edition, Version 1.1.7
IBM OS/2 Warp Server for e-business (Internal revision 14.039F_UNI)
java full version "JDK 1.1.7A IBM build o117-19990506 (JIT enabled: javax)"
Installed from the runtime.exe update (5,540,519 bytes) on top of the Java runtime shipped with OS/2 Warp Server for e-business.
Uses native threads and javax just-in-time compiler.
Modified the CONFIG.SYS file to increase the thread limit from 1024 to 4095 (THREADS=4095) and the initial swap file size from 2 MB to 32 MB (SWAPPATH=C:\OS2\SYSTEM 2048 32768).

IBM JDK 1.1.7 Windows

28 January 1999—IBM Win32 Java Virtual Machine Version 1.1.7
Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
java full version "JDK 1.1.7 IBM build n117p-19990128 (JIT enabled: ibmjitc)"
Installed from ibm117gm.exe (10,132,439 bytes).
Uses native threads and ibmjitc just-in-time compiler.

Sun HotSpot 1.0 Windows

30 April 1999—Sun HotSpot VM 1.0 FCS
Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
Sun JDK 1.2.1 base with HotSpot VM (1.0fcs, mixed mode, build E).
Installed from hotspot1_0-win.zip (1,622,666 bytes).
Uses native threads and an adaptive compiler.
The heap options must be preceded by the capital letter "X", and the stack options are not available.

Microsoft SDK 3.2 Windows

05 April 1999—Microsoft SDK 3.2 Windows Build 3177
Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
jview version 5.00.3177
Installed from MSJavx86.exe (6,631,656 bytes) and ClassR.exe (1,593,064 bytes).
Uses native threads and just-in-time compiler.
Heap and stack command line options are not available.

Sun JDK 1.2.1 Windows

27 March 1999—Sun JDK 1.2.1 Windows 95/98/NT Production Release
Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
java full version "JDK-1.2.1-A"
Installed from jdk1_2_1-win.exe (20,514,654 bytes).
Uses native threads and the symcjit Symantec Just-in-Time compiler.
The heap options must be preceded by the capital letter "X", and the stack options are not available.

Novell JDK 1.1.5 NetWare

25 January 1999—Novell JVM for NetWare
Novell NetWare Version 5.00
java full version "1.1.5" (JVM dated 01/08/99)
Installed from jvm.exe (18,107,065 bytes).
Uses native threads and symcjit just-in-time compiler (Symantec Java! JustInTime Compiler Version 3.00.040(x) for JDK 1.1.x).
Set Maximum Packet Receive Buffers = 1000 (default is 500).
Modified the java.cfg file to enable the just-in-time compiler (JAVA_COMPILER=symcjit).

Sun JDK 1.2 Solaris

26 February 1999—Sun JDK 1.2 Solaris Production Release
Sun Solaris 7 Desktop Intel Platform Edition with patches 106981-04 and 107079-02.
java full version "Solaris_JDK_1.2_01"
Installed from Solaris_JDK_1.2_01_i386.tar.Z (26,165,449 bytes).
Uses native threads and sunwjit just-in-time compiler.
The heap and stack options must be preceded by the capital letter "X".
Increased the per-process file descriptor hard limit to 4096 by setting the rlim_fd_max variable in /etc/system.

Transvirtual Kaffe 1.0b4 Linux—beta Java VM

09 April 1999—Transvirtual Technologies Kaffe OpenVM 1.0 Beta 4
Red Hat Linux Intel 6.0 (Linux 2.2.5, glibc 2.1.1)
Kaffe Virtual Machine "Engine: Just-in-time Version: 1.0b4 Java Version: 1.1"
Installed from the kaffe-1.0.b4-2 package shipped with Red Hat Linux 6.0.
Uses jthreads user-level threads and kaffe.jit just-in-time compiler.
Did not increase the per-process file descriptor hard limit beyond its default of 1024 since it requires rebuilding the Linux 2.2 kernel.

Blackdown JDK 1.1.7 Linux

04 November 1998—Blackdown JDK 1.1.7 Linux
Red Hat Linux Intel 6.0 (Linux 2.2.5, glibc 2.1.1)
java full version "Linux_JDK_1.1.7B_v3_green_threads"
Installed from jdk_1.1.7-v3-glibc-x86.tar.bz2 (10,130,082 bytes).
Uses user-level threads and no just-in-time compiler.
Did not increase the per-process file descriptor hard limit beyond its default of 1024 since it requires rebuilding the Linux 2.2 kernel.

JDK 1.1.7 FreeBSD

21 December 1998—JDK 1.1.7 for FreeBSD
FreeBSD 3.1-RELEASE
java full version "jdk1.1.7-FreeBSD:1998/12/21"
Installed from jdk1.1.7.V98-12-21.tar.gz (12,920,606 bytes).
Uses user-level threads and no just-in-time compiler.
Did not increase the per-process file descriptor hard limit beyond its default of 1024 since the performance at high connection levels did not justify running the tests.