The Volano Report
- Last updated on March 24, 2000. The most recent Volano Report is available here.
What's new?
Much to my surprise, Appeal's JRockit has arrived on the scene to post the best network scalability scores of any Java virtual machine I've tested. In fact, it's the only Java virtual machine I've found which is able to support 4,000 connections on Windows NT or Windows 2000 without modification.
Appeal has made Java threads as lightweight as they're advertised to be, mapping their thin threads onto native threads as required. JRockit seems to be particularly suited for pure Java servers which must handle many dedicated network connections. Like TowerJ, though, it's not free. Let's hope for a Linux port of JRockit!
What is VolanoMark?
For background information and download locations, please see the Java Benchmarks page on Volano's Web site. Since the server side of the VolanoMark benchmark is our VolanoChat server, these test results help our customers choose the best Java virtual machine for their environment.
VolanoMark is a pure Java server benchmark characterized by long-lasting network connections and high thread counts. In this context, long-lasting means the connections last several minutes or longer, rather than just a few seconds. The VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the clients to take turns broadcasting their messages to the group. At the end of the test, it reports a score as the average number of messages transferred by the server per second.
Bug Id 4075058, "java.io: Add support for non-blocking I/O," explains why you need lots of threads in order to handle dedicated connections to a pure Java server. JSR 51, "New I/O APIs for the Java Platform," explains how Sun plans to fix that.
The tests
The tests presented in this report look at the performance and network scalability of 18 Java virtual machines on 7 operating systems using a common Intel hardware platform running VolanoMark™ 2.1. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of dedicated socket connections:
See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments, along with a table showing the compiler and threading model of each Java platform.
Is it fast?
Run rules
The performance test was executed with the following commands on a local loop-back connection, using the heap size options shown below where possible:
- Server
java -ms8m -mx64m COM.volano.Main- Client
java -ms8m -mx64m COM.volano.Mark -count 100
See the COM.volano.Mark command synopsis for a complete description of all options.
The operating system was rebooted before each set of tests for a particular Java virtual machine. The server side was not restarted before each run of the client benchmark. The client benchmark ran four times, with the final score as the average of the last three results.
Results
Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.
| Java Platform | Score |
|---|---|
|
Tower TowerJ 3.1.4 Linux (a) IBM JDK 1.1.8 Windows NT IBM JDK 1.1.8 OS/2 Microsoft VM 3234 Windows NT Microsoft VM 3229 Windows 2000 Appeal JRockit 1.0 Windows 2000 (a) IBM JDK 1.1.8 Linux Sun HotSpot Server 2.0 Windows NT Sun JDK 1.3 RC1 Windows NT Sun JDK 1.2.2-001 Windows NT Sun JDK 1.2.1_04 Solaris Novell JDK 1.1.7 NetWare (b) Sun/Inprise JDK 1.2.2 Linux Blackdown/Inprise JDK 1.2.2 Linux Blackdown JDK 1.2.2 RC4 Linux Transvirtual Kaffe 1.0.5 Linux (c) Blackdown JDK 1.1.7 Linux JDK 1.1.8 FreeBSD |
|
| Java Platform | Operating System | Results | Score |
|---|---|---|---|
| Red Hat Linux 6.0 Intel | 2303, 2313, 2313, 2300 | 2309 | |
| Windows NT Workstation 4.0 | 2244, 2267, 2267, 2263 | 2266 | |
| OS/2 Warp Server for e-business | 2145, 2141, 2118, 2150 | 2136 | |
| Windows NT Workstation 4.0 | 2034, 2040, 2030, 2018 | 2029 | |
| Windows 2000 Server Release Candidate 2 | 1948, 1965, 1976, 1968 | 1970 | |
| Windows 2000 Server | 1910, 1926, 1925, 1931 | 1927 | |
| Red Hat Linux 6.0 Intel | 1317, 1823, 1766, 1752 | 1780 | |
| Windows NT Workstation 4.0 | 1638, 1664, 1668, 1675 | 1669 | |
| Windows NT Workstation 4.0 | 1570, 1593, 1578, 1605 | 1592 | |
| Windows NT Workstation 4.0 | 1477, 1481, 1479, 1471 | 1477 | |
| Solaris 7 Desktop Intel Platform Edition | 1409, 1408, 1400, 1389 | 1399 | |
| NetWare 5 | 1227, 1232, 1229, 1226 | 1229 | |
| Red Hat Linux 6.0 Intel | 1035, 1039, 1042, 1039 | 1040 | |
| Red Hat Linux 6.0 Intel | 1032, 1037, 1039, 1037 | 1038 | |
| Red Hat Linux 6.0 Intel | 951, 952, 953, 950 | 952 | |
| Red Hat Linux 6.0 Intel | 906, 905, 909, 909 | 908 | |
| Red Hat Linux 6.0 Intel | 284, 285, 286, 283 | 285 | |
| FreeBSD 3.2-RELEASE | 169, 173, 173, 172 | 173 |
Notes:
- TowerJ and JRockit are the only Java platforms listed here which are not available for free and are not intended for use with client Java applications.
- The JVM for NetWare reported "java.net.SocketException: Software caused connection abort" once and "java.net.SocketException: Socket closed" 42 times when the server side was killed.
- The Kaffe OpenVM failed on the first test run with "Kaffe: exception.c:308: dispatchException: Assertion `!intsDisabled()' failed."
Does it scale?
Run rules
The network scalability test was executed with the following commands over an isolated 10-Mbps Ethernet connection, using the heap and stack size options shown below where possible:
- Server
java -ms8m -mx128m -ss32k COM.volano.Main- Client
-
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 50
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 100
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 150
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 200
Note: IBM JDK 1.1.8 on Linux required an initial heap size of
64 MB (using the option -ms64m) in order to complete the
network scalability tests shown in Table 2 below, so you should judge
its relative performance instead by the loopback scores in Table 1
above.
See the COM.volano.Mark command synopsis for a complete description of all options.
The client test driver was executed under the Sun JDK 1.2.1_03 Production Release for Solaris using Solaris 7 on a dual 200-MHz Sun UltraSPARC with 256 megabytes of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs.
Results
Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.
|
|
|
|
|
|||
| 1000 | 2000 | 3000 | 4000 | |||
|---|---|---|---|---|---|---|
| ||||||
| Java Platform | 1000 | 2000 | 3000 | 4000 | Notes |
|---|---|---|---|---|---|
| 2486 | 1357 | 739 | 421 | No errors! | |
| 2571 | 1755 | 1072 | ---- | "java.lang.OutOfMemoryError" at 3508/4000 connections. | |
| 1521 | ---- | ---- | ---- | "java.lang.OutOfMemoryError" at 1098/2000 connections. | |
| 1695 | 979 | 535 | ---- | "java.lang.OutOfMemoryError" at 3796/4000 connections. | |
| 1408 | 708 | 406 | ---- | "java.lang.OutOfMemoryError" at 3785/4000 connections. | |
| 2354 | 1901 | 1237 | 688 | No errors! | |
| 722 | 556 | ---- | ---- | "java.lang.OutOfMemoryError: cannot create any more threads" at 2007/3000 connections. Required an initial heap size of 64 MB instead of 8 MB. | |
| ---- | ---- | ---- | ---- | "HotSpot Virtual Machine Error, EXCEPTION_ACCESS_VIOLATION, Error ID: 4F533F57494E13120E43505002D9" at 880/1000 connections. | |
| 1521 | 871 | 591 | ---- | "java.lang.OutOfMemoryError" at 3476/4000 connections. | |
| ---- | ---- | ---- | ---- | "java.lang.OutOfMemoryError" at 926/1000 connections. | |
| 1821 | 1060 | 695 | ---- | Hangs with disk thrashing at 3940/4000 connections. "Segmentation Fault - core dumped" when client test driver is killed to stop the test. | |
| 1575 | ---- | ---- | ---- | Hangs with disk thrashing at 1960/2000 connections. | |
| 1174 | ---- | ---- | ---- | "SIGSEGV 11* segmentation violation" at just over 1000 connections. See the second of the two bugs placed into Sun Bug Id 4298809. Only the first bug has been fixed. | |
| 1177 | 718 | 473 | 310 | No errors! | |
| 1141 | 666 | 452 | ---- | "java.lang.StackOverflowError" after a total of 6816 cumulative connections, causing it to fail 816 connections into the 4000-connection test. See Sun Bug Id 4299454. | |
| ---- | ---- | ---- | ---- | Hangs with zero percent CPU usage after starting, even at just 500 connections. Dumps core when the client side is killed to stop the test. | |
| 165 | ---- | ---- | ---- | Dumps core at just over 1000 connections. | |
| 37 | ---- | ---- | ---- | Client begins failing at 1017 connections with "java.net.NoRouteToHostException: Connection timed out". |
Environments
All tests ran identical copies of VolanoMark 2.1.2 on identical hardware -- a 200-MHz Intel Pentium Pro processor with a 256-kilobyte L2 cache and 256 megabytes of RAM on an Intel VS440FX motherboard running American Megatrends AMIBIOS Version 1.00.18.CS1 dated "08/28/98-13:59". The network card is a 3Com Etherlink III PCI Bus-Master Adapter Model 3C590.
- Tower TowerJ 3.1.4 Linux
-
- Tower Technology TowerJ High Performance Java Compiler Version 3.1.4.0
- Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
- TowerJ Compiler
"version 3.1.4.0 x86-linux" - Installed from
TowerJ_3_1_4_0_x86_linux.class(12,913,370 bytes). - Uses user-level threads and static native pre-compilation.
- Used the command line option
-Dtowerj.pollfor the network scalability test. - Built executable with TowerJ project files
Main.tjandMark.tj. - Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096). Increased the maximum tasks per user to 4090 (from 256) and the system-wide task limit to 4090 (from 512). Compiled the Linux kernel for the "PPro/6x86MX" processor family.
- IBM JDK 1.1.8 Windows NT
-
- IBM Developer Kit for Windows
- Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
- Java version
"JDK 1.1.8 IBM build n118p-20000108 (JIT enabled: ibmjitc V3.5-IBMJDK1.1-20000108)" - Installed from
ibm-jdk-n118p-win32-x86.zip(10,824,610 bytes). - Uses native threads and
ibmjitcjust-in-time compiler.
- IBM JDK 1.1.8 OS/2
-
- IBM OS/2 Warp Developer Kit, Java Technology Edition, Version 1.1.8
- IBM OS/2 Warp Server for e-business (Version 20.45, Internal revision
14.039F_UNI) - Java version
"JDK 1.1.8 IBM build o118-19990728 (JIT enabled: javax V3.5-IBMJDK1.1-19990728)" - Installed from
javainuf.exe(20,805,936 bytes) andjavaintk.exe(15,273,452 bytes). - Uses native threads and
javaxjust-in-time compiler. - Modified the
CONFIG.SYSfile to increase the thread limit from 1024 to 4095(THREADS=4095)and the initial swap file size from 2 MB to 32 MB(SWAPPATH=C:\OS2\SYSTEM 2048 32768).
- Microsoft VM 3234 Windows NT
-
- Microsoft Virtual Machine Build 3234
- Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
- Java version
"1.1.4" - jview version
5.00.3229 - msjava.dll version
5.0.3234.0 - Installed from
msjavx86.exe(5,449,664 bytes). - Uses native threads and just-in-time compiler.
- Heap and stack command line options are not available.
- Microsoft VM 3229 Windows 2000
-
- Microsoft Virtual Machine Build 3229
- Microsoft Windows 2000 Server Release Candidate 2 (Version 5.0.2128)
- Java version
"1.1.4" - jview version
5.00.3229 - Uses native threads and just-in-time compiler.
- Heap and stack command line options are not available.
- Appeal JRockit 1.0 Windows 2000
-
- Appeal Virtual Machines JRockit 1.0
- Microsoft Windows 2000 Server Build 2195
- Java version
"1.2" - JRockit version
"JRockit JVM 1.2 version 1.0 build excelsior45" - Uses thin threads and an adaptive compiler.
- Used the additional options
-Xgc:scand-Xexpandstacksas recommend by Appeal. Did not use the-Xss32kstack size option on the network test, since-Xexpandstackswas used instead.
- IBM JDK 1.1.8 Linux
-
- IBM Developer Kit and Runtime Environment for Linux, Java Technology Edition, Version 1.1.8
- Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
- Java version
"JDK 1.1.8 IBM build l118-19991221 (JIT enabled: jitc)" - Installed from
ibm-jdk-l118-linux-x86.tgz(10,560,355 bytes). - Uses native threads and
jitcjust-in-time compiler. - Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096). Increased the maximum tasks per user to 4090 (from 256) and the system-wide task limit to 4090 (from 512). Compiled the Linux kernel for the "PPro/6x86MX" processor family.
- Ran the network scalability test with an initial heap size of 64 MB instead of 8 MB in order for it to work. With an initial heap size of 8 MB, the performance was too poor to complete the 2,000 connection test without having the clients receive timeout errors.
- Sun HotSpot Server 2.0 Windows NT
-
- Sun Java HotSpot Server VM Version 2.0 Beta for Win32 Platforms
- Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
- Sun JDK base version
"Classic VM (build JDK-1.2.2-001, native threads, symcjit)" - Sun HotSpot version
"Java HotSpot(TM) Server VM (2.0rc1, mixed mode, build I)" - Installed from
jdk1_2_2-001-win.exe(20,449,862 bytes) andhotspot2_0rc1-win.exe(1,305,731 bytes). - Uses native threads and an adaptive compiler.
- The heap options must be preceded by the capital letter
"X".
- Sun JDK 1.3 RC1 Windows NT
-
- Sun Java 2 Software Development Kit, Standard Edition, v 1.3.0 (J2SE) Release Candidate 1 (RC1)
- Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
- Java version
"1.3.0rc1, Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0rc1-T), Java HotSpot(TM) Client VM (build 1.3.0rc1-S, mixed mode)" - Installed from
j2sdk1_3_0rc1-win.exe(30,847,507 bytes). - Uses native threads and an adaptive compiler.
- The heap and stack options must be preceded by the capital letter
"X".
- Sun JDK 1.2.2-001 Windows NT
-
- Sun Java 2 SDK, Standard Edition Version 1.2.2-001 for Microsoft Windows Production Release
- Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
- Java version
"Classic VM (build JDK-1.2.2-001, native threads, symcjit)" - Installed from
jdk1_2_2-001-win.exe(20,449,862 bytes). - Uses native threads and the
symcjitSymantec just-in-time compiler. - The heap and stack options must be preceded by the capital letter
"X".
- Sun JDK 1.2.1_04 Solaris
-
- Sun J2SE: Java 2 SDK, Standard Edition (1.2.1_04 Localized) Production Release for Solaris
- Sun Solaris 7 Desktop Intel Platform Edition with patches 106981-07, 107079-17, 107637-01, and 107082-08 (which obsoletes 107608-01).
- Java version
"Solaris VM (build Solaris_JDK_1.2.1_04, native threads, sunwjit)" - Installed from
Solaris_JDK_1.2.1_04_i386.bin(18,591,552 bytes) and1.2.1_04_patches_i386_5.7.tar(4,011,008 bytes). - Uses native threads and
sunwjitjust-in-time compiler. - The heap and stack options must be preceded by the capital letter
"X". - Increased the per-process file descriptor limit to 8192 by setting
the
rlim_fd_maxvariable in/etc/system.
- Novell JDK 1.1.7 NetWare
-
- Novell JVM for NetWare
- Novell NetWare 5 Support Pack 2 Version 5.00c dated April 23, 1999
- Java version
"1.1.7 B" - Installed from
jvm.exe(18,015,477 bytes). - Uses native threads and
symcjitjust-in-time compiler (Symantec Java! JustInTime Compiler Version 3.10.106(x) for JDK 1.1.x). - Set Maximum Packet Receive Buffers = 1000 (from default of 500).
- Modified the
java.cfgfile to enable the just-in-time compiler(JAVA_COMPILER=symcjit).
- Sun/Inprise JDK 1.2.2 Linux
-
- Java 2 SDK, Standard Edition Version 1.2.2 for Linux Production Release with Inprise just-in-time compiler
- Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
- Java version
"1.2.2", "Classic VM (build 1.2.2-L, green threads, javacomp)" - Installed from
jdk1_2_2-linux-i386.tar.gz(21,580,986 bytes) andjavacomp-1.2.13-glibc2.1.tar.gz(202,058 bytes). - Uses green threads and
javacompjust-in-time compiler. - The heap and stack options must be preceded by the capital letter
"X". - Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096).
- Blackdown/Inprise JDK 1.2.2 Linux
-
- Blackdown Java Platform 2 SDK Version 1.2.2 with Inprise just-in-time compiler
- Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
- Note that Blackdown requires glibc 2.1.2 for native threads, but VolanoMark uses green threads so it works with glibc 2.1.1.
- Java version
"1.2.2", "Classic VM (build Linux_JDK_1.2.2_RC4, green threads, javacomp)" - Installed from
jdk-1.2.2-RC4-linux-i386-glibc-2.1.2.sh(22,485,129 bytes) andlibjavacomp.so(118,468 bytes). - Uses native or green threads and
javacompjust-in-time compiler. - Used the
"-green"option for user-level threads and the"-Djava.compiler=javacomp"option for the Inprise just-in-time compiler. - The heap and stack options must be preceded by the capital letter
"X". - Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096).
- Blackdown JDK 1.2.2 RC4 Linux
-
- Blackdown Java Platform 2 SDK Version 1.2.2
- Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
- Note that Blackdown requires glibc 2.1.2 for native threads, but VolanoMark uses green threads so it works with glibc 2.1.1.
- Java version
"1.2.2", "Classic VM (build Linux_JDK_1.2.2_RC4, green threads, sunwjit)" - Installed from
jdk-1.2.2-RC4-linux-i386-glibc-2.1.2.sh(22,485,129 bytes). - Uses native or green threads and
sunwjitjust-in-time compiler. - Used the
"-green"option for user-level threads. - The heap and stack options must be preceded by the capital letter
"X". - Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096).
- Transvirtual Kaffe 1.0.5 Linux
-
- Transvirtual Technologies Kaffe OpenVM Version 1.0.5
- Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
- Kaffe Virtual Machine
"Engine: Just-in-time v3 Version: 1.0.5 Java Version: 1.1" - Installed from the
kaffe-1.0.5.tar.gz(3,145,844 bytes). - Uses user-level threads and
kaffe.jitjust-in-time compiler. - Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096). Increased the maximum tasks per user to 4090 (from 256) and the system-wide task limit to 4090 (from 512). Compiled the Linux kernel for the "PPro/6x86MX" processor family.
- Blackdown JDK 1.1.7 Linux
-
- Blackdown Java-Linux port of JDK 1.1.7
- Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
- Java version
"Linux_JDK_1.1.7B_v3_green_threads" - Installed from
jdk_1.1.7-v3-glibc-x86.tar.bz2(10,130,082 bytes). - Uses user-level threads and no just-in-time compiler.
- Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096). Increased the maximum tasks per user to 4090 (from 256) and the system-wide task limit to 4090 (from 512). Compiled the Linux kernel for the "PPro/6x86MX" processor family.
- JDK 1.1.8 FreeBSD
-
- JDK 1.1.8 for FreeBSD
- FreeBSD 3.2-RELEASE
- Java version
"jdk1.1.8-FreeBSD:1999/7/19" - Installed from
jdk1.1.8_ELF.V99-7-19.tar.gz(11,337,773 bytes). - Uses user-level threads and no just-in-time compiler.
- Increased the per-process file descriptor limit to 4096 (from 1064)
and the system-wide file descriptor limit to 8192 (from 1064) by using
sysctlto modify thekern.maxfilesperprocandkern.maxfilesvariables in/etc/rc.local.
| Java Platform | Operating System | Compiler | Thread Model |
|---|---|---|---|
| Red Hat Linux 6.0 Intel | static | many-to-one | |
| Windows NT Workstation 4.0 | mixed mode | one-to-one | |
| OS/2 Warp Server for e-business | mixed mode | one-to-one | |
| Windows NT Workstation 4.0 | JIT | one-to-one | |
| Windows 2000 Server Release Candidate 2 | JIT | one-to-one | |
| Windows 2000 Server | mixed mode | many-to-many | |
| Red Hat Linux 6.0 Intel | JIT | one-to-one | |
| Windows NT Workstation 4.0 | mixed mode | one-to-one | |
| Windows NT Workstation 4.0 | mixed mode | one-to-one | |
| Windows NT Workstation 4.0 | JIT | one-to-one | |
| Solaris 7 Desktop Intel Platform Edition | JIT | many-to-many | |
| NetWare 5 | JIT | many-to-one | |
| Red Hat Linux 6.0 Intel | JIT | many-to-one | |
| Red Hat Linux 6.0 Intel | JIT | one-to-one or many-to-one | |
| Red Hat Linux 6.0 Intel | JIT | one-to-one or many-to-one | |
| Red Hat Linux 6.0 Intel | JIT | many-to-one | |
| Red Hat Linux 6.0 Intel | none | many-to-one | |
| FreeBSD 3.2-RELEASE | none | many-to-one |
Compiler
- static
- A static compiler which takes Java source files or class files and translates them into a native executable or shared library before they run.
- mixed mode
- A Java VM which translates some of the Java bytecodes into native instructions on the fly while the Java program runs, while letting a Java interpreter handle other parts of the program. Specifically, I use this term to refer to adaptive compilers which use runtime execution profiles to optimize the resulting native instructions.
- JIT
- A Java VM which translates the Java bytecodes into native instructions on the fly before executing them.
- none
- A Java VM which uses a normal Java bytecode interpreter.
Thread Model
- one-to-one
- Each Java thread is mapped one-to-one onto its own operating system thread.
- many-to-one
- All of the Java threads in the process are mapped onto one operating system process.
- many-to-many
- More than one Java thread is mapped to one operating system thread, and a Java thread may be mapped to different operating system threads during its lifetime.
Contact John Neffenger with questions or comments.