| Figure-: | Timings for the connection setup time. |
Web Services is the current band wagon and many developers and managers are jumping on board, eagerly pushed by large tool vendors. From a developers perspective Web Services might be viewed as yet another distributed computing model, where SOAP is yet another on-the-wire protocol and WSDL yet another interface definition language. Right or wrong - it is a fair way to look at it. Using this perspective, I wrote a small benchmark comparing XML-RPC with CORBA and RMI. The figures shows that one should carefully judge why using web services. There are many strong arguments for using web services, however performance is not one of them.
Distributed system has been around for decades. Simple socket connections with proprietary on-the-wire protocols was replace by remote procedure calls (RPC) during the 80's. The major benefits was the idea of abstracting away all technical details of parameter marshalling, connection management, byte ordering etc and provide a simple procedure call semantics instead. When object-oriented programming became the lingua franca among developers during the 90's, the idea of an object request broker was born. The major improvement compared to RPC was remote references, which allowed fine-grained client dependent server side state, a.k.a. server-side objects or servants. During the years CORBA (Common Object Request Broker Architecture) has evolved to a solid international standard, supported by major tool vendors.
Both RPC and CORBA is programming language neutral. However, when Java can be chosen at both sides of a network connection, the preferred choice is Java RMI (Remote Method Invocation). RMI has all benefits of CORBA (although not all support technologies around a decent industrial strength ORB), but improves upon it by adding dynamic code distribution as well, together with an extremely easy to use programming model.
Common for RPC, CORBA and RMI are an (1) on-the-wire protocol for marshalling request arguments and the return values and a language for defining the public callable interface, a so called (2) interface definition language. Strictly speaking, we need to separate between the protocol for setting up and maintaining a call and the protocol for marshalling the arguments and return values, but I leave that out for brevity. RPC is not exactely one standard, rather it is a communication semantics. Therefore, one can find several on-the-wire protocols such as XDR and ASN.1 and several interface definition languages. CORBA on the other hand is a standard, and is using IIOP as the on-the-wire protocol and IDL as the interface definition language. Finally, RMI is using plain Java as the interface definition language and JRMP with object serialization as the on-the-wire protocol. These three technologies has as their overall objective to simplify network communication hassles and encapsulate low-level details.
Web Services provides the developer with the ability to interact with a web application at the on-the-wire protocol level, because the data representation is text and not binary, as is the case for the other three technologies. The on-the-wire protocol for web services is called SOAP (Simple Object Access Protocol) and is a XML standard. A web service is generally described in WSDL (Web Service Description Language), the interface definition language of choice. WSDL is another XML standard. The transport protocol for web services is usually HTTP, but a SOAP message can be transported by, for example, an email (SMTP).
As a developer, one can approach web services at the data representation level (SOAP), the message level (XML Messaging) or the remote procedure level (XML-RPC). If you are not interested in data representation and transport technicalities, the latter is the technology of choice. That means, looking at XML-RPC as a competitor to CORBA, Java RMI or plain old RPC. From this perspective one can judge a new technology according to easy of use, interoperability and performance.
An enterprise application might focus on interoperability if it must be accessable from the public internet. However, a private enterprise application running at an intranet or a VPN, might favor performance and bandwith instead. I recently had the pleasure of investigating web services from the perspective mentioned above. The rest of this article will discuss how I implemented a benchmark comparing the performance of Java XML-RPC (JAXRPC) with Java CORBA and Java RMI.
Web services uses text/XML as the data representation format. Obviously, it can't perform equally well as a binary format. My major question, for the pilot project, was; How bad?
I decided to make a quick test and leave out some aspects. That means, there are many aspects to take into account, such as many different implementations (vendor products) and many platforms. Faced at all possibilities, I took one road. Download and install Apache Axis and then develop a benchmark using the standard tools for RMI and CORBA bundled with the latest JDK 1.4.1.
The service methods are defined by one Java interface. From that interface are code generated for both JAXRPC and CORBA (Listing-1 and 2). There is one servant implementation shared by all technologies. Both the servant and the client test program have no knowledge about the transport technology used.
|
||
interface PerformanceTest {
public void ping();
public int multiplyByTwo(int n);
public String toUpperCase(String s);
public String transportValues(int integerValue, double realValue, String textValue);
public String transportBean(DataBean bean);
} |
||
|
|
||
class DataBean implements Serializable {
private int integerValue = 0;
private double realValue = 0;
private String textValue = "Hello JaxRpc";
public DataBean() { }
public DataBean(int integerValue, double realValue, String textValue) {
this.integerValue = integerValue;
this.realValue = realValue;
this.textValue = textValue;
}
//Getters and Setters...
} |
||
|
The names tells exactely what the methods are doing. The RMI and CORBA server-parts are using a special server program, but the JAXRPC part is using Apache Axis running in Tomcat. The RMI and CORBA server starts a name service, creates a servant objects and binds them to a name service. The interface ServantContainer (Listing 3) abstracts the details of this. You can find all source code to this article in a ZIP file, if you scroll down to the end of this article.
|
||
interface ServantContainer {
public String getName();
public String getServantName();
public void setServantName(String servantName);
public int getPort();
public void setPort(int port);
public void startNameService() throws Exception;
public void bindServant() throws Exception;
public void shutdown() throws Exception;
} |
||
|
When the client starts, it reads the command line arguments and decides which technology to use, i.e. RMI, CORBA or JAXRPC and which work load to run plus other parameters like host name. The client create a ServiceProxyCreator (Listing-4) object and a Workload (Listing-5) object. The creator knows how to establish a connection to a specific technology and the workload knows which test to perform. These interfaces can easily be implemented so other workloads or technologies can be tested.
|
||
interface ServiceProxyCreator {
public String getName();
public int getPort();
public void setPort(int port);
public String getHost();
public void setHost(String host);
public String getWebappName();
public void setWebappName(String webappName);
public String getServiceName();
public void setServiceName(String serviceName);
public PerformanceTest getPerformanceTest() throws Exception;
} |
||
|
|
||
interface Workload {
public String getName();
public String getDescription();
public void runOnce(int n) throws Exception;
public void setPerformanceTest(PerformanceTest t);
} |
||
|
Each service methods corrsponds to a simpel Workload implementation. There are five Workloads - each calling one specific service method -, which are listed below.
Each Workload is run several time, to capture run-time optimization, such as caching and the JVM hotspot. The timings are performed for the following run cases.
You can dig into the details by
All Workloads were run against each of the three technologies RMI, CORBA and JAXRPC respectively. The raw table data can be viewed in the tables below. Each data item is the number of milliseconds for one round-trip.
| [ms] | JAXRPC | CORBA | RMI | ||
| Connection | 2 454.0 | 1 211.0 | 1 152.0 | ||
| 1 time | 1 001.0 | 120.0 | 0.8 | ||
| 10 times | 54.1 | 18.0 | 4.0 | ||
| 1000 times | 13.9 | 3.5 | 1.2 | ||
| 10 times | 13.0 | 4.0 | 2.0 | ||
| 1 time | 20.0 | 0.8 | 0.8 | ||
| Max time | 1 001.0 | 120.0 | 4.0 | ||
| Min time | 13.0 | 0.8 | 0.8 | ||
|
|||||
| [ms] | JAXRPC | CORBA | RMI | ||
| Connection | 2 434.0 | 841.0 | 501.0 | ||
| 1 time | 1 061.0 | 30.0 | 0.8 | ||
| 10 times | 60.1 | 5.0 | 5.0 | ||
| 1000 times | 16.7 | 3.3 | 1.3 | ||
| 10 times | 13.0 | 2.0 | 2.0 | ||
| 1 time | 40.0 | 0.8 | 0.8 | ||
| Max time | 1 061.0 | 30.0 | 5.0 | ||
| Min time | 13.0 | 0.8 | 0.8 | ||
|
|||||
| [ms] | JAXRPC | CORBA | RMI | ||
| Connection | 2 474.0 | 641.0 | 541.0 | ||
| 1 time | 1 031.0 | 50.0 | 20.0 | ||
| 10 times | 58.1 | 9.1 | 4.0 | ||
| 1000 times | 15.6 | 2.6 | 1.4 | ||
| 10 times | 14.0 | 3.0 | 3.0 | ||
| 1 time | 30.0 | 0.8 | 0.8 | ||
| Max time | 1 031.0 | 50.0 | 20.0 | ||
| Min time | 14.0 | 0.8 | 0.8 | ||
|
|||||
| [ms] | JAXRPC | CORBA | RMI | ||
| Connection | 2 433.0 | 631.0 | 521.0 | ||
| 1 time | 1 072.0 | 60.0 | 20.0 | ||
| 10 times | 63.1 | 8.0 | 2.0 | ||
| 1000 times | 17.4 | 2.8 | 1.5 | ||
| 10 times | 16.0 | 4.0 | 2.0 | ||
| 1 time | 40.0 | 0.8 | 20.0 | ||
| Max time | 1 072.0 | 60.0 | 20.0 | ||
| Min time | 16.0 | 0.8 | 1.5 | ||
|
|||||
| [ms] | JAXRPC | CORBA | RMI | ||
| Connection | 2 434.0 | 641.0 | 531.0 | ||
| 1 time | 1 091.0 | 50.0 | 30.0 | ||
| 10 times | 62.1 | 7.0 | 4.0 | ||
| 1000 times | 18.3 | 2.6 | 2.1 | ||
| 10 times | 16.1 | 4.0 | 3.0 | ||
| 1 time | 30.0 | 10.0 | 0.8 | ||
| Max time | 1 091.0 | 50.0 | 30.0 | ||
| Min time | 16.1 | 2.6 | 0.8 | ||
|
|||||
There are two surprises in the tables - at least I think so. First, JAXRPC is not slower than RMI and CORBA - it is substantially slower. Second, RMI in spite of its rumour is slightly faster than CORBA. Maybe the relative figures between RMI and CORBA might change if one replaces the JDK version of CORBA with another implemention from a CORBA vendor.
I had expected a significant difference between sending three values as arguments and as a java bean. However, as you can see in table 4 and 5, at the max and min values, there are no significant difference. Especially not for JAXRPC, although we can observe that in the best case (min time) CORBA prefer sending values as arguments but RMI the opposite.
The tables also shows us that it is not a big difference between the five work loads. Therefore, I have collapsed them all into one new table, showing the mean values (Table-6). This table also shows in the rightmost three columns, the relationships between the three technologies.
| [ms] | JAXRPC | CORBA | RMI | JAXRPC/CORBA | JAXRPC/RMI | CORBA/RMI | ||
| Connection | 2 445.8 | 793.0 | 649.2 | 3.1 | 3.8 | 1.2 | ||
| Max time | 1 051.2 | 62.0 | 15.8 | 17.0 | 66.5 | 3.9 | ||
| Min time | 14.4 | 1.2 | 0.9 | 12.4 | 15.2 | 1.2 | ||
|
||||||||
The relationships clearly shows that even during optimal conditions (min time), i.e. many repeated runs, JAXRPC is still 15 times slower than Java RMI and 12 times slower than CORBA. For a single cold start round trip it is much worse. JAXRPC is 66 times slower than RMI and 17 times slower than CORBA.
By comparing (Java) CORBA with RMI, we can see that they are quite equal in the best case (1.2 for min time) and CORBA is slightly slower than RMI in the worst case (3.9 for max time). Problably there are some extra overhead structures in the run-time system of CORBA, that after some warm-ups become cached.
The last row of table 6 shows us the kind of relative figures we can expect from the three technologies. An alternative way of showing these numbers is to compute the number of round-trips per seconds, see table 7.
| C&RT/s | JAXRPC | CORBA | RMI | ||
| Connection | 0.4 | 1.3 | 1.5 | ||
| Max time | 1.0 | 16.1 | 63.3 | ||
| Min time | 69.3 | 857.0 | 1 056.6 | ||
|
|||||
Table 7 shows that for a very simple web service method invocation we can expect around 70 calls/seconds. However, using CORBA which is language neutral exactely as web services, we can perform around 850 calls/seconds and if we can go for a Java only solution using RMI, we can perform more than 1000 calls/seconds.
The table also shows us that in the worst case, we can call a web service method just once a second. Compare this to CORBA and RMI. In the worst case of CORBA we can perform 16 calls/second and using RMI we still can perform around 60 calls/second during the worst conditions.
Another important observation, is the connection setup time. If we have a web service that setups a connection and performs one service method call, it is likely that the latency will be dominated by the connection setup time. It takes two and a half second to establish a connection to a web service. As I understand it must be due to client-side caching of XML and run-time system object, because the transport is HTTP, which by itself is state-less.
The relative differences between the technologies becomes very appearant if the numbers are put into diagrams. The diagrams below shows the connection setup time and the worst vs. the best round-trip times.
|
| ||
|
| ||
|
As you can see from figure 2, there is a very large span between the worst time and the best time: From one second down to 14 milliseconds. Even CORBA and RMI shows speed-up, but not as dramatic as JAXRPC. If we put one single work load into a diagram and shows the time it takes for 1, 10, 1000, 10, 1 round-trips respectively, something very interesting shows up. Look at diagram 3 below.
| ||
|
The first single round-trip takes abount one second to complete. After some substantial warm-ups, the last single round-trip takes just 30 milliseconds! It's exactely the same kind of call as the first one taking 1091 milliseconds. Therefore, caching and other optimization techniques are indeed important and something you as a developer has to watch out for.
This observation implies that it seems to be a good idea to perform warm-up service calls in the background, when the user is busy with other things such as filling in forms. Then when it's time for a real business call, the latency have been significant reduced and the user experience a responsive application instead of a sluggish one. The same can be said for the connection setup time as such. However, in general, it seems to be harder to realize than warm-ups of service calls.
At last, the benchmark was executed on a platform with these characteristics
Web Services in general and XML-RPC in particular has many good promises for the future. When interoperability and easy access of enterprise business operations are the key focus, XML-RPC is a very good candidate to investigate and compare to CORBA. Simply, because it is so easy to inspect exactely what "goes on the wire".
On the other hand, if performance such as low latency, large number of service calls per time unit and large bandwidth is the key aspects for an enterprise application, XML-RPC is not the first hand choice.
My quick study shows that during the best conditions, when the JVM Hotspot has been given its chance to perform run-time optimization, the Java version of XML-RPC called JAXRPC, still is 15 times slower than Java RMI and 12 times slower than JDK CORBA. It is possible that another CORBA implementation might perform better than the JDK version.
For a small number of service calls (a.k.a. cold starts), JAXRPC is 66 times slower than Java RMI and around 17 times slower than JDK CORBA.
This quick study, shows that more investigation is needed and a common benchmark should be defined that can be used as a reference for comparing the performance of different JAXRPC implementations and how they compete with the old-timers CORBA and RMI.
Jens Riboe is an independent consultant based in Stockholm and London. He is the founder of Ribomation, a training and consulting company in Sweden during the period 1995-2000, where he developed and lectured many courses in Java, EJB, JSP/Servets, Swing, C++, UML, Perl, Erlang, real-time programming and much more. He implemented one of Swedens first internet bookshops back in August 1995, Ribomation Books. He has a background as an assistant professor at KTH (Royal Inst. of Technology) in Stockholm. He is also a SUN Certified Java programmer, developer and web component developer.