Thursday, March 27, 2008

Web Services in Java (Part 1)


Web Services in Java (Part 1)




When learning Web Services in Java, you have two complicating factors:

1) Web Services in Java are built on a tall stack of Java technologies and Web specifications: JAXB, JAX-RPC, JAXM, JAXP, SAAJ, WSDL, UUDI, etc. And to gain a detailed understanding of Web Services you need some understanding of all of these technologies. That's a steep learning curve!

2) There are multiple Web Service implementations, toolkits, and IDE plugins for in Java. All of them work differently to some extent, and all of them are in some degree of flux as these technologies and toolkits evolve and mature. Even when two separate Web Service toolkits adhere to the same standards, they may differ greatly in their implementation.

These two factors present a significant barrier to developers learning how to create robust and secure Web Serivces. And this situation is further complicated by lots of poor and outdated documentation on the subject. Most books and online tutorials I have read take the approach of teaching all the details of all of the myriad technologies (JAXB, JAXM, SAAJ, JAX-RPC, etc.) before you even begin to construct your first "Hello World" Web Service. This may be a convenient way for the author to package her or his knowledge, but it's not a very effective way to learn. I find that many developers learn fastest by jumping right in and getting something working immediately, even if they don't understand how it works, and then expand their knowledge from there in any direction they want.

I suspect that authors tend to focus so much on the Java API's and Web technologies because those are the main elements of Web Services that are universal across all implementations. If they go down the path of creating a complete Web Service by example, they would have to cover different ways of doing a Web Service with each different toolkits, IDE plugins, and Application Servers; and anything they wrote would be outdated in a year or less. So, it's often easier to just present the core technologies and hope the developer can figure out the rest. (ha)

For this article, I'm going to demonstrate creating a Web Service using the Java Web Service Developer Pack 2.0 from Sun, and deploying the Web Serivice to JBoss 4.0. So if you are using Apache Axis, this example probably won't work for you exactly as-is. Sorry. That's the nature of this beast. I plan writing a future article on how to create an Apache Axis Web Service.

The good news is that it doesn't take very much to get a simple Web Service working. The modern toolkits have become easier to use, and do most of the nasty detail work for you. Once you get a simple Web Service working, you can expand to a reasonable set of production-grade functionality without tackling all the learning curves at once. Then, over time, you can choose to drill down into individual advanced aspects of Web Services as the need arises.

For this example, there are at least 4 different paths I could take to start constructing this Web Service, depending on various design decisions and considerations. (WSDL-centric vs. Java-centric, EJB vs. Servlet, etc) But I'm not going to get into all of that discussion. I'm just going to charge ahead on one simple path, and save all the alternatives and philosophy for another article.

Here we go!

Java Web Services : Step 1 - Create a new Web Application

The Web Service for this example is going to be deployed as a regular WAR file. So in your IDE of choice, create a standard Web Application directory structure including source (src) and WebRoot/WEB-INF directories.

There are no Web Service JAR files needed for your WEB-INF/lib directory. Your application server should already have Web Service support libraries installed. In our case, JBoss 4.0 comes with Web Service support.


Java Web Services : Step 2 - Create a Java Interface for your Web Service

This interface will represent the functionality you want to expose with this Web Service.

You can name your interface anything you want. I recommend naming it <Domain>Server where "<Domain>" is any name that represents the services offered. Like "WeatherServer" or "UserServer" or "OrderServer", etc.

This interface must extend the java.rmi.Remote interface. And each method in your interface must throw java.rmi.RemoteException.

Your interface can declare any regular Java methods. The method parameters and method return types can be of any primitive type (int, float, long, etc.), or standard type (String, Long, Calendar, etc.) or an array of objects. You can also use any custom Java Objects of your own. However, there are some limits and design considerations you'll discover when using your own objects. For now, try and keep it simple.

Here is my sample interface:


package com.example.webservice.user;

import java.rmi.Remote;
import java.rmi.RemoteException;
import com.example.webservice.exception.GeneralException;

public interface AccountManagementServer extends Remote{

public boolean isUserKeyValid(String pid)
throws RemoteException, NoSuchAccountException, GeneralException;

public PortableUser getUser(String userName)
throws RemoteException, NoSuchUserException, GeneralException;
}






These methods throw some custom exceptions I have created.

Here is the code for one of those exceptions:


package com.example.webservice.exception;
public class GeneralException extends Exception {
public GeneralException(String message) { super(message); }
public String getMessage() { return super.getMessage(); }
}

There's nothing difficult or special in creating an Exception that can be thrown across a Web Service, but one minor rule you have to follow is that there must be a getter method for any parameter used in a constructor. So, since this constructor takes a "message" as a parameter, we must declare a getMessage() method in this class. (The parent class "Exception" has a getMessage method, but that doesn't count.)


Java Web Service : Step 3 - Create the Implementation


Create a Java class that implements the Web Service interface. Also create any support classes that you want to use.

This class must implement javax.xml.rpc.server.ServiceLifecycle, and implement the interface you defined previously.

You can name this implementation class anything you want, but I recommend just appending "Impl" to the Interface Name. Like "WeatherServerImpl" or "OrderServerImpl".


package com.example.webservice.user;

import java.rmi.RemoteException;
import javax.xml.rpc.ServiceException;
import javax.xml.rpc.server.ServiceLifecycle;
import javax.xml.rpc.server.ServletEndpointContext;
import com.example.webservice.exception.GeneralException;
import com.example.webservice.exception.NoSuchAccountException;
import com.example.webservice.exception.NoSuchUserException;

public class AccountManagementServerImpl implements
AccountManagementServer, ServiceLifecycle {

private ServletEndpointContext ctx;

public boolean isUserKeyValid(String pid) throws NoSuchAccountException,
GeneralException
{
try {
return UserService.isUserKeyValid(pid);
} catch (NoSuchAccountException nsae) {
throw nsae;
} catch (Exception ex) {
throw new GeneralException(ex.toString());
}
}

public PortableUser getUser(String userName)
throws RemoteException, NoSuchUserException, GeneralException {

User user = null;
try {
user = UserService.getUser(userName);
} catch (NoSuchUserException nsue) {
throw nsue;
} catch (Exception ex) {
throw new GeneralException(ex.toString());
}
if (user == null) return null;

//Convert the big User object down to the small PortableUser object
PortableUser pu = new PortableUser();
pu.setUserName(user.getUserName());
pu.setPassword(user.getPassword());
pu.setPortalId(user.getPortalId());
return pu;
}

public void init(Object context) throws ServiceException {
ctx = (ServletEndpointContext) context;
}

public void destroy() {
}
}

The code above uses another application service called "UserService". The source code is not provided here, but it is not important. The key concept is that your Web Service implementation should re-use an pre-existing application services whenever possible. Keep as much business logic out of your Web Service as possible.

In the above example, one of the methods in my Web Service returns an object of type "PortableUser". This is just a simple data class that I created as follows:


package com.example.webservice.user;

public class PortableUser {
private String userName;
public String getUserName() { return userName; }
public void setUserName(String userName) { this.userName = userName;}

private String portalId;
public String getPortalId() { return portalId; }
public void setPortalId(String portalId) { this.portalId = portalId;}

private String password;
public String getPassword() { return password; }
public void setPassword(String password) { this.password = password;}
}

I created the object "PortableUser", because my application's regular "User" object is a huge object with a large number of methods and nested child objects. If I had used my User object as a return value for my Web Service, performance would have suffered significantly.

Besides, I only wanted to expose a subset of my User object anyway. In cases like these, you should make lightweight Java classes (Data Transfer Objects) that package the values you want to use in your Web Service.


Java Web Service : Step 4 - Create a configuration file

We are now going to use a Web Service generator tool to create the Web Service WSDL and JAX-RPC mapping files. If you don't know what WSDL and JAX-RPC is, don't worry about it. The WSDL file describes your Web Service to the outside world, and the JAX-RPC mapping files tie the WSDL into your Java Classes.

The Web Service generator tools can generate WSDL files from Java classes, OR they can generate Java classes from WSDL files. In our case, we started by defining a Java class (an interface), and so we will be generating the WSDL. But there are times when you might want to design a Web Service starting with a hand-made WSDL file. But that's another design discussion to be addressed in a future article.

In Apache Axis, the two relevant tools are called (appropriately) "WSDL2Java" and "Java2WSDL".

In the JWSDP, there is only one tool, called "wscompile", and it can perform either conversion. I'm going to use wscompile for this example.

For wscompile, you will need to create a config file for generating your Web Service. This config file is only used to generate the WSDL and mapping files. You can name the config file anything you want, but I recommend you name it config-<Domain>Service.xml. Example: "config-UserService.xml". If you have more than one Web Service package to generate, you will need to have more than one config file, which is why I recommend against naming it just "config.xml".

There are 5 things you must define in your config file. Two of these are namespaces, and there really are a lot of different ways you could name your namespaces, but I'll leave that for a future discussion. What I offer below are just loose suggestions we can use for now:



config-AccountManagementService.xml

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE configuration>
<configuration xmlns="http://java.sun.com/xml/ns/jax-rpc/ri/config">
<service name="AccountManagementService"
targetNamespace="http://com.example.webservice.user/"
typeNamespace="http://com.example.webservice.user/types"
packageName="com.example.webservice.user">
<interface name="com.example.webservice.user.AccountManagementServer"/>
</service>
</configuration>

Notes about the above configuration file:

  • The Web Service name is "AccountManagementService".

  • The "targetNamespace" defines the namespace to use for the Web Service.*

  • The "typeNamespace" defines the namespace for your Web Services types. I simply added "types" to the end of my targetNamespace.

  • The packageName is the name of the Java package where your Web Service classes live, and where any generated classes are to be placed. (We will not be generating classes in this example.)

  • The interface name is the full name of your Java Interface for the Web Service.


*A namespace is just a universally unique string. It is not a URL, even though it's a common practice to format namespaces as URL's. It's also common practice to include Java package name information in the namespace name. The format I've chosen here does both....and I'm starting to think that's confusing.

Another way to name a namespace is to specify a real http domain, and then use the URL path to try and specify the remaining parts of your package path that aren't referenced in the domain (in reverse order). If your website is "myapp.example.com", and your package name is com.example.myapp.webservice.user, then your namespace could be "http://myapp.example.com/webservice/user". This is less confusing, but not representative of the true package name.

The source of the confusion here is that Java packages are named in descending order (com.example.myapp) but URL's are always named in ascending order (myapp.example.com). And if you make any effort to mix the two things, you'll have problems. =)

So for my example Web Service here, don't take my namespace naming method too seriously.


Java Web Service : Step 5 - Generate the WSDL file and the JAX-RPC mapping files


wscompile.sh is part of the Java WebServices Developer Pack from Sun. And in this example, I'm going to show using wscompile from the command line, and using wscompile from an ant task. You can choose to run wscompile either way. The command line version might get you going faster, but for long-term manageability I recommend eventually creating the ant task.

The Java Web Services Development Pack (with wscompile) can be downloaded from here: http://java.sun.com/webservices/jwsdp/index.jsp

You could also use Java2WSDL from Apache Axis. Both will generate somewhat similar results, assuming both tools have done a good job of adhering to the same versions of the same standards.

Once you have the JWSDP installed, you can run wscompile.sh from the command-line as follows:

wscompile.sh -cp WebRoot/WEB-INF/classes -verbose -define -f:rpcliteral
-nd WebRoot/WEB-INF/wsdl
-mapping WebRoot/WEB-INF/mapping-AccountManagementService.xml
config-AccountManagementService.xml

You can use "wscompile.sh -help" to get detailed information on what all these parameters do.

For automated building and less command-line fussing, I recommend using an ant task for wscompile. To run the wscompile ant task, you need the following JAR files in your ant's classpath. All of these JAR files can be found in the JWSDP:

activation.jar
jaas.jar
jaxrpc-api.jar
jaxrpc-impl.jar
jaxrpc-spi.jar
jta-spec1_0_1.jar
mail.jar
relaxngDatatype.jar
resolver.jar
saaj-api.jar
saaj-impl.jar
xmlsec.jar
xsdlib.jar

Once these libraries are in your ant's path, the following task will generate the WSDL and mapping files you need:

<taskdef name="wscompile" classname="com.sun.xml.rpc.tools.ant.Wscompile" />
<target name="generate">
<wscompile verbose="true" classpath="WebRoot/WEB-INF/classes"
define="true" features="rpcliteral" nonclassdir="WebRoot/WEB-INF/wsdl"
mapping="WebRoot/WEB-INF/mapping-AccountManagementService.xml"
config="config-AccountManagementService.xml"/>
</target>

Notes about the above ant task:

  • "classpath" is the path of your interface and related classes, as well as any external jars your code needs.

  • "define" indicates that we are generating just the WSDL and mapping files, and no other classes.

  • "features" are the various options you can use for specifying what type of WSDL to generate.

  • "mapping" is the location of the output JAX-RPC mapping files.

  • "nonclassdir" is the location to place the WSDL file (and other nonclass files that are generated)

  • "config" is the name of the config file for wscompile



Whether you use the ant file, or the command-line, you must execute this wscompile from the project's root directory, because that's how I've setup this example. (That's why you see "WebRoot/WEB-INF" prepended to the wsdl and mapping locations.) You could also run wscompile from the WebRoot/WEB-INF directory itself, and then you wouldn't need those directories specified in all your parameters above.



Java Web Service : Step 6 - Add your service to your web.xml file

A Web Service can either be created as a Web Component (essentially a servlet), or an EJB. In this example, I have created this Web Service to be a Web Component, and so it must be defined in web.xml as a servlet. The servlet-class is just the Web Service implementation class I've already created.

<?xml version="1.0" encoding="UTF-8"?>

<web-app xmlns=http://java.sun.com/xml/ns/j2ee
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"
version="2.4">

<servlet>
<servlet-name>AccountManagementService</servlet-name>
<servlet-class>com.example.webservice.user.AccountManagementServerImpl</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>AccountManagementService</servlet-name>
<url-pattern>/AccountManagementService</url-pattern>
</servlet-mapping>
</web-app>



Java Web Service : Step 7 - Add your service to your webservices.xml file

If you don't have a webservices.xml file, you can just add one to your WEB-INF directory. This is the file that will connect all the pieces together, and specifies which WSDL file goes with which mapping file and which servlet.

Add service to webservices.xml


<?xml version="1.0" encoding="UTF-8"?>

<webservices
xmlns="http://java.sun.com/xml/ns/j2ee"
xmlns:impl="http://com.example.webservice.user/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=
"http://java.sun.com/xml/ns/j2ee
http://www.ibm.com/webservices/xsd/j2ee_web_services_1_1.xsd"
version="1.1">

<webservice-description>
<webservice-description-name>AccountManagementService</webservice-description-name>
<wsdl-file>WEB-INF/wsdl/AccountManagementService.wsdl</wsdl-file>
<jaxrpc-mapping-file>
WEB-INF/mapping-AccountManagementService.xml
</jaxrpc-mapping-file>
<port-component>
<port-component-name>PortComponent</port-component-name>
<wsdl-port>impl:AccountManagementServerPort</wsdl-port>
<service-endpoint-interface>
com.example.webservice.user.AccountManagementServer
</service-endpoint-interface>
<service-impl-bean>
<servlet-link>AccountManagementService</servlet-link>
</service-impl-bean>
</port-component>
</webservice-description>

</webservices>

Here are the key parts of this file you have to get right:

  1. The xmlns:impl (near the top) must be the same as the namespace you used in your config file for generating the WSDL.

  2. The wsdl-file specifies the location of your WSDL file relative from WebRoot

  3. The jaxrpc-mapping-file specifies your mapping file, also relative from WebRoot.

  4. The wsdl-port will be "impl:" + <your interface class name> + "Port". When you learn more, you'll learn what a "Port" is, and see how the Port name is defined in the WSDL file, and how it is referenced by this "impl" namespace.

  5. The service-endpoint-interface is the full name of your interface class.

  6. The servlet-link is the "servlet-name" you defined in your web.xml



Java Web Service : Step 8 - Package and deploy these files

There's nothing special about building a Web Service war file. Just package up all if the class files you have created, plus the generated XML and WSDL files. A simple ant-build for the war should suffice:



<target name="build-war">
<war destfile="wsexample.war" webxml="WebRoot/WEB-INF/web.xml">
<classes dir="WebRoot/WEB-INF/classes" includes="**/*"/>
<lib dir="WebRoot/WEB-INF/lib" includes="*.jar"/>
<webinf dir="WebRoot/WEB-INF" includes="**/*.xml,**/*.wsdl"
excludes="web.xml"/>
</war>
</target>


Deploy your WAR file to your application server (in my case JBoss). If all is well, you should see messages in the log that indicate your Web Service has self-installed gracefully. (Note: When using JBoss, you must have the jboss-ws4ee.sar module installed).

Java Web Service : Step 9 - Verify the Web Service is installed

You can test that your Web Service is installed using the following URL:

http://localhost/wsexample/AccountManagementService?wsdl

When you call your Web Service with the "?wsdl" parameter, it should respond by giving you it's WSDL.

Alternatively, you can go to http://localhost/ws4ee. You'll get two links that will verify that Web Services are running properly on your server, and another link that will show that your Web Service is installed.

These tests will only ensure that your Web Serivice has a proper WSDL file. To really test the full functionality of your Web Service, you need to create some kind of test client that can call your services. You can either write a simple client of your own (covered in the next step), or you can use some of the various IDE plugin's on the market (like the Web Services Explorer for Eclipse) that will self-discover the details of your service.


Java Web Service : Step 10 - Test the Web Service

Making a Web Service client is just a miniature version of the previous steps you went through to make a Web Service itself.

You start off by making any kind of Java project you want. It doesn't have to be a Web Application.

This time we'll be generating client classes instead of WSDL. Create your configuration file for wscompile as follows:

config-AccountManagementServer.xml

<configuration
xmlns="http://java.sun.com/xml/ns/jax-rpc/ri/config">
<wsdl location="http://localhost/wsexample/AccountManagementService?wsdl"
packageName="com.example.webservice.user.client"/>
</configuration>

Specify the URL of your Web Service (with ?wsdl appended) in the "location" attribute. Specify the package name of the classes to generate in the "packageName" attribute.

Next, run wscompile as follows:

wscompile.sh -verbose -gen:client -f:rpcliteral -s src -d src -keep
-mapping mapping-AccountManagementServer.xml
config-AccountManagementServer.xml

OR, if using ant:

<taskdef name="wscompile" classname="com.sun.xml.rpc.tools.ant.Wscompile" />
<target name="generate">
<wscompile verbose="true" client="true" features="rpcliteral" sourceBase="src"
base="src" keep="true" mapping="mapping-AccountManagementService.xml"
config="config-AccountManagementService.xml"/>
</target>

Finally, here is a Test Class you can run that calls the Web Service methods:


package com.example.webservice.user.test;

import java.net.URL;
import javax.xml.namespace.QName;
import javax.xml.rpc.Service;
import javax.xml.rpc.ServiceFactory;
import com.example.webservice.user.client.AccountManagementServer;
import com.example.webservice.user.client.PortableUser;

public class ServiceTest {

public static final String host = "http://localhost";
public static final String path = "/wsexample";

public static void main(String args[]) {
try {
URL url = new URL(host + path + "/AccountManagementService?wsdl");
QName qname = new QName("http://com.example.webservice.user/",
"AccountManagementService");
ServiceFactory factory = ServiceFactory.newInstance();
Service service = factory.createService(url, qname);
AccountManagementServer userServer = (AccountManagementServer)
service.getPort(AccountManagementServer.class);
PortableUser user = userServer.getUser("demo");
System.out.println(user.getUserName());
System.out.println(user.getPortalId());
System.out.println(user.getPassword());
} catch (Exception e) {
System.out.println("ServiceTest: " + e);
}
}
}

And that concludes our end-to-end Web Service example. =)





Scalable Java Web Applications, Part 2

Saturday, August 20, 2005

Scalable Java Web Applications: Part 2

I've recently discovered another way not to handle load-balancing for a Java Web Application...at least not with Tomcat. :)

As mentioned in my previous article Scalable Java Applications, in order to maintain a user's session data in a web application that is distributed across a server farm, your two basic options are:

1) Share the user's data between all of the servers.
2) Use "Server Affinity" to pin a user to one given server while they are logged in, and store their session data on that server.

Server Affinity has some pretty hefty benefits in terms of simplicity, speed of development, performance, and scalability.

However, if your web site encrypts all communication through SSL (https), it can be difficult to achieve Server Affinity. Standard network hardware cannot read and interpret the incoming requests if they are encrypted.

The solution recommended by our network hardware vendor was to add an SSL-izer box in front of our load balancer box. The SSLizer decrypts all incoming traffic, and encrypts all outbound traffic. You host your Site Certificate on the SSLizer box instead of on the Web Servers.

So the incoming path goes: Browser -> SSLizer -> Load Balancer -> Web/Java Server.

The outbound path goes: Web/Java Server -> Load Balancer (maybe) -> SSLizer -> Browser

All inbound communication is decrypted by the SSLizer and forwarded onto the load balancer as an http request on port 80. This allows the load balancer to easily read the requests and determine their destination, thus allowing for Server Affinity.

Piece of cake right? Nope!

The problem is that the Web Servers and Java Application Servers are tricked into believing that they are running an http site, when in reality this is an https site. (See "Level 2 Kluge" under The Kluge Scale). And every place in our system where absolute URLs are generated for the browser, the system will put "http" in the URL instead of "https".

The Application Server assumes these http requests are coming directly from the user's browser, and it has no way of knowing otherwise (that I know of).

The first problem we ran into was with the "base" tag in all of our JSP pages. The &ltbase> tag is standard HTML. It establishes the root path for all relative URL's on a page. However, we were using standard JSP logic to generate the <base&rt; tag, and so Java put "http" into the base URL. And thus all relative references in the page read as "http" instead of "https". This caused security warnings on the user's browser, as well as other odd behavior.

So, I went through our entire application removing all base tags, and re-working all of our URL's to be independently relative. (This is a bit error-prone and will cause future maintenance issues, but I guess we can live with it.)

The more serious problem we ran into next was with HTML redirects, which are generated from many places in a Web application. A "redirect" is an action where the application sends a response to a browser that forwards the browser to a different page. This is quite common, such as for sending the user a "GET" redirect response for a "POST" request. (See section in previous article Scalable Java Web Applications on the use of GET and POST in HTML.)

Tomcat is fundamentally designed to generate fully qualified URLs for all redirect requests. This behavior cannot be overridden. And it is even part of the Sun Specification for Java Servlets. Here is the relevant paragraph from the Sun specification of the "sendRedirect" method:

The sendRedirect method will set the appropriate headers and content body to redirect the client to a different URL. It is legal to call this method with a relative URL path, however the underlying container must translate the relative path to a fully qualified URL for transmission back to the client. If a partial URL is given and, for whatever reason, cannot be converted into a valid URL, then this method must throw an IllegalArgumentException.

On an HTML redirect, Tomcat prepends "http" to the redirect URL because Tomcat believes that the incoming request is "http". And Tomcat sends an "http" redirect to the browser. When the browser receives an "http" redirect in response to its original "https" request, the browser will pop up a security warning to the user. (The Firefox browser does not show such a warning, but Internet Explorer does.)

Example: If I try to redirect the browser to "/myApplication/pageOne.html", Tomcat will generate an HTML redirect to http://www.mysite.com/myApplication/pageOne.html. And this will cause a browser security warning popup.

So, once we installed the SSLizers, our application was popping up security alerts every other page. That is unacceptable.

For the Java and JSP code that I write, I can add code to fully quality all the redirects myself, and prepend "https" to the front of them. And that avoids this problem for my code.

But there are other redirects that happen internal to other Java Libraries that I can't control. Struts and JSF make use of a "redirect" tag in their XML control layer, and the Tomcat Login Process also makes use of redirects. The Tomcat Login Process is especially problematic because it was designed to prevent any and all alterations to its behavior. And the Tomcat Login process uses a redirect to forward the user into the application after they have successfully logged in.

The login page is the LAST place you want your users seeing any kind of security warning from the browser.

Here are the things I tried that did not work:

Dead End #1: In the server.xml file for Tomcat, you can specify the "scheme" for a given Connector. I can specify a scheme of "https", so that Tomcat will assume that all traffic for this controller will be https. Of course, this means that I'll never be able to use regular http on my site, but I can live with that for this particular application. I gave it a try, and I found out that the stubborn Tomcat Login Process won't allow that. If you set your Connector scheme to "https", the Tomcat Login process will then tack port ":80" to the end of your domain in your URL, thus forcing you BACK to port 80. Making an https request over port 80 causes an invalid request error on any Web Server.

Dead End #2: I spent a few hours experimenting with Apache mod_rewrite to see if I could construct a trap that would remove that port ":80" from inbound requests. What I discovered was that the request was failing before it even reached mod_rewrite, because the listener on port 80 was receiving encrypted garbage.

Dead End #3: I created a standard Java Servlet Filter so that I could alter all outbound traffic, and replace "http" with "https" in all outbound redirects. It turns out that j_security_check (the Tomcat Login Process) bypasses all Servlet Filters by design.

Dead End #4: I created my own login servlet, so I could manipulate the request data, and then forward the request into j_security_check. That way I could authenticate the user myself, and then hand them over to Tomcat. It turns out j_security_check does not allow itself to be a target of a servlet forward. And there is no way to intercept the login process AFTER j_security_check either.

Dead End #5: Finally, I came up with a truly evil idea: I called the login page in such a way that I didn't tell j_security_check where to send the user after they have logged in. This would cause j_security_check to generate an HTML 400 error after login. In my Application Configuration (web.xml), I setup a trap for 400 errors so that they are automatically sent to a custom JSP page that I wrote. The custom JSP page then detects if it came from j_security_check, and if so, it forwards the user into the application! Talk about a Rube Goldberg machine! However, it turns out that j_security_check invalidates the user's session if a 400 error is generated. Drat!

I tell you, the people who wrote j_security_check thought of everything! The philosophy is that a secure login process must be atomic (un-dividable). If you let a developer inject their own code anywhere into the login process, you risk compromising security if the developer's code breaks or is hackable.

Final Solution
Well, there is no final solution. The fundamental design of the SSLizer is in direct conflict with the fundamental design of Tomcat. We will have to work with our vendors to try something else, but I'm not sure this is a hardware issue.

Perhaps we are at conflict because we are handling load balancing and SSL with hardware, when Tomcat seems to assume this is a software concern. Tomcat has its own software load balancing mechanisms, and so do commercial Application Servers. It seems almost as if the designers of Tomcat never really considered making Tomcat work behind a hardware load balancer and SSLizer.

Perhaps commercial grade Java Application Servers have configuration settings that allow them to be told that they are operating behind an SSLizer. Perhaps they have more configuration options for controlling the default protocol for outbound traffic.

Work Around
As a temporary work-around, I have pulled a "Neo Kluge", and I have altered the Tomcat Engine itself to send relative URL's for every redirect. I downloaded all the source code for Tomcat from the Apache site, and traced down every place redirects are generated. I managed to find one key point that all redirects have in common (that was lucky), and so I only had to alter the class org.apache.catalina.connector.Response. I rebuilt "catalina.jar" and deployed it to our production Tomcat servers, and everything works perfectly now.

But obviously, this fix cannot be allowed to remain in production for very long. It's just too much of a maintenance nightmare to maintain our own version(s) of Tomcat.

On a side note, if I wanted to fix just the Tomcat Login Process and nothing else, I found that I could alter org.apache.catalina.authenticator.FormAuthenticator. This class controls the login process, as well as that final redirect that happens upon a successful login.

If we decide to replace the Tomcat Login Process with some other module that doesn't have this problem, we would still have redirect problems to deal with in the JSF framework. But I also downloaded the source code for Apache MyFaces (the implementation of JSF that we are using), and I traced down its redirect logic to the "redirect" method of the class org.apache.myfaces.context.servlet.ServletExternalContextImpl. That method can be altered to override the way redirect URL's are generated. And what's nice about this solution is that you can drop this altered class right into your Web Application's source directory, and it will override the same class in the JSF jar files. This way, you don't have to build your own version of the JSF jar files. (This is still a horrible kluge, but it's at least a tiny bit better than modifying Tomcat itself.)

Look for new postings to this blog in the comming days to see how we finally resolve this problem.

To Be Continued....
Well, we finally came up with a semi-decent solution to the problem with redirects in Tomcat.

Here is what you do:

In Tomcat's server.xml, you can configure the Connector to run behind a proxy using the proxyName and proxyPort parameters. Here is the relevant piece of Tomcat documentation on these parameters:

"The proxyName and proxyPort attributes can be used when Tomcat is run behind a proxy server. These attributes modify the values returned to web applications that call the request.getServerName() and request.getServerPort() methods, which are often used to construct absolute URLs for redirects. Without configuring these attributes, the values returned would reflect the server name and port on which the connection from the proxy server was received, rather than the server name and port to whom the client directed the original request."

Well, in my case, I'm running Tomcat behind an SSL decoder, not a Proxy, but the effect is the same, and the solution is the same.

So, in the Connector attributes, I set the "proxyName" to the server name of my website as seen from the outside world ("mysite.mydomain.com"). I set "proxyPort" to 443. I set "scheme" to "https", and I set "secure" to "true". And it works!

When Tomcat generates absolute URL's, it knows to use these parameters to build the URL rather than the values from the incoming request.

This solution seems easy in hindsight. But when I was coming at this from the other direction, I didn't know what to look for. I was doing tons and tons of reading on the topic of "SSL with Tomcat" and not on the topic of "proxy servers with Tomcat".

It's one of those "magic word" solutions where you can't find the answer until you guess the magic word to search for.

But I posted many messages on newsgroups and other forums, and nobody else knew the answer to this either. So, I don't feel too bad. :)

So, overall, we have a pretty good end-to-end solution. The only issue we still have is that by configuring the Connector this way, we've locked the entire site into https. We could never have parts of the site be just "http", because we've hard-coded the Connector to treat everything as https.

But that's not so terribly bad. We could still create a separate site for some http content, and jump back and forth between the two sites I suppose.

Scalable Java Web Applications, Part 1

Monday, July 11, 2005

Part I

Scalable Java Web Applications

The things they never teach you in any Java book or Java class...

I had a few things go wrong on a project of mine recently that reminded me of just how complicated it can be to make a create a highly-functional stateless web application.

Java is making it simple to create Enterprise Scalable applications, and yet a robust and clean method of maintaining state from page to page in a browser eludes us to this day.

If you have a rich web application where the users follow a workflow across multiple screens, your application will have to be stateful to some extent. You have to have the ability to "remember" what a user enters on one screen and carry that forward from screen to screen.

And what we have to work with is a combination of half-baked browser technologies that are each serve some limited little purpose, yet can be glued together in a "Frankenstein Kluge" to sorta-kinda make everything work.

Don't take my "half-baked" comment the wrong way; these technologies all work well for what they were originally intended for, but none of them were intended to be a robust state-saving mechanism.

There are many issues to consider with each of these technologies.

  • Cookies: Cookies can be disabled on a browser. They can be hacked or spoofed or stolen unless care is taken to properly encrypt them. Cookies are also treated in erratic ways on most browsers. Example: In the Internet Explorer browser, opening a "New" window will replicate all the cookies from the existing open window, but launching a new IE browser from an icon will not. So an application will either remember you, or not, depending on if you open a new "window" or new "instance" of your browser. And 99.9% of the web-surfing population of the world will never understand the subtle difference between those two states.
  • GET and POST parameters: GET and POST parameters are intended to be used for user input. But they can also be used to piggy-back state data from page to page (possibly as hidden fields). The problem is, all parameters look the same to the server, so this make it easy for a user to hijack your application data. Don't expose key fields to users, and beware of cross-site scripting, SQL injection, buffer overflows, parameter replacement, and all other manner of input manipulation.
  • Hidden fields: There are just a flavor of an HTML parameter, so they inherit all the weakness and dangers of HTML parameters. And hidden fields are only hidden until someone does "View Source". They are a clumsy and risky and tedious method for saving state...but they do work.
  • URL Re-writing: Kluge. Every place you have a URL reference in your entire application, you add some code to "inject" a session id into the GET or POST parameters. And this session id is a key into a table that contains the user's data on the server side. This is a common alternate to using cookies. In theory, this sounds easy. But in practice, I've found that URLs come from all over the place, and URL re-writing becomes a major plumbing project to implement correctly and completely.
  • JavaScript: JavaScript isn't really used to preserve application data from page to page, but JavaScript can inject more dynamic behavior into an existing page, and thus perhaps cut down on server round-trips. However, JavaScript can be disabled. It is difficult to work with, and doesn't work the same in all browsers.
  • Applets: You can say goodbye to your stateless HTML pages and code your entire application as a big stateful Java applet. Yay! However, Java has a long track record of browser compatibility issues, as well as producing awkward and sluggish graphical user interfaces. And most browsers are still too clumsy at identifying and handling the plugin for Java. Microsoft managed to effectively kill Java Applets during the browser wars. Microsoft has finally gotten out of the Java business, and modern Java plugins are getting smarter, but it might be too-little too-late for Java Applets to make a comeback. We'll see.
  • Server Session data: Server Session data is dependent on cookies (or URL-rewriting), and so inherits all of the weaknesses of these technologies (including the browser "New Window" problem). Server Session data only lives in the memory of one server (unless you take other measures described later in this article), so it is vulnerable to server fail-over, and switch-over. If you store application state in the session, you can be vulnerable to the "back" button on a browser knocking your application out of sync. You also have to be mindful of memory usage in the session, because your session store will be multiplied by the number of users you have. Since the server has no way of knowing when the user closes the browser, the server must implement an expiration mechanism for session data. And that in itself presents another set of timeout/logout issues to contend with.

Each of these little technologies does it's own thing in it's own little space, and by gluing them together you have a full Web Application.

It has been my experience that most Java Web applications today use a combination of Hidden Fields and Session Data to manage application information. The really huge and highly-specialized web applications (like Google and Amazon and eBay) are all total custom jobs. They have the resources and army of developers it takes to do their own plumbing for everything. Google makes crazy use of JavaScript, but they have the resources to pull that off.

But for the rest of us the I.T. world, we need to develop highly functional websites cheaply and quickly, and so we need to rely on frameworks and technologies to do as much of the plumbing as possible for us. We need cookies. We need server-side session data.

The Best Practice is to use Hidden Fields for any state data (tracking where the user is and what the user is doing) and using Session Data as a stateless reserve (things you want to remember about a user regardless of what page they are on).

In practice, this IS difficult to code properly, and requires a detailed technical design, with careful layout of the dataflow for the entire application.

Beware of the multiple-submit problem. Example: User presses a button to "submit" an order to the database, and then hits the "Back" button on the browser and presses "submit" again. Did they intend to place a 2nd order? Or did they intend to update the first order? Can your application tell the difference and handle either?

Beware of the refresh problem. Example: User presses a button to "submit" an order to the database, and the user gets a result screen that says something like "Thank you for your order". If the user simply hits "Refresh" on that screen, they will cause the order to be submitted again, because the "submit" operation was part of the request/response process that generated the result screen.

There may also be a few key commit point in an application where you need some hard mechanism to detect and prevent duplicate submissions, perhaps using tokens or timestamps on objects. Most modern frameworks (Struts, JSF, etc.) have some tools that can help.

The good news is that if you follow all the tips and Best Practices I'm outlining in this article, you will be able to develop an application that is immune to back buttons and refreshes without too much plumbing work on your part.

The following is a pretty good article on the subject of redirects and duplicate submissions. It's technically very accurate, if you look past the somewhat flakey writing in places:

http://www.theserverside.com/articles/article.tss?l=RedirectAfterPost

LOAD BALANCING
Okay, I'm on page 3 of this particular blog posting, and I still haven't gotten to the problem I encountered on MY project. :)

My problem was with load balancing and server session data. But in order to understand why load balancing is such a problem for a web application, you have to understand the nature of the technologies that the web is built upon.

As mentioned above, a modern Java application needs to store some amount of data in the Server Session. Writing a session-free Web Application would be complicated and cost-prohibitive, and incompatible with modern frameworks that assume session usage.

But if you've chosen to use Server Session data, how do you scale your application up to multiple servers? How can you store application data in a single server session when your user might be load-balanced across multiple servers? The issues involved with architecting a Web Farm for a java application are addressed in the following article:

http://www.onjava.com/pub/a/onjava/2001/09/26/load.html

For most applications, you will have three choices:

1) Don't load-balance an individual user. Once a user hits one server, keep them locked to that server. Any session storage they use will always be available to them because they never leave that server. This method is called "Server Affinity". All the users on the whole system are still load-balanced over multiple servers, but each individual user is locked into one server.

2) Replicate all session data across all servers. No matter which server the user goes to, their data will be waiting for them.

3) Store session data in a central repository that all servers can access. To prevent a single point of failure, this central repository should itself be distributed across multiple servers.

As with all things in technology, there are trade-offs for each of these.

From my own little informal polling of places I've worked at, and people I know, and articles I've read on the web, and people I've met at JavaOne, it seems that most applications use option #1.

However, option #1 can be difficult for your admins to make work 100%.

Option #1 is the most simple and straight-forward from the point-of-view of the application developer. You write your Java applications to use session data any way you choose, and your application will run in a server farm the exact same way it does on your single desktop machine. I love option #1 because it gives us developers a lot fewer plumbing issues to worry about.

The problems with option #1 are as follows: Some ISP services (like AOL) will use dynamic IP addresses on outbound traffic. So, the user's IP address might change as the user moves from page to page, and that makes it difficult for a Load Balancer to figure out that this is the same user that placed the previous request. But a "smart" Load Balancer will also use cookies to determine a user's identity. However, cookies get encrypted if your application runs under SSL (HTTPS). So, you need a "really smart" Load Balancer that can also decode SSL. (Or some kind of other external SSL encoder/decoder box sitting in front of the balancer.) At this point, we are talking about some pricey hardware that takes considerable time to configure. And even after going through all this trouble, this solution provides no fail-over if a server goes down. All users on the crashing server will be lost, because their session data was never replicated to any other machine.

Some admins have told me that even the smartest Load Balancers on the market get it wrong on rare occasion, and lose track of a user's Server Affinity. I can't confirm this from personal experience though. I've had really good experiences on every project I've been on that used option #1.

Option #2 gives you a more robust solution that can allow for true fail-over of all users to other servers. Plus, option #2 doesn't require smart hardware or other products that provide server affinity for a given user. With option #2, you can have regular load balancers.

The problems with option #2 are as follows: The data in your session will likely have to be serializable so that it can be transmitted between servers. This is tricky to accomplish in Java, and it is not easy to test. If any of your objects contain something in their data chain that isn't serializable, your application won't distribute session data on the server farm, and you won't catch that bug until you actually deploy the application to such a farm. Also, there is considerable overhead in maintaining copies of all user data on all servers in the farm. Some Application Server vendors are better and smarter at managing this than others. Your milage will vary.

Option #3 gives you all the robustness and fail-over of option #2, but it offloads the session management tasks to another group of servers.

The problems with option #3 are as follows: In addition to your data needing to be serializable, the mechanism you use to store session data will likely be custom and/or proprietary depending on which Application Server vendor you are using. Saving session data to an external server is not functionality that is part of the Java standard, but is rather something left to the imagination of each developer and/or vendor.

Option #3 is the most robust and complete, but also requires the most amount of plumbing and overhead to accomplish.

For the current project I'm working on, we've been coding all along assuming we'd be using option #1. So we have not paid any attention to making sure our session data was serializable. Well, we finally moved our application to production, and it turns out our Load Balancer doesn't support Server Affinity through SSL. I guess it wasn't clever enough.

Our Server Admin didn't like option #1 anyway, because it's not a 100% solution that supports fail-over. So, we decided to try option #2. Which means I spent a few days re-writing a bunch of our objects that we store in the server session, because these objects contained member objects that were not serializable (and could not be made serializable because they were 3rd-party objects).

But now the application works in a server farm, and we have 100% fail-over. It's really cool! However, this does add a lot of overhead to the servers to replicate the sessions, and it adds a fair amount of overhead to us Java coders who have to be so careful what we stick in the session, and make sure it's really serializable.

And after everything we'd had to do to get this working (and I still have much work to do), I can't help but think that the Java Community should have come up with some kind of more standardized toolkit or solution for dealing with session data. I'm thinking that the HTTPSession object should not even allow a coder to store non-serialized data into the session. Even if your application uses option #1 today, it might need option #2 or #3 in the future, so maybe it's best to start early with making sure all session data is serializable?

Microsoft .NET has the same problem as Java. You have to make your C# objects serializable if you want to use option #2 or option #3.

Despite the effort of Java and .NET to make Enterprise Web Development more simple, there are still quite an array of technical plumbing issues that make Web Development complicated and risky and buggy. And we just have to keep all these issues in mind as we design and code the application, because they are difficult to fix late in the project. :)

Monday, March 24, 2008

Dell Servers Promise More Power for Less Money

Dell Servers Promise More Power for Less Money
By Lauren Simonds
March 18, 2008

Today Dell announced two new members of its PowerEdge server family, the R300 and T300, which are, according to the company, designed to provide high performance at an affordable price.

Dell claims the R300 and T300 are "the industry’s highest-performing one-socket quad-core servers." That claim is based on the servers' processor, a quad-core Intel Xeon x5460, that's typically found in more expensive, higher-performing two-socket servers.

Using the quad-core Intel processor also allows the new servers to offer three times the amount of memory capability of other one-socket servers. "Typically the memory scalability of a one-socket server is 8GB," Sally Stevens, director of PowerEdge servers, said in a teleconference today. "The R300 and T300 servers can scale up to 24 GB of memory."

What does all this socket talk mean to small businesses? Barry Jennings, Dell's chief researcher for small business, said the single-socket R300 and T300 servers "give small businesses the best bang for the buck. It's like having a Ford Mustang engine in a Ford Focus chassis," he said. "No other company offers this. There's no other single-socket server with this chipset and capability on the market today."

In its lab tests, Dell said that the rack-mounted R300 performed up to 26 percent faster than HP's DL320 G5p and up to 51 percent over IBM's System X 3250. Lab testing also showed that the T300 tower server performed up to 31 percent better than HP's ML310 G5 server, and up to 51 better than IBM's System X 3200.

Dell designed the R300 and T300 as general-purpose servers suitable across all industries and markets. "They're capable of running resource-intensive applications, such as Exchange," Jennings said. "You can also run multiple applications on them, such as Exchange plus print-and-file plus databases."

Jennings said that, in addition to faster performance, the benefits small businesses could expect from running the R300 or T300 include increased reliability, decreased heat buildup and increased energy savings. "The rack-mounted R300 is the most energy efficient server – not just single-socket — on the market today. Period."

Other benefits include security features to help SMBs protect their data. They include a TPM or Trusted Platform Module, which Sally Stevens said provides better authentication and encryption capabilities. Internal USB lockdown prevents anyone (including disgruntled employees) from plugging in a 4GB USB key and walking out with your database. Similarly, locking bevels on the servers prevent chassis intrusion.

"The R300 and T300 servers offer increased simplicity by letting small businesses do more with one server," said Jennings. "That translates to saving money with a leaner, cleaner IT infrastructure."

Pricing starts at $1,2249 for the R300 and at $999 for the T300.

Do You Really Need a Network Server?

Hardware & Equipment
Do You Really Need a Network Server?
By Ronald Pacchiano
September 22, 2006

Over the years I have provided computer consulting services for many businesses. While some of these small businesses have been simple one-man operations, many of them were larger. The thing that I always found surprising is that the majority of these companies never bothered implementing a client/server-based network. Instead they just continued adding on workstations to their old peer-to-peer network.

Peer-to-peer networks don't provide you with much in the way of security, and resource sharing can be problematic. You can have problems accessing workstations, lose data due to viruses or spyware or experience intermittent Internet problems. PCs networked in a peer-to-peer fashion are adequate when you only have a few people on the network, but once you have more then five or six people, you really need to consider investing in a network server.

Convincing a small-business owner to make this type of investment can be a hard sell. Unlike large corporations, small businesses don't have the benefit of an IT department and/or the deep pockets necessary to maintain a complex IT infrastructure. However, network servers don't have to be overly expensive or complex for you to benefit from them. And while adding a network is not a trivial or inexpensive undertaking, the benefits you gain far outweigh any shortcomings.

What Exactly is a Server?
Many people are under the misconception that a server is no different from a typical desktop PC. This couldn't be further from the truth. While almost any PC that meets the minimum hardware requirements can run the server operating system, that doesn't make it a true server. A desktop system is optimized to run a user-friendly operating system, desktop applications and to facilitate other desktop oriented tasks. Even if the desktop had similar processor speeds, memory and storage capacity, it still isn't a replacement for a real server. The technologies behind them are engineered for different purposes

A server is designed to manage, store, send and process data 24 hours a day, seven days a week, 356 days a year. Thus, servers need to be far more reliable then their desktop counterparts. To accomplish this, servers offer features not typically found in a desktop PC. Some servers are, do or include:

  • Dual processors (either equipped with or capable of supporting)
  • Redundant hard drives or power supplies
  • Hot-swappable components
  • Scalable to meet current and future needs
  • Process data faster and more efficiently

So now that you know what makes up a server, what can a server do for you? There are multiple reasons to have a fileserver. Some of the more important ones are the following:

  • File and network security
  • Increased reliability
  • Centralized data storage and shared resources
  • Virus management
  • Centralized backup LI>

Let's take a closer look at each of these.

File and Network Security — The most important role of a file server is the network security it provides. By creating individual user and group accounts, rights can be assigned to the data stored on the network preventing unauthorized people from accessing materials they shouldn't view. For example, the people on the sales floor don't need access to employee's personal records. That information is reserved for HR or the company owners.

The server lets you manage file access on one system rather than on each workstation individually; which saves time and effort. Plus if one person's workstation fails, that employee can go to another workstation to continue working on the same files.

Also, everyone can store their documents within their own personal folder on the server. That provides a two-fold benefit. First, each individual is the only person who can see the data stored in that individual's personal folder. Second, since all of your employee data is stored on the network, it gets backed up nightly with the rest of the network data, thus ensuring that nothing will be lost due to a crashed workstation.

Increased Reliability — Servers are designed to run at all times, even in the event of a hardware failure. That's why many servers are equipped with redundant power supplies. Normally, if a power supply dies, the server automatically shuts down, which means lost data and unproductive employees. With a secondary power supply running in tandem, the lost of one of the power supplies doesn't effect normal system operations.

The same goes for a server's storage system. Unlike an average desktop PC that uses a single hard drive, a server will typically use multiple hard drives working in a RAID configuration to prevent data lose or an interruption in workflow due to the failure of a solitary hard disk. There are many different levels of RAID to choose from, and it can be done via either a hardware RAID controller or thru software. The most popular configurations of RAID are RAID-1 and RAID-5.

With the redundant hard drive or power supply engaged, you're still left with the problem of replacing the failed hardware. On a desktop, when any hardware fails you need to shut the system down in order to repair it. This isn't an acceptable condition for a server since whenever the server goes down your employees are unable to function. That's why many servers are also equipped with hot-swappable hard drives and power supplies. Like with the redundant systems, these hot-swappable components let you replace faulty hardware without interrupting the entire office.

Centralized Data Storage and Shared Resources — With a network server, all of the people on the network can make use of various network resources right from their desks, which increases efficiency. Some of these resources include the following:

  • Centralized data storage (RAID array)
  • Network attached storage (NAS) devices
  • CD/DVD towers
  • Printers and fax servers LI>

Virus Management — One of the greatest threats to your network is the possibility of infection from viruses, spyware and spam. So having good, updated, anti-virus software installed on your systems is a necessity. In an office of 10 people or less, systems can be maintained individually. Anything more than that, though, can become a real burden. In those circumstances, an anti-virus package that combines workstation and server virus protection into a single solution makes more sense.

Numerous vendors make anti-virus suites designed specifically around the needs of a small business. A package like this lets you manage every aspect of the anti-virus software from a centralized location; thereby reducing administration and maintenance cost.

From a single PC, the administrator can deploy the latest anti-virus software to each workstation on the network, run network wide virus scans, patch software and update virus definition files. The software is designed to use minimal system resources and run in the background of a client PC, constantly watching for signs of trouble. Many tasks, such as virus sweeps and definitions updates can be automated, giving you security and piece of mind.

Centralized Backup — Storing all of your company and employee data in one location lets you perform backups reliably and quickly. You'll never need to worry about what data is stored on which workstation as you do in a peer-to-peer network. Today you can use almost any media type for backup purposes. In addition to the traditional tape drive, CDs, DVDs, removal storage and even NAS devices are acceptable. Depending on your budget and your data retention needs, any of these options would work well. Make sure you have a scheduled weekly backup (at the very least), although a daily backup would be better.

Bring it all Together
A true server operating system makes all of this possible, and you're going to need to invest in a good one to get the most out of your new hardware. When it comes to choosing an operating system for your server, there really aren't a lot of options — Windows or Linux. A Linux-based OS does a fine job. It's reliable and has modest hardware requirements. No matter which you choose, you'll need a qualified technician to install and configure them properly.

In my opinion, you be better off using a Windows-based operating system like Windows 2003 Server or Microsoft Small Business Server 2003. SBS 2003 is actually a combination Windows 2003 Server and a collection of other Microsoft server products such as Microsoft Exchange and Microsoft SQL Server. Either product makes an ideal choice for a small business.

The majority of the server setup and configuration is automated, requiring only minimal input from the installer. Therefore it can be configured by a person of nominal IT experience, so installation cost shouldn't get to out of hand. Once it has been set up, managing day-to-day functions such as creating user accounts, managing printers, configuring groups, and installing new hardware is pretty straightforward. Anyone familiar with previous versions of Windows will find themselves right at home in Windows 2003 Server.

If you're not running a server in your small business and you have more than five employees, look into the benefits you can gain by doing so. The information provided here only touches the surface of what a proper network can do for you. Options like VPN access, e-mail services, Web hosting and database management will also be available to you once you go down the server road.

A Practitioner's Approach to Performance Testing

Performance Engineering - a Practitioner's Approach to Performance Testing

March 2008

Discuss this Article


Context/Introduction

"The application is horribly slow.", "I don't get the response even after I get my coffee.", "This application is useless". Sounds familiar? How many times have we heard these quotes or or felt like that ourselves? The common thread between these statements is that the performance of the application is not good.

Performance - the (in)famous buzzword. What is it? What does it mean? In this article, we'll touch upon what is involved in testing an application for performance.

With every passing day, organizations are becoming more and more conscious about the performance of their Enterprise Solutions. As the IT industry matures and the technology evolves, so does the awareness about expectations from an Enterprise Application.

Focusing just on the design / implementation and Zero-functional-defect solutions are things of the past. With increasing maturity in technology and IT staff, the 'Non-functional' aspects of the system are fast becoming focus-areas.

So what exactly are the non-functional aspects and/or requirements?

Non-functional requirements (NFRs) tell the IT team, about the kinds of usage and load the application will be subjected to, and the expected response time. We'll go into the details of this "response time" shortly.

NFRs define the Service Level Agreements (SLAs) for the system and hence the overall Performance of the Enterprise Application. Besides performance SLAs, NFRs also cover several other aspects, such as security, but for this article we are concerned with performance related objectives only.

Managing and ensuring the NFRs (SLAs) for an Enterprise Application is called Performance Engineering. Performance engineering is a vast discipline in itself which includes Performance Modeling, Performance Prototyping, Performance Testing, different types of analyses, Performance Tuning, etc. This article will not explain Performance Engineering, Queuing Theory and the science behind the various laws. This article just covers the basics about the Performance Engineering and key activities in Performance Testing.

How to describe Performance of a 'System'

The performance of any system can be expressed in many ways by different stakeholders of the system. For example, When a business analyst gives performance (or non-functional) requirements, it might be in some format as follows:

  1. "there will be at least 100 users on the system all the time",
  2. "System should respond back in 'acceptable time'".

These statements have to be translated to somewhat more technical terms as described below. Once we understand these terms, we'll reword these Performance requirements.

To define the performance of any System (Software/hardware/abstract) following technical parameters should always be used in conjunction -

  • Response Time: Response time specifies the time taken by the system to respond back with the expected output. For an enterprise application Response time is defined as the time taken to complete a single business transaction. It is usually expressed in seconds.
  • Throughput: Throughput refers to the rate which the system churns expected outputs when the designated input is fed in the system. In other words, for an Enterprise Application, throughput can be defined as the total number of business transactions completed by the application in unit time (per second or per hour).

Usually, per second or per hour is the standard, since per day (or per diem) is a very wide unit. Most business users utilize an Application during typical 8 hour business window. Normally there are some peaks & some troughs in the input, so the volumes per day should not be averaged to an hour. All the mathematical distributions (normal, Poisson, uniform) come handy over here.

  • Resource Utilization: For any system to consume an input and produce the designated output, certain resources would be required. The amount of resources consumed by the system during processing that request, defines the resource utilization. There can be different resources factored in, such as processor, disk (i/o controller), memory etc. Utilization of 80% is considered an acceptable limit. Normally utilization of 70% warrants ordering of additional hardware.

Now that we know basic technical terms for expressing performance facts about a system, we'll try to rephrase the above Performance Requirements.

  1. We should ask the question whether all the users would be carrying out a transaction simultaneously or they would be just logged on. The answer of this question would lead us to the throughput expectations. In case of pre-existing applications (or using applications having similar profile), we can also arrive at the transactions/sec or throughput requirements from logs files (e.g., HTTP logs of web-applications.)
  2. The vagueness of "acceptable" times needs to be removed by defining the response time requirements clearly, e.g. 2 seconds, 2 hours or 2 days.

Let's begin then...

Performance test planning

He who fails to plan, plans to fail

As in all aspects of life, planning is very important in Performance Testing. We present the simplistic approach to planning a Performance testing exercise in the sub-sections below.

How to represent the SLAs? - Workload model/distribution/pattern

The first step of any IT project is usually requirements gathering. Similarly, for any performance testing project, its very important to gather the SLAs / NFR of the system against which the tests will designed and executed.

Performance testing depends as much on how well the SLAs are gathered as much as on how well they are represented. A well represented non-functional requirement can help a long way in the rest of the planning and analysis activities.

The various throughput rates, load figures, list of transactions, types of transactions, response times and projected growth in coming years are captured in a document called the Workload. Workload captures the load pattern, load distribution, transaction distribution, peak windows etc.

The workload should be thoroughly reviewed by the various stakeholders of the system, like, architects, business users, analysts, designers and performance engineers.

What to test? - Identifying Transactions

Once the Workload model has been prepared and reviewed thoroughly, the next step is to Identify candidate transactions. If we select too few transactions, the system might contain a serious SLA mismatch and if we select all the transactions, we might be heading towards a never ending performance test cycle.

The famous 80-20 rule comes in handy here. In most applications, 20% of the transactions cover 80% of application's core functionality (to be confirmed with the help of application experts). Once the workload model has been created, this step is fairly easy and straight forward.

It's also important to classify the transactions as the Performance Testing would be carried out differently for each type. The transactions could be very broadly categorized as online - where user submits a request and gets a response, and batch - where user submits a list of jobs and does not wait for the completion. The transaction mix also has to be identified, viz. transaction A has 50% requests, transaction B has 30%, and transaction C has 20%. All of A, B, C can occur together. This helps in deciding the mixed tests.

Also at times, there might be certain transactions which do not qualify as candidate transactions according to the 80/20 rule, but might be critical to business. Such transactions should also be considered for performance tests and all such transactions can be clubbed in one run to get a benchmark reading for analysis.

Where to perform tests? - Environment Setup & Planning

For Performance Testing, it is ideal to get an environment with the same capacity as the production/live environment. Many developers are of the opinion that one deliverable/outcome of performance testing/engineering exercise is to provide hardware projection for production use, which is perfectly right. Capacity projection and Production hardware gauging is altogether a vast topic. So for simplicity sake, we restrain ourselves to Production like environment (considering an earlier release of the application is already live on some hardware).

Besides the deployment hardware, there are many other pieces of hardware that would be needed to facilitate load testing. These can be broadly classified as load generation nodes, stub nodes and monitoring.

Load generation nodes are the machines used to generate load for the applications. These nodes are used to host the load generating software, such as LoadRunner, WinRunner, etc. Some open source alternatives are also available. Some custom scripts will have to be written which can run on these nodes to generate load on the system. These tools also help in monitoring the resource utilizations.

However it is not mandatory to use only these tools. Depending on the complexity some custom stubs may be written to do the testing.

Monitoring nodes usually can be the same as load generating nodes, since most of the load-generating software packages are also equipped with monitoring capabilities. Monitoring is an inherent aspect of Performance Testing. All the resource utilizations need to be monitored throughout the duration of the test to

  • ensure all the utilizations are within limits
  • identify the bottlenecks
  • The rate at which transactions are completing
  • Number of failed transactions...etc...

Stub nodes are the ones which will be usually hosting a stub to simulate some part of the software or external system that can not be included in the performance test runs - such camera or scanner devices that are operated manually, which wouldn't be a valid option for a repeatable performance test.

Data Preparation

Any enterprise system is usually programmed to process data. Thus, data planning for the performance testing of the system is a vital step. This directly affects the success of the Performance Testing exercise.

Data required for load tests can be broadly categorized as -

Initial/Retention Data - The performance of database is very much dependent on the amount of data present in the tables. When the application is in production, there is certain amount of data present in respective tables. While carrying out performance testing, it is necessary to have at least equal amount of data in the Test environment. In fact, depending on the system upgrade plans, the data in the test environment could be a multiple of production data.

Having same amount of data is one thing, and using the same distribution of data as in production is another. The distributions of data straight away affect the index performance and in turn the application performance.

The amount of data and its distribution should be validated by business and/or IT dept. before proceeding with Performance Testing.

Test Data - Once the retention data is fine, we turn our attention to test data. The same concepts as amount and distribution apply to test data as well. This time, we have to take concurrency into consideration too.

Designing scenarios

Any testing activity is comprised of the test cases and test-suite encompassing those tests. For performance testing there are some standard tests which are usually conducted on the candidate transactions of the system, depending on the nature of those transactions. The following subsections give a brief insight in each of those test scenarios.

Isolated runs

"Isolated" means running one transaction at a time. By running a single transaction and observing its behavior, we can see if it is able to utilize the hardware well. We can also see if it is able to meet its throughput requirements by running alone at least. The readings taken can also help in capacity planning (part of advanced Performance Engineering).

Usually isolated runs are preceded by Baseline / Calibration runs, where in just a single run of each transaction is executed to get the benchmark response times for all candidate transactions.

Mixed runs

"Mixed" means running all/multiple transactions simultaneously. This helps in identifying if all the transaction throughputs are met when multiple transactions are occurring parallely, i.e., it also tells us the effect of transactions on each other.

There are two schools of thought in the order of execution of these tests.

  • First, run mixed test in the beginning. If everything is fine, then we are saved efforts for isolated tests.
  • Second, run isolated tests in the beginning followed by mixed test. This is more useful when you want to determine the scalability of the application and do any capacity planning.

Trickle feed

This is mainly applicable in case of batch type of loads - e.g. the load may come as 'n' messages per five minute block.

Backlog runs

This is a typical batch load. Sometimes it might happen that the application receiving messages (say Application A) is down for long time, but the feeding application (say Application B) is continuously putting messages. Now when the application A comes up, how it behaves, can be found out by this testing.

Endurance

It has been observed that whenever a system is running for multiple days or months without downtime, its memory utilization increases or throughput falls or some exceptions occur. To find out if the application behaves fine even if run for longer duration, this test is used.

Stress

Sometimes, as the load on the system increases, e.g. on Christmas Eve for retail or shipping applications, the system throws errors or behaves unexpectedly. The stress test targets exactly this behavior. Determining that the software handles the load gracefully without crashing is the aim of this test.

Preparing Reports

Presenting results in an understandable format is normally a neglected area. If you don't do that, you are reaping only 20-30% benefits of Performance Testing. Properly presented results will enable business stakeholders to make informed decisions. Also, when it is time to do Performance Testing for the next version of the same application and if you don't have properly documented results (and if the people who did the exercise are gone), then you need to start all over again.

Key graphs

Here are some key graphs that should be created while executing Performance Testing and should be presented in a Performance Test Report.

CPU Utilization vs. No. of users

This graph tells us whether the CPU utilization increases with increasing users. It also tells us about bottlenecks as well as what is the maximum number of users that can be supported.

Throughput vs. number of users

This graph tells us if throughput increases with increasing number of users. This graph should have a similar shape as CPU Utilization graph.

Response Time vs. throughput

The response time should be fairly constant with increasing throughput. If it is not, it is most likely a bottleneck.

Besides the above mentioned graphs, depending on the nature of transactions and criticality of various components, a few other graphs can also be included in the report, such as network bandwidth & latency graphs, memory utilization graphs, disk / i-o utilization graphs for disk/i-o intensive operations, etc.

Best Practices

Automation with handy scripts

Script writing is an inherent part of any Performance Testing exercise. A lot of time is spent in ensuring the begin state of the test is proper. It might involve restarting servers, deleting extra data other than retention data, cleaning up log files, taking backup of some data, checking if some pattern exists in a log file, etc. Since each test needs to be run for 'n' number of times, automating these small tasks goes a long way in reducing time needed for each run.

Identify DB backups after crucial runs

During the course of Performance Testing activity there are bound to be different patches on application, database side. These changes are seemingly trivial but potentially devastating. It is a good idea to take the backup of data in the database and configuration files for the application before applying new patches to either application or DB.

Cross check Little's law

One of the basic queuing theory principles applied in SPE (Software Performance Engineering) is Little's law (a). It states that the total number of users in the system is equal to the product of throughput and response time. While calibrating your load test runs, one should always cross check the test results (throughput, response time, user-load simulated) to identify whether the load generator nodes themselves are not becoming the bottleneck in the system.

Designing scenarios to achieve right mix of transactions

Do not tune, if you don't have to.

There is a thin line separating Performance testing from Performance tuning. Unless really required, the tuning of various parameters of different components in the system should not be done. You will find many articles, which warn against tuning the systems. Most of the system components come with default values that are optimal. If we start looking for tuning of all the components, the duration of performance testing exercise will no longer be under control.

People Aspect

Performance Testing is a very strategic activity in the lifecycle of any Application. More often than not, you will have to deal with stakeholders who are apprehensive about the need of this activity, some people (especially DB, OS admins) who don't want the extra load this activity puts on them, software vendors who don't want additional defects on them, and many more. For the success of Performance Testing, it is important to take all these people along & explain (or thrust as the case might be) the inevitability of this exercise & how life would be easier when the application goes in production.

Environment availability is the biggest constraint which might force working in shifts, long hours, etc.

References

Authors

Alok Mahajan is Technical Architect with Infosys Technologies and specializes in Performance Engineering and Database designs. He has a total experience of more than7 years and has worked extensively on Performance Engineering assignments for more than 2 years, which uniquely positions him to write on the subject of performance. His areas of interest include Business Intelligence, Databases, Datawarehouse design/modeling.

Nikhil Sharma is Technical Architect with Infosys Technologies. For more than7 years, he has been involved in architecting and designing Enterprise Applications primarily on J2EE platform. For past2 years he has been working on Performance Engineering projects. He is also a Sun Certified Enterprise Architect for J2EE Platform (SCEA). His areas of interests are SOA, Integration Architectures, Portals.


PRINTER FRIENDLY VERSION

MySQL InnoDB Performance Tuning for the Solaris 10 OS


Article

MySQL InnoDB Performance Tuning for the Solaris 10 OS



By Luojiia Chen, Updated February 2008
Abstract: You can maximize the performance of MySQL on the Solaris platform through configuration and tuning of the database server, along with optimizing the Solaris OS for MySQL. This paper is intended to help you define tuning parameters and tune them in your environment.
Contents

- Introduction
- InnoDB User Threads
- Using the Optimized Time Library to Minimize the time(2) System Call
- Using the Solaris OS mtmalloc Memory Allocator in MySQL
- Using 64-bit MySQL on the Solaris Platform
- Building MySQL With Sun Studio 11 Software
- Optimizing File System Performance
- InnoDB Data and Index Cache Size
- Transaction Log Flushing Mode
- Log Buffer Size
- Checkpoint Operation
- Query Cache Size
- Conclusion
- Resources
- About the Author

Introduction

MySQL is one of the world's most popular open source databases. A key strength of the MySQL database is its excellent performance and scalability, making it well suited to serve as a back end for web sites, data warehousing, and other data-intensive applications in the enterprise environment.

To maximize the performance of MySQL on the Solaris Operating System, database server configuration and tuning are important, as well as the optimization of the Solaris OS for MySQL. However, no universal MySQL server tuning parameters apply to all workloads on all platforms; the appropriate parameters will depend on your particular workload, hardware and OS platform, and MySQL usage. This paper is intended to help you define these parameters and tune them in your environment.

MySQL includes several storage engines, including MyISAM, InnoDB, HEAP, and Berkeley DB (BDB). InnoDB and BDB storage support Atomic, Consistent, Isolation, and Durable (ACID) transactions with commit, rollback, and crash recovery capabilities, and only InnoDB storage supports row-level locks with queries running as non-locking consistent reads by default.

The InnoDB storage engine supports all four isolation levels: read uncommitted, read committed, repeatable read, and serializable. InnoDB also has the feature known as referential integrity with foreign key constraints support, and it supports fast record lookups for queries using a primary key. Because of these and other powerful functions and features, InnoDB is often used in large, heavy-load production systems. This paper covers the different ways to improve the use of CPU, memory, and disk drive resources for the Solaris 10 OS. Topics include using optimized libraries, compiling 64-bit MySQL with Sun Studio 11 software, tuning the Solaris UFS file system, and MySQL server configuration and tuning for the InnoDB storage engine on the Solaris platform.

InnoDB User Threads

MySQL is a single-process, multithreaded application. One master thread has the highest priority among all the MySQL threads to control the server. The main thread is idle most of the time and “wakes up” every 300 milliseconds (msec) to check whether an action is required, such as flushing dirty blocks in the buffer pool.

In addition to the main threads, a dedicated set of user threads run at normal priority in the thread pools to handle simultaneous client requests. For each client request, one single thread is created to process the client request and send back the result to each client once the result is ready. And there is one single user thread that waits for input from the console and a group of utility threads running at lower priority to handle some background tasks.

Right now, MySQL cannot scale well with the number of the user threads handling client requests. Performance scales up efficiently with each additional user thread until it reaches the peak performance point. After that, increasing the number of user connections will decrease MySQL performance because of thread concurrency contention. For applications where the number of user connections is tunable, you need to determine -- for different workloads -- the optimum number of user connections for peak performance.

We ran the SysBench CPU-bound benchmark test (1M-row data can be filled in the InnoDB data and index cache buffer) on a four-way UltraSPARC IV processor-based server. MySQL performance peaked at 16 user connections, and started to drop from 32 user connections, as shown in the following graph. (Please note: Results may vary.)



Figure 1: MySQL 5.0.7 SysBench Connection Scalability Test
Figure 1: MySQL 5.0.7 SysBench Connection Scalability Test

This test shows that for the SysBench workload, peak MySQL performance on the UltraSPARC IV processor-based server can be achieved by setting the number of user connections as 4*CPUs on a 1-4 CPU system. The peak MySQL performance scales in a nearly linear fashion up to 4 CPUs, and the scalability ratio starts to drop from 8 CPUs. The following figure shows the scalability test result using systems with 1-24 UltraSPARC IV processors. (Please note: Results may vary.)



Figure 2: MySQL 5.0.7 SysBench CPU Scalability Test
Figure 2: MySQL 5.0.7 SysBench CPU Scalability Test

For applications where the number of user connections is not tunable, the innodb_thread_concurrency parameter can be configured to set the maximum number of threads concurrently kept inside InnoDB. You need to increase this value when you see many queries in the queue in show innodb status. In MySQL versions below 5.0.8, set this value over 500 can disable the concurrency checking, so there will be as many threads concurrently running inside InnoDB as needed to handle the different tasks inside the server. This variable changed from MySQL 5.0.8: setting it equal or greater than 20 will disable the concurrency checking; and it changed again from MySQL 5.0.19 and above: you will need to set innodb_thread_concurrency to be 0 to disable the concurrency checking. In some workloads running on the Solaris platform, when you see a large user level lock (LCK in the prstat –mL output), reducing this parameter increases the efficiency of CPU usage to improve overall performance. Tuning this parameter according to the behavior of your system at runtime can affect performance significantly.

Using the Optimized Time Library to Minimize the time(2) System Call

On the Solaris platform, the time(2) system call actually takes a trap to the kernel to call gethrestime(), which is expensive and time consuming. When MySQL executes each query, it performs a time(2) system call at the start and at the end to measure how long the query takes. For some workloads, MySQL can spend more than 30 percent of system time on the time(2) system call, consuming a large amount of system CPU cycles as illustrated below:

# truss -c -p 385
syscall seconds calls errors
read 28.286 450958 3248
write 19.516 231648
open .000 2
close .000 2
time 45.247 848307
lseek .329 6878 6800
alarm .140 2218
fdsync 1.140 5520
fcntl .364 6510
lwp_park 12.288 187383
lwp_unpark 11.134 187381
poll 4.535 67263
sigprocmask 2.072 36030
sigtimedwait .381 2506
yield .741 9829
lwp_kill .201 2512
pread .000 2
pwrite 1.040 5527
-------- ------ ----
sys totals: 127.447 2051007 10048
usr time: 101.288
elapsed: 123.750

On the Solaris platform, we can optimize the time(2) system call by implementing the faster gethrtime(3C) system call instead of calling gethrestime() in the kernel. If your workload is spending lots of system CPU resources on the time(2) system call (you can verify this by checking the truss output), it may make sense to re-compile the MySQL DB by linking it to the optimized time library described in A Performance Optimization for C/C++ Systems That Employ Time-Stamping. Do this by adding LIBS='-lfasttime' at the beginning of the configure file under the MySQL source tree home to implement gethrtime(3C), or you can set LD_PRELOAD (32-bit) or LD_PRELOAD_64 (64-bit) to point to the location of the libfasttime.so library. We ran the OSDL Database Test 2 (DBT2) workload test and improved the MySQL performance by about 7 percent by using the optimized time(2) in the 10-warehouse workload test on an 8-way UltraSPARC system.

Using the Solaris OS mtmalloc Memory Allocator in MySQL

The malloc routine on the Solaris platform behaves rather well from a memory footprint perspective; however, the default single-thread malloc in libc handles the concurrent memory allocation requests one by one in a queue, slowing down performance in multithreaded applications. The Solaris OS offers several malloc implementations, including mtmalloc, libumem, and hoard to improve memory allocation performance for multithreaded applications. However, it can be difficult to tell which malloc implementation is best for different applications as that depends on the memory allocation pattern of the application and how the different algorithms in the different malloc implementations fit the memory allocation pattern of the application.

MySQL actively uses malloc() and free() to allocate memory for unexpectedly large strings. The malloc calls block mysqld threads for HEAP contention. By measuring with different memory allocators, replacing malloc with mtmalloc can significantly improve MySQL DB performance. We saw about a 65 percent improvement in the SysBench CPU-bound benchmark test (1M-row data can be filled in the InnoDB data and index cache buffer). We ran this test on an 8-way dual-core UltraSPARC IV processor-based system with MySQL 5.0.7.

To use mtmalloc, you can preload the mtmalloc library by setting the LD_PRELOAD or LD_PRELOAD_64 environment variable in the MySQL startup script on the Solaris platform, so that you do not have to rebuild the MySQL DB to link to the mtmalloc library.

If it is 32-bit MySQL, preload mtmalloc by setting LD_PRELOAD as follows:

LD_PRELOAD=/usr/lib/libmtmalloc.so (x86)
LD_PRELOAD=/usr/lib/libmtmalloc.so(sparc)

If it is 64-bit MySQL, set LD_PRELOAD_64 as follows:

LD_PRELOAD_64=/usr/lib/amd64/libmtmalloc.so(x64)
LD_PRELOAD_64=/usr/lib/sparcv9/libmtmalloc.so(64-bit sparc)

MySQL 5.0 GA binary for Solaris OS released by MySQL has been built to use libmtmalloc with "--with-mysqld-libs=-lmtmalloc" configuration flag, so you will not need to set the LD_PRELOAD or LD_PRELOAD_64 environment variable to use mtmalloc.

Using 64-bit MySQL on the Solaris Platform

The Solaris OS is a full 64-bit computing environment providing binary compatibility with 32-bit applications, so that both 64-bit and 32-bit MySQL can run well on the Solaris platform. Compared to 32-bit MySQL, 64-bit MySQL can address more memory for the data cache, code cache, and metadata caches inside MySQL to reduce disk I/O. In addition, with wider CPU operations in the 64-bit computing environment, 64-bit MySQL operates faster than 32-bit MySQL. The following figure depicts the performance data in the SysBench CPU-bound test (1M-row data can be filled in the InnoDB data and index cache buffer) on an 8-way UltraSPARC IV based server. (Note: Results may vary.)



Figure 3: MySQL 4.1.11 CPU-Bound SysBench Test
Figure 3: MySQL 4.1.11 CPU-Bound SysBench Test
Building MySQL With Sun Studio 11 Software

To build 64-bit MySQL on the Solaris platform using the Sun Studio 11 release, use the compiler flags and configuration options as shown here:

With the Solaris OS for x64 platforms:

CC=cc CFLAGS="-xO4 -mt -fsimple=1 -ftrap=%none -nofstore
-xbuiltin=%all -xlibmil -xlibmopt -xtarget=opteron
-xarch=amd64 -xregs=no%frameptr"
CXX=CC CXXFLAGS="-xO3 -mt -fsimple=1 -ftrap=%none -nofstore
-xbuiltin=%all -xlibmil -xlibmopt -xtarget=opteron
-xarch=amd64 -xregs=no%frameptr"
LDFLAGS="-xtarget=opteron -xarch=amd64"
./configure --prefix=/usr/local/mysql
--localstatedir=/usr/local/mysql/data
--libexecdir=/usr/local/mysql/bin --with-extra-charsets=complex
--with-server-suffix=-standard --enable-thread-safe-client
--enable-local-infile --with-named-curses=-lcurses
--with-big-tables --disable-shared --with-readline
--with-archive-storage-engine --with-innodb

With the Solaris OS for SPARC platforms:

CC=cc CFLAGS="-xO4 -mt -fsimple=1 -ftrap=%none
-xbuiltin=%all -xlibmil -xlibmopt -xstrconst -xarch=v9"
CXX=CC CXXFLAGS="-xO3 -noex -mt -fsimple=1
-ftrap=%none -xbuiltin=%all -xlibmil -xlibmopt
-xarch=v9"
./configure --prefix=/usr/local/mysql
--localstatedir=/usr/local/mysql/data
--libexecdir=/usr/local/mysql/bin
--with-extra-charsets=complex
--with-server-suffix=-standard
--enable-thread-safe-client --enable-local-infile
--with-named-z-libs=no --with-big-tables
--disable-shared --with-readline
--with-archive-storage-engine --with-innodb

To compare the performance of the MySQL database built with different compiler releases (Sun Studio 10 and Sun Studio 11), we used the DBT2 test suite. The workload represents a wholesale parts supplier operating out of a number of warehouses and associated sales districts. The tasks involve the mixture of read-only and update-intensive transactions to enter and deliver orders, record payments, check the status of the orders, and monitor the level of stock at the warehouses. The nine tables include Warehouse, District, Item, Stock, Customer, Order, New order, Order-line and History, scaling with the number of warehouses (except the Item table).

We ran the DBT2 test using the 10-warehouse database on the Solaris 10 OS on four dual-core 2200-MHz AMD Opteron processor-based Sun Fire x64 servers. In this test case, most database queries were cached in the innodb buffer. Most of the CPU time was spent on processing the queries, so the system was CPU bound during this test. The test metric was the throughput, that is, new-order transactions per minutes. As shown in our test result data in the following figure, MySQL built with Sun Studio 11 performed around 13 percent better than MySQL built with Sun Studio 10 software. (Note: Results may vary.)



Figure 4: MySQL 5.0.15 DBT2 on Solaris 10 OS
Figure 4: MySQL 5.0.15 DBT2 on Solaris 10 OS

In addition to the optimized performance improvement for the MySQL database that Sun Studio 11 offers compared to Sun Studio 10 in the x64 architecture-based system, Sun Studio 11 also provides the multicore and chip multithreading (CMT) optimizations in the UltraSPARC processor-based system. The Sun Studio 11 release also includes an advanced graphical debugger tool to easily set breakpoints, examine variables, navigate the call stack, and debug the multithreaded code inside MySQL. The sophisticated performance analysis tool in Sun Studio 11 software provides additional data space profiling ability on the UltraSPARC processor-based system to evaluate the performance costs associated with the application memory references. The detailed instructions on how to use these new features in the debugger and performance analyzer tools in Sun Studio 11 software can be found in the comprehensive user guide.

Sun Studio 11 software also bundles the dmake tool, which can compile MySQL source code in parallel. dmake can significantly improve compilation performance compared to make on a multiprocessor system. On an 8-core Sun Fire T2000 system with T1 processors, it took only 9 minutes to compile MySQL source code using dmake -j 64 -- this was over 3 times faster than using make, which took 29 minutes.

Optimizing File System Performance

File system cluster size can have a big impact on system performance -- particularly when MySQL is running a workload with a database size much bigger than system memory. On the Solaris platform, UFS file system cluster size (the maxcontig parameter) is set as 128 by default. The file system block size on the Solaris 10 OS for SPARC platforms (x86/x64) is 8 Kbytes. You can get the value of maxcontig and bsize using the mkfs –F or fstyp –v command on the file system. This will trigger read-ahead for the whole file system cluster length (128*8 Kbytes), even in random I/O, which can saturate a disk and significantly degrade performance.

One way to solve this problem is to reduce the value of the maxcontig parameter so that the disk I/O transfer size matches the DB block size. You can change the maxcontig value by using the tunefs –a maxcontig# command on the file system. The shortcoming of this solution is that it may impact the performance of other workloads running large sequential I/O from the client.

Another solution is to enable file system Direct I/O by mounting the file system with the option --forcedirectio, since file system Direct I/O will automatically disable the read-ahead. In addition, since MySQL has its own data and cache buffers, using Direct I/O can disable the file system buffer to save the CPU cycles from being spent on double buffering. The following figure shows performance data on a Sun Fire V65x server using the SysBench I/O bound test (100M-row data cannot fit in the InnoDB data and index cache buffer). This test compared the performance using default maxcontig, setting maxcontig as 5 (the disk transfer rate size is 5*4 Kbytes), and using Direct I/O. (Note: Results may vary.)



Figure 5: MySQL 4.1.11 I/O-Bound SysBench Test
Figure 5: MySQL 4.1.11 I/O-Bound SysBench Test
InnoDB Data and Index Cache Size

MySQL doesn't access the disk directly; instead, it reads data into the internal buffer cache, reads/writes blocks, and flushes the changes back to the disk. If the server requests data available in the cache, the data can be processed right away. Otherwise, the operating system will request that the data be loaded from the disk. The bigger the cache size, the more disk accesses can be avoided. The default value of 8 Mbytes is too small for most workloads. You will need to increase this number when you see that %b (percentage utilization of the disk) is above 60 percent, svc_t (response time) is above 35 msec in the iostat –xnt 5 trace output, and a high amount of read appears in the FILE IO part of the show innodb status output.

However, you should not set the value of the innodb_buffer_pool_size parameter too high to avoid the expensive paging for the other processes running without enough RAM, because it will significantly degrade performance. For systems running on a single dedicated MySQL process, it should be fine to set the innodb_buffer_pool_size parameter up to a value between 70 and 80 percent of memory since the footprint of the MySQL process is only around 2 to 3 Mbytes.

Transaction Log Flushing Mode

InnoDB flushes the transaction log to disk approximately once per second in the background. As a default, the log flushes to the disk on each transaction commit. The safest way to avoid transaction loss in the event of MySQL, OS, or hardware crashes is to use the mode innodb_flush_log_at_trx_commit = 1.

For workloads running with many short transactions, you can reduce disk writing by setting the innodb_flush_log_at_trx_commit parameter with different values.

When setting this value at 0, there is no log flushing on each transaction commit. It can reduce the disk I/O to improve performance, however, the transaction might be lost if MySQL crashes.

When setting this value at 2, the log flushes to the OS cache (file system cache) instead of the disk on each transaction commit. It can also reduce the disk I/O and perform a little slower than when the value is set at 0; however, no transaction loss occurs in the event that MySQL crashes (although a loss will probably occur if the OS or hardware crashes).

Log Buffer Size
For large transactions, before the log buffer is flushed on each transaction commit, if your setting innodb_flush_log_at_trx_commit is at 1, the log can be loaded into the log buffer instead of flushing to the disk in the background to reduce the disk I/O. If you see large log I/O in the show innodb status output at runtime, you probably need to set a larger value for the innodb_log_buffer_size parameter. For most workloads without long transactions, it is not necessary to waste memory resources by setting a higher value for the log buffer. It is usually fine to set it between 8 and 64 Mbytes.

Checkpoint Operation

The recovery management subsystem inside InnoDB flushes database pages and transaction operations to the log files for the purpose of backup and recovery. It implements a fuzzy checkpoint operation by continually flushing modified database pages from the buffer pool in small batches. InnoDB writes to the log files in a circular fashion, so if the log file has reached the configure limit set by the innodb_log_file_size parameter, the checkpoint operation is executed at once to flush the modified database pages. This is done in order to make sure the committed modification pages are available in the log files in case of recovery.

The size of each log file should be chosen to avoid executing checkpoint operations too often. The bigger log file size reduces disk I/O in checkpointing. You will need to increase this parameter when you see large page writes in the BUFFER POOL AND MEMORY part of the show innodb status output. However, the larger size of the log file increases the “redo” recovery time in case of a server crash.

Query Cache Size

In addition to the data and index buffer cache, MySQL version 4.0.1 and later has a nice feature called query cache that stores the identical SELECT queries issued by clients to the database server. This makes it possible to locate and re-issue the same queries without repetitive hard parsing activities. MySQL also stores the query's result set in the query cache, which can significantly reduce the overhead of creating complex result sets for queries from the disk or memory caches, reducing both physical and logical I/O. For some applications executing the same queries from user clients, the query cache can greatly improve the response time.

The query_cache_size parameter is used to allocate an amount of memory to cache the frequently executed queries and return the result set back to the client without real query execution. The query_cache_type parameter is used to enable or disable query cache in different ways. To decide how to set these two parameters, you need to check the qcache_inserts, qcache_hits, and qcache_free_memory during runtime. qcache_inserts shows the number of queries added to the query cache, qcache_hits shows the number of query results taken from the query cache without real query execution, and qcache_free_memory shows the free available memory for query cache not being used.

If you see a high value for qcache_hits compared to your total queries at runtime or a low value for qcache_free_memory, you probably need to increase the value of the query_cache_size parameter accordingly. Otherwise, you would decrease the value of the query_cache_size parameter to save memory resources for the other MySQL cache buffers. If qcache_hit is 0 in the runtime, you would completely turn off the query cache by setting query_cache_type as 0, together with setting query_cache_size as 0.

The query_cache_limit parameter sets the maximum result sets stored in the query cache. A low ratio of qcache_hits to qcache_insert at runtime may also be caused by too low a setting for the query_cache_limit parameter. In this case, you will need to increase the value of the query_cache_limit parameter to store the large query result sets in the query cache.

Conclusion
The outstanding performance and scalability of MySQL can be enhanced even further by tuning the storage engines within MySQL for specific workloads running on Sun systems. While there are many variables that impact performance and many parameters for tuning for each workload, this paper has attempted to provide general guidelines and practical advice to help you optimize the performance of InnoDB on the Solaris platform. We welcome your feedback on this paper and your insights into achieving higher performance for MySQL on Sun systems.

Resources
About the Author
Luojia Chen is a software engineer in Sun's Market Development Engineering organization, in the open source team. She is currently responsible for MySQL adoption and migration for Sun's latest technologies, and she is focused on making MySQL run and scale well on the Solaris platform. She can be reached at luojia.chen@sun.com.