The New Builder Pattern

The New Builder Pattern

The idea

I like to create immutable objects, especially after reading Josh Bloch's excellent "Effective Java" book. If an object is immutable, it has only one possible state and it is a stable one, so once you successfully build an object, you don't need to care about state transitions that can make your object unstable or corrupted. And immutable objects can be shared even in a multithreaded application. There are many other pros of immutability (you can read some of them here).

There is a classical way of making immutable objects in Java which consists of making all fields final (and private, of course), using only constructors to modify them (so that the only moment when a field is modified is during its construction) and making the class final (to avoid adding "setter" methods to subclasses). When you only have a couple of fields, that's fine, but when you have many of them you end up with a constructor with many arguments, which is ugly and difficult to use. If you have optional parameters, you can have a constructor with all the parameters and some other shorter constructors that have the mandatory parameters and some optional ones, that invoke the big constructor, like this:

public class Foo {

private final String mandatoryOne;
private final String mandatoryTwo;
private final String optionalOne;
private final String optionalTwo;

public Foo(String mOne, String mTwo, String optOne, String optTwo){
this.mandatoryOne = mOne;
this.mandatoryTwo = mTwo;
this.optionalOne = optOne;
this.optionalTwo = optTwo;
}

public Foo(String mOne, String mTwo, String optOne){
this(mOne, mTwo, optOne, null);
}
...
}

This can be a bit messy when you add more optional parameters, you end up with a lot of constructors like these and it has a lot of boilerplate code.The use of setters for the optional parameters is not an option, because this leads to non immutable objects (some object can change the state of your object with one of those setter methods).
Some time ago, thinking about this problem, I thought a solution could be to use a Javabean object, with one setter per field (even for the mandatory ones), but with a kind of "seal" method, that would "mark" the object as built and since that moment, an IllegalStateException would be thrown if a setter was called. Nevertheless, I wasn't very satisfied with this approach, because the setter methods that sometimes can be called and sometimes not would be confusing for the caller.

Then I found the New Builder pattern, explained by Josh Bloch in this PDF presentation, which is different from the original GoF Builder pattern. This pattern uses a public inner static class as a builder. The constructor of the original class is made private, so the only way to build objects is with the inner builder class. The builder has a setter method for each optional parameter and uses a fluent idiom that allows chaining of these method calls. I like this pattern a lot, because it solves the problem elegantly and effectively.

The implementation

In Josh Bloch's presentation there wasn't a detailed implementation of the pattern, although it was very clear the idea and the intention so I have searched for it in the Internet.

In Richard Hansen's blog you can find an implementation that seems to be more close to what Josh explains: the builder is a static nested class of the class from which it has to make instances, the builder's constructor is public (so you invoke the builder with 'new'), and the builder has the same fields as its enclosing class. The 'build()' method copies the content of the builder's fields into a new instance of the enclosing class. What I don't like about this implementation is this duplication of fields (for each field in the original class you have a duplicate field in the builder).

In Robbie Vanbrabant's blog there is a variation of this pattern, which avoids the boilerplate code using a base class for the builder and some reflection to build the object from the builder. I don't like the use of an interface for the builder, because that way you can't add a new optional parameter without breaking existing code that uses the builder (if you change the signature of a public interface the classes that use it have to change their code to implement the new methods). Also, I don't like the use of reflection because it's slower than the normal access to fields, but I do like the way it avoids duplication of fields in the builder.

The implementation I like most is the one found in Mario Hochreiter's blog. The builder is a nested public static class, but it changes the fields of its enclosing class directly, it doesn't use duplicates. It doesn't use reflection and the builder is a class, not an interface. The only problem I see is that, in theory, with a reference to a builder, you can change the state of the object it built, so you don't have the guarantee that the object is immutable. So I would add a check before each "setter" of the builder that would throw an IllegalStateException if the object has been already built and a check before the 'build' method itself to ensure the object is not built more than once. Also, I would make the mandatory parameters final.

So, with the example of Mario, I would implement this pattern this way:

public class ID3Tag {

private final String title;
private final String artist;
private String album;
private int albumTrack;
private String comment;

public static class Builder {

 private boolean isBuilt = false;
 private ID3Tag id3tag;

 public Builder(String title, String artist) {
   id3tag = new ID3Tag(title, artist);
 }

 public Builder album(String val) {
   if (isBuilt){
     throw new IllegalStateException("The object cannot be modified after built");
   }
   id3tag.album = val;
   return this;
 }

 public Builder albumTrack(int val) {
   if (isBuilt){
     throw new IllegalStateException("The object cannot be modified after built");
   }
   id3tag.albumTrack = val;
   return this;
 }

 public Builder comment(String val) {
   if (isBuilt){
     throw new IllegalStateException("The object cannot be modified after built");
   }
   id3tag.comment = val;
   return this;
 }
 // ... a lot more optional parameters

 public ID3Tag build() {
   if (isBuilt){
     throw new IllegalStateException("The object cannot be built twice");
   }
   isBuilt = true;
   return id3tag;
 }
}

private ID3Tag(String title, String artist) {
 this.title = title;
 this.artist = artist;
}
}

The usage of this class would be:

ID3Tag tag = new ID3Tag.Builder("My Title", "My author")
                     .comment("Great song").build();

I have found a similar pattern, called the Essence pattern, described here and here by Dr Herbie. This pattern uses direct access to the fields of the builder (like in a C++ structure) instead of using "setter" methods and it doesn't use "chaining" of modifications like in the New Builder Pattern ("...builder.option1(value1).option2(value2)...").

Initialization on demand holder idiom

In software engineering, the Initialization on Demand Holder idiom (design pattern) is a lazy-loaded singleton. The idiom can be implemented in both single-threaded/serial and concurrent environments, but care must be taken to correctly implement the idiom under concurrent conditions.

Example Java Implementation

This implementation from Bill Pugh (see links below) is a well-performing and concurrent implementation valid in all versions of Java. The original implementation from Bill Pugh has been modified to reduce the scope of LazyHolder.something to package and to make the field final.

public class Something {
      private Something() {
      }

      private static class LazyHolder {
              private static final Something something = new Something();
      }

      public static Something getInstance() {
              return LazyHolder.something;
      }
} 

How it works

The implementation relies on the well-specified initialization phase of execution within the Java Virtual Machine (JVM); see section 12.4 of Java Language Specification (JLS) for details.

When the class Something is loaded by the JVM, the class goes through initialization. Since the class does not have any static variables to initialize, the initialization completes trivially. The static class definition LazyHolder within it is not initialized until the JVM determines that LazyHolder must be executed. The static class LazyHolder is only executed when the static method getInstance is invoked on the class Something, and the first time this happens the JVM will load and initialize the LazyHolder class. The initialization of the LazyHolder class results in static variable something being initialized by executing the (private) constructor for the outer class Something. Since the class initialization phase is guaranteed by the JLS to be serial, i.e., non-concurrent, no further synchronization is required in the static getInstance method during loading and initialization. And since the initialization phase writes the static variable something in a serial operation, all subsequent concurrent invocations of the getInstance will return the same correctly initialized something without incurring any additional synchronization overhead.

When to use it

Use this pattern if the initialization of the class is expensive and it cannot be done safely at class-loading time and the initialization is highly concurrent. The crux of the pattern is the safe removal of the synchronization overhead associated with accessing a singleton instance.

A Java Builder Pattern

A Java Builder Pattern

There's a Builder pattern that Joshua Bloch has briefly described in a couple of his "Effective Java Reloaded" sessions at Java One. This Builder is not necessarily a replacement for the original design pattern. The problems this Builder pattern can solve are too many constructors, too many constructor parameters, and over use of setters to create an object.

Here are some examples of the pattern in use. These examples create various Widgets with two required properties and several optional ones -

Widget x = new Widget.Builder("1", 1.0).
                    model("1").build();
Widget y = new Widget.Builder("2", 2.0).
                    model("2").manufacturer("222").
                    serialNumber("12345").build();
Widget z = new Widget.Builder("3", 4.0).
                    manufacturer("333").
                    serialNumber("54321").build();

The basic idea behind the pattern is to limit the number of constructor parameters and avoid the use of setter methods. Constructors with too many parameters, especially optional ones, are ugly and hard to use. Multiple constructors for different modes are confusing. Setter methods add clutter and force an object to be mutable. Here is an class skeleton of the pattern -

public class Widget {
public static class Builder {
  public Builder(String name, double price) { ... }
  public Widget build() { ... }
  public Builder manufacturer(String value) { ... }
  public Builder serialNumber(String value) { ... }
  public Builder model(String value) { ... }
}

private Widget(Builder builder) { ... }
}

Notice that Widget has no public constructor and no setters and that the only way to create a Widget is using the static inner class Widget.Builder. Widget.Builder has a constructor that takes the required properties of Widget. Widget's optional properties can be set using optional property methods on the Widget.Builder. The property methods of Widget.Builder return a reference to the builder so method calls can be chained.

A really nice feature of this pattern is the ability to do pre-creation validation of an object state. When setters are used to set object state during creation it is virtually impossible to guarantee that object has been properly created.

Here is the full source for Widget and its Builder -

public class Widget {
 public static class Builder {
  private String name;
  private String model;
  private String serialNumber;
  private double price;
  private String manufacturer;

  public Builder(String name, double price) {
    this.name = name;
    this.price = price;
  }

  public Widget build() {
    // any pre-creation validation here
    Widget result = new Widget(name, price);
    result.model = model;
    result.serialNumber = serialNumber;
    result.manufacturer = manufacturer;
    return result;
  }

  public Builder manufacturer(String value) {
    this.manufacturer = value;
    return this;
  }

  public Builder serialNumber(String value) {
    this.serialNumber = value;
    return this;
  }

  public Builder model(String value) {
    this.model = value;
    return this;
  }
 }

private String name;
private String model;
private String serialNumber;
private double price;
private String manufacturer;

/**
 * Creates an immutable widget instance.
 */
private Widget(String name, double price) {
  this.name = name;
  this.price = price;
}

public String toString() {
  return super.toString() + " {"
          + "name="
          + getName()
          + " model="
          + getModel()
          + " serialNumber="
          + getSerialNumber()
          + " price="
          + getPrice()
          + " manufacturer="
          + getManufacturer()
          + "}";
}

public String getManufacturer() {
  return manufacturer;
}

public String getModel() {
  return model;
}

public String getName() {
  return name;
}

public double getPrice() {
  return price;
}

public String getSerialNumber() {
  return serialNumber;
}
}

Notice that Widget's private constructor takes the required properties and that the Builder sets the optional properties. Another thing to note is that widget is an immutable object as implemented.

Memory Monitoring with Java SE 5

Memory Monitoring with Java SE 5

Last updated Mar 10, 2006.

One of the benefits that Java introduced over C++ was the concept of automatic memory management. Specifically, Java Virtual Machines (JVM) provide a garbage collection thread that frees memory when the JVM's heap exhausts available memory. The implementation of the garbage collection strategy is dependent on the JVM vendor, but it suffices to say that the garbage collection process can be expensive. The Sun JVM defines garbage collection in two modes:

• Minor "copy" collections
• Major "Mark-Sweep-Compact" collections

A minor collection runs relatively quickly and involves moving live data around the heap in the presence of running threads. A major collection is a much more intrusive garbage collection that suspends all execution threads while it completes its task. In terms of performance tuning the heap, the primary goal is to reduce the frequency and duration of major garbage collections.

The IBM JVM implements a different architecture and graduates its garbage collection behavior based off of the state of the heap. Garbage collections can range from minimal impacting sweeps (fueled by a mark phase executed concurrently to executing threads) to a worst case scenario of a full mark-sweep-compact, stop-the-world garbage collection.

Because Java applications and their respective components all run inside the JVM and rely on objects instances loaded into the heap, tuning the heap is of paramount importance. But before the heap can be properly tuned, you need insight into the performance of the heap.

Until the release of Java SE 5, monitoring options included:

• Heap size and usage information obtained through calls to the java.lang.Runtime class
• Verbose garbage collection log parsing, enabled by passing startup parameters to the JVM to instruct it to generate verbose logging information every time a garbage collection occurs
• Proprietary JVM APIs, such as Sun's jvmstat interface
• Dangerous proprietary strategies such as injecting a DLL or shared object into the JVM process space to read process memory

Each of these strategies has advantages and disadvantages in terms of overhead and ease of implementation, but the result is that there is no single perfect strategy.

One of the benefits of performance monitoring application servers is that all major vendors have adopted the Java Management Extensions (JMX) and exposed monitoring information in the form of managed beans (MBeans.) With the release of Java SE 5, a JMX registry and support for MBeans was added to the JVM itself, so now that same luxury is available to stand-alone applications as well as enterprise applications. In addition, the Java SE 5 specification has defined new classes and interfaces in the java.lang.management package that expose JVM runtime statistics. A handful of those new interfaces are implemented by managed beans that expose runtime information about the behavior of the JVM memory, both heap and non-heap process memory. In this article we look at three bean interfaces:

• MemoryMXBean
• MemoryPoolMXBean
• GarbageCollectionMXBean

All management MBeans can be loaded through a helper class: java.lang.management.ManagementFactory. The ManagementFactory provides a set of static methods that return the requests MBeans:

MemoryMXBean memorymbean = ManagementFactory.getMemoryMXBean();
List mempoolsmbeans = ManagementFactory.getMemoryPoolMXBeans();
List gcmbeans = ManagementFactory.getGarbageCollectorMXBeans();

The MemoryMXBean provides information about memory usage, including both heap and non-heap memory. Specifically it provides the following two methods:

MemoryUsage getHeapMemoryUsage()
MemoryUsage getNonHeapMemoryUsage()

These methods return an instance of java.lang.management.MemoryUsage that defined four key attributes:

• init: the initial amount of memory that the JVM requested from the operating system during startup
• used: the amount of memory currently in use
• committed: the amount of memory that is guaranteed to be available for use by the JVM; it can change over time and is always guaranteed to be greater than or equal to the used memory
• max: the maximum amount of memory that can be used by the JVM in the specified area (heap or non-heap)

The MemoryPoolMXBean provides information about specific memory pools within the JVM memory spaces, both heap and non-heap. For those familiar with the Sun heap, these pools include the Eden Space, Survivor Space, Tenured Generation, and Permanent Generation.

In addition to others that are presented later. But the point is that these are the logical partitions that memory is subdivided into. For each MemoryPoolMXBean, you can discover the following information:

• Current Usage
• Peak Usage
• Usage at the last collection
• The type of memory space (heap or non-heap)
• The memory managers that operate on this space, for example "Copy" and "MarkSweepCompact"

Finally, the GarbageCollectorMXBean provides collection times and collection counts for each type of memory pool.

Listing 1 uses each of the aforementioned MBeans in conjunction with some memory mismanagement to display information about the JVM memory.

Listing 1. Java5ManagementTest.java

package com.javasrc.management;

import java.lang.management.*;
import java.util.*;

public class Java5ManagementTest
{
public static void dumpMemoryInfo()
{
  try
  {
    System.out.println( "\nDUMPING MEMORY INFO\n" );
    // Read MemoryMXBean
    MemoryMXBean memorymbean = ManagementFactory.getMemoryMXBean();
    System.out.println( "Heap Memory Usage: " + memorymbean.getHeapMemoryUsage() );
    System.out.println( "Non-Heap Memory Usage: " + memorymbean.getNonHeapMemoryUsage() );

    // Read Garbage Collection information
    List gcmbeans = ManagementFactory.getGarbageCollectorMXBeans();
    for( GarbageCollectorMXBean gcmbean : gcmbeans )
    {
      System.out.println( "\nName: " + gcmbean.getName() );
      System.out.println( "Collection count: " + gcmbean.getCollectionCount() );
      System.out.println( "Collection time: " + gcmbean.getCollectionTime() );
      System.out.println( "Memory Pools: " );
      String[] memoryPoolNames = gcmbean.getMemoryPoolNames();
      for( int i=0; i mempoolsmbeans = ManagementFactory.getMemoryPoolMXBeans();
    for( MemoryPoolMXBean mempoolmbean : mempoolsmbeans )
    {
      System.out.println( "\nName: " + mempoolmbean.getName() );
      System.out.println( "Usage: " + mempoolmbean.getUsage() );
      System.out.println( "Collection Usage: " + mempoolmbean.getCollectionUsage() );
      System.out.println( "Peak Usage: " + mempoolmbean.getPeakUsage() );
      System.out.println( "Type: " + mempoolmbean.getType() );
      System.out.println( "Memory Manager Names: " ) ;
      String[] memManagerNames = mempoolmbean.getMemoryManagerNames();
      for( int i=0; i<1000000; s = "My String ">

Listing 1 presents a class that displays JVM memory management information before and after allocating one million strings. It begins by retrieving and displaying the memory usage information through the MemoryMXBean. In my execution, the following are the states before and after the run:

// Before
Heap Memory Usage: init = 33554432(32768K)
        used = 241680(236K)
     committed = 33357824(32576K)
        max = 33357824(32576K)
Non-Heap Memory Usage: init = 29556736(28864K)
          used = 12055504(11772K)
       committed = 29851648(29152K)
          max = 121634816(118784K)

// After
Heap Memory Usage: init = 33554432(32768K)
        used = 218656(213K)
     committed = 33357824(32576K)
        max = 33357824(32576K)
Non-Heap Memory Usage: init = 29556736(28864K)
          used = 12131600(11847K)
       committed = 29884416(29184K)
          max = 121634816(118784K)

In this case, the minimum and maximum values for the heap are set to 32MB, which is because I used the following parameters on startup:

-Xms32m –Xmx32m

The used memory dropped from 236K to 213K after the run, which we will discover is the result of several garbage collections.

The garbage collection information is retrieved from the GarbageCollectorMXBean. The Sun JVM with its default configuration implements two garbage collectors: Copy and MarkSweepCompact. In my sample execution, the following are the states of garbage collection before and after the run:

// Before
Name: Copy
Collection count: 0
Collection time: 0
Memory Pools:
 Eden Space
 Survivor Space

Name: MarkSweepCompact
Collection count: 0
Collection time: 0
Memory Pools:
 Eden Space
 Survivor Space
 Tenured Gen
 Perm Gen
 Perm Gen [shared-ro]
 Perm Gen [shared-rw]

// After
Name: Copy
Collection count: 63
Collection time: 12
Memory Pools:
 Eden Space
 Survivor Space

Name: MarkSweepCompact
Collection count: 0
Collection time: 0
Memory Pools:
 Eden Space
 Survivor Space
 Tenured Gen
 Perm Gen
 Perm Gen [shared-ro]
 Perm Gen [shared-rw]

From this output you can see that creating and discarding one million Strings resulted in 63 copy collections that accounted for 12 milliseconds to run. If we change the code to not discard the Strings between iterations then we will see MarkSweepCompact collections occur.

Finally we display information about the various memory pools by accessing the MemoryPoolMXBeans. This returns several memory pools in the Sun JVM:

• Code Cache: contains memory used for compilation and storage of native code
• Eden Space: pool from which memory is initially allocated for most objects
• Survivor Space: pool containing objects that have survived Eden space garbage collection
• Tenured Gen: pool containing long-lived objects
• Perm Gen: contains reflective data of the JVM itself, including class and memory objects
• Perm Gen [shared-ro]: read-only reflective data
• Perm Gen [shared-rw]: read-write reflective data

The following displays sample output for the four primary memory pools (Eden, Survivor Space, Tenured Generation, and Permanent Generation) after the test has completed:

Name: Eden Space
Usage: init = 2162688(2112K) used = 90784(88K) committed = 2162688(2112K) max = 2162688(2112K)
Collection Usage: init = 2162688(2112K) used = 0(0K) committed = 2162688(2112K) max = 2162688(2112K)
Peak Usage: init = 2162688(2112K) used = 2162688(2112K) committed = 2162688(2112K) max = 2162688(2112K)
Type: Heap memory
Memory Manager Names:
 MarkSweepCompact
 Copy

Name: Survivor Space
Usage: init = 196608(192K) used = 16(0K) committed = 196608(192K) max = 196608(192K)
Collection Usage: init = 196608(192K) used = 16(0K) committed = 196608(192K) max = 196608(192K)
Peak Usage: init = 196608(192K) used = 127928(124K) committed = 196608(192K) max = 196608(192K)
Type: Heap memory
Memory Manager Names:
 MarkSweepCompact
 Copy

Name: Tenured Gen
Usage: init = 30998528(30272K) used = 127856(124K) committed = 30998528(30272K) max = 30998528(30272K)
Collection Usage: init = 30998528(30272K) used = 0(0K) committed = 0(0K) max = 30998528(30272K)
Peak Usage: init = 30998528(30272K) used = 127856(124K) committed = 30998528(30272K) max = 30998528(30272K)
Type: Heap memory
Memory Manager Names:
 MarkSweepCompact

Name: Perm Gen
Usage: init = 8388608(8192K) used = 127800(124K) committed = 8388608(8192K) max = 67108864(65536K)
Collection Usage: init = 8388608(8192K) used = 0(0K) committed = 0(0K) max = 67108864(65536K)
Peak Usage: init = 8388608(8192K) used = 127800(124K) committed = 8388608(8192K) max = 67108864(65536K)
Type: Non-heap memory
Memory Manager Names:
 MarkSweepCompact

From this output you can surmise that for a 32MB heap running on Windows with default configuration that Eden was allocated 2112KB, each of the two survivor spaces received 192KB, and the tenured generation received the remaining 30272KB, all adding up to the 32768KB heap. An interesting observation is that the permanent space is allocated an initial 8MB and can grow up to 64MB. By polling this information you can analyze the behavior of the entire heap. For those of you with a passion for this type of work, the MBeans also define a notification interface that you can find in the Java SE 5 Javadocs.

Finally, if you would like to view this information at runtime, the Sun JVM provides the Java Monitoring and Management Console (JConsole) that can connect to a running JVM and present this information. You can read more about it in a Sun article by Mandy Chung: Using JConsole to Monitor Applications.

Summary

Memory monitoring in Java 1.4.x and earlier was something of a black art, but with the introduction of Java SE 5 and the adoption of public interfaces such as JMX, the task has now been made simple. With the technical details of obtaining the information out of the way, you are free to focus on the real business value: analyzing those metrics.

Setting up a VPN in Windows Server 2003

Setting up a VPN in Windows Server 2003 or Windows XP is something that is fairly easy to do in no time at all. Here is a no fluff guide on how to get it done.

A Virtual Private Network (VPN) is a method used for creating a secure encrypted tunnel between two network endpoints. Creating a VPN in Windows XP or Windows Server 2003 has many advantages in a variety of different network scenarios. Setting up a simply VPN is a very quick and painless process that can be achieved in no time at all.

1. Open the Control Panel
2. Double-click the “Network Connections” icon
3. Click “Create New Connection” in the left hand side of the window.
4. Click “Next” and then select “Setup an advanced connection” and click “Next” again
5. Select “Accept incoming connections” and click “Next”
6. Click “Next”
7. Select “Allow virtual private connections” and click “Next”
8. Place a checkmark next to the users you wish to allow to connect and click “Next”
9. Click “Next” and then click “Finish”

It’s as simple that! As an added note, if you are behind a router you will want forward port 1723 which is standard for PPTP. There are obviously several other ways to implement VPN but this is your basic setup.

Innodb Performance Optimization Basics

Innodb Performance Optimization Basics

Posted By peter On November 1, 2007 @ 9:17 am In Innodb | 30 Comments

Interviewing people for our [1] Job Openings I like to ask them a basic question - if you have a server with 16GB of RAM which will be dedicated for MySQL with large Innodb database using typical Web workload what settings you would adjust and interestingly enough most people fail to come up with anything reasonable. So I decided to publish the answer I would like to hear extending it with basics of Hardware OS And Application optimization.
I call this Innodb Performance Optimization Basics so these are general guidelines which work well for wide range of applications, though the optimal settings of course depend on the workload.

MySQL Crash Recovery

MySQL Crash Recovery

MySQL is known for its stability but as any other application it has bugs so it may crash sometime. Also operation system may be flawed, hardware has problems or simply power can go down which all mean similar things - MySQL Shutdown is unexpected and there could be various inconsistences. And this is not only problem as we’ll see.

MySQL has angel process mysqld_safe which will restart MySQL Server in most cases. It is great, unless you have run into some bug which causes it to crash again - such crashes qucikly following one another are kind of worse because they explore many less tested code paths in MySQL and so problem potential is larger.

So lets look at the problem which happen during the crash which might need to take care of or which may seriously affect MySQL Performance.

MyISAM Corruption - If you’re writting to MyISAM tables there is very large chance of them becoming corrupted during the crash. Note corruption may be hidden and do not expose itself instantly - you may notice wrong query results days after crash. Sometimes corrupted tables may be reason for further crashes or hangs, and corruption may spread itself further in the table. You probably do not want any of these so it is very good idea to run MySQL with myisam_recover option which will make sure all improperly closed MyISAM tables are checked first time it is accessed. This option is however rather painful to use with web applications - users may issue different queries which may trigger check/repair running for many tables at onces, which typically make system extremely slow and also can use up all allowed connections or run out of memory ( myisam_sort_buffer_size is normally set pretty lage). If this becomes the problem I use tiny script which moves out all MyISAM tables out of MySQL database directory, checks them with MyISAMchk and moves them back to running server. This looks scary but it works great - until table is checked and ready application gets error rather than stalling forever which allows application to become partially functional as soon as possible. This hack is needed only in some cases - in most cases using Innodb for tables which you need to be recovered fast is better solution.

Innodb Recovery - Unless you have some hardware problems (99%) or found new Innodb bug (1%) Innodb recovery should be automatic and bring your database to consistent state. Depending on innodb_flush_lot_at_trx_commit setting you may lose few last committed transactions but it is it. It is Performance of this process which may cause the problems. As I already wrote innodb_log_file_size and innodb_buffer_pool_size affect recovery time significantly as well as your workload. I should also mention if you have innodb_file_per_table=1 your recovery speed will depend on number of Innodb tables you have, as well as many other operations, so beware.

Binary log corruption - Binary log may become corrupted and out of sync with database content. This will sometimes break replication but if you’re just planning on using binary log for point in time recovery it can go unnoticed. sync_binlog Is helping by syncing binary log, but at performance penalty. If using Innodb you also might with to use innodb-safe-binlog option in MySQL 4.1 so your Innodb log and binary log are synchronized. In MySQL 5.0 XA is taking care of this synchronization.

.frm Corruption - Few people know MySQL is not really ACID even with Innodb tables, at least not for DDL statements.
There is a chance of failing for example during CREATE statement with table created in Innodb dictionary but .frm not created or not completely written. Partially written .frm files or .frm being unsync with internal Innodb dictionary may cause MySQL to fail with wierd error messages. In MySQL 4.1 sync_frm option was added which reduces this problem as time window when it can happen is much less. Still if failure happens just during writting .frm file nasty things may happen, not to mention such potentially multiple operation DDL statements as RENAME TABLE - these are most vulnerable.

master.info corruption - If slave happens to crash you can also have relay logs corruption and master.info being corrupted. Not to mention MyISAM tables can contain partially completed statements as well as some of updates totally lost. The safe approach it to reclone the slaves if they crash or you can take the risks and try to continue. Sometimes you might be able to manually find appropriate position even if master.info file is out of sync but I would not be basing my failure handling scenarios.

Cold Start - If you restart MySQL server its caches (key_buffer, innodb_buffer_pool, query_cache,table_cache) are cleaned, so may be OS caches. This may reduce performance dramatically. So if you’re bringing server back after crash you might want to populate caches. For MyISAM key_cache this can be done by using LOAD INDEX INTO CACHE statement, for other storage engines it can be done by issuing large index scan queries. Full table scan queries allow to preload table data ether in storage engine caches or in OS cache. You can save these into .sql file and use –init-file to make sure it is run on startup. The other approach is to prime server with real servers (ie clone queries from other slave) before putting traffic to it.
In case application is not highly available so there is only one server you might with to start serving only some users initially (returning error to others) and gradually increase the load as server warms up. This may sound strange but makes a lot of sense as not only waiting for pages which never load is more frustrating for users than getting honest “try again later” message, but also - warmup takes longer time on extreme load.

Innodb statistics - Unlike MyISAM Innodb does not store index cardinality in tables, instead it computes them on first table access after startup. This may take significant time if you have very large number of tables (Some users have hundreds of thousands of tables per database host). This one is pretty much part of cold start problems but I wanted to point out it separately. To warmup this data you might run select 1 from _table_ limit 1 for each table or any other statement - it is table open which is important.

There are other problems which you may experience related to MySQL Crash Recovery - Restoring data from backup, corrupted Innodb tablespace recovery etc but I should write about them some other time.

How To Hack A Windows XP Password

1. start cmd
2. 'net user': it will list users/accounts of the computer, for example 'Administrator', 'cesar', etc....
2. 'net user *': it will ask for password for
4. enter new password

Should MySQL and Web Server share the same box ?

Should MySQL and Web Server share the same box ?

Posted By peter On October 16, 2006 @ 6:09 am In lamp, production, tips | 7 Comments

This is interesting question which I thought it would be good to write about. There are obviously benefits and drawbacks for each of methods.

Smaller applications usually start with single server which has both MySQL and Web server on it. In this case it is not usually the question but once application growths larger and you need to have multiple servers you may decide ether to grow system in MySQL+Apache pairs or split MySQL And Web Server and place them on different boxes.

Generally using separate boxes for MySQL and Web Servers is rather good practice.

It is more secure - Compromising your web server does not directly give access to your database, even though most applications have enough database access permissions to be allow intruder to trash/dump data.

It is easier to analyze - Troubleshooting bottlenecks on shared boxes is more complicated compared to systems running only MySQL or only Web server. In this case you already know who is troublemaker by simply looking at system wide stats.

Easier to maintain - Same thing if box happens to run multiple things it is harder to maintain. I would not call the difference significant in this case though.

Easier to balance - Lets say you have Web application and just added some new feature, ie chat application which increases load on your web server but does not really affect database part of load. If you can operate database farm and web server farms separately you can simply increase number of web servers.

It is less expensive - You typically want database boxes to be secure, using good hardware with ECC memory to avoid database corruption, use RAID to avoid loosing database with any hard drive loss etc. Database boxes also generally require more monitoring and maintainence such as backups so you end up using some serious hardware for this boxes to keep their
number manageable. With Web boxes it is different - you’re quite OK using crappy hardware for them as all you need is CPU power. If box starts to misbehave it is easy to shut it down without affecting site operations. Also you rarely would have data corruption due to web boxes memory failure, more likely you’ll have web server crashes and this sort of things. You can ether clone web servers from template hard drive or even have them disk less booting by NFS.

So if using dedicated boxes is so great why to think about sharing MySQL and Web server at all ? Well mostly it is for cheap guys.
In many applications you will find database servers to be IO bound so CPUs are doing virtually nothing and you’re wasting resources. This is the reason for some cheap environments to have Web servers also on database boxes, might be only handling partial load etc.

I would however only use it in last resort - placing some data crunching scripts on database server is often better use of their free CPU time.

Second thing you may feel bad about it is Web Servers memory. Getting certain amount of memory is pretty cheap, ie 4GB of memory per box costs very close to 2GB, while jump from 16GB to 32GB may be much more expensive (even in price by GB).
So you can get Web boxes with relatively plenty of memory cheap but unless you’re running 500 Apache children with mod_P (php,perl,python) per box (which is probably bad idea anyway).

The good use for such extra memory is probably caching - Web page caching, if you do not have separate layer for it, local memory or cacheing type of caching (depending on your application needs) is very good idea.

One more benefit of local access to MySQL is latency. This was problem many years ago with 10Mbit network but with 1Gbit networks being commodity these days you should not worry too much about it, unless you have each page generated by 1000+ queries, which is bad idea already.

One case I should mention when shared MySQL and Web server makes sense is Web Services architecture when you can have certain boxes providing you with some simple “Services” - these could be small enough to be single shared box (or pair of shared boxes for HA). In such cases I would think about Web Server mainly being provider of different protocol to access your data - it is typically simple and would not require much of CPU and other resources itself.

For example you can see this “Shared” kind of architecture in CNET systems using ATOMICS component to talk to MySQL over HTTP. (not what I’m great fan of this idea though).

---

7 Comments (Open | Close)

7 Comments To "Should MySQL and Web Server share the same box ?"

#1 Comment By John Latham On October 16, 2006 @ 7:58 am

Special case for shared box: circular multi-master replication topology, with each node running web & db, datasources point to localhost. This will (in principle) scale linearly with number of boxes, until propagation delays become problematic (but less of an issue if using sticky sessions). Useful for read-heavy apps.

#2 Comment By peter On October 16, 2006 @ 8:18 am

John,

Thank you for comment.

It scales linearly only from the first glance. In reality it has problems with scaling writes (you mention it already) second as database size grows it may change from CPU bound workload to IO bound workload which slows things down dramatically.

This is not to mention conflicting updates and complicated failure recovery for circular replication.

In general I can only see it used then conflicting updates are not an issue and application can’t be made aware of multi server configuration.

#3 Comment By Michael On October 16, 2006 @ 2:43 pm

Do you have any comments on using VMWare to partition your web servers / databases as virtual machines on one or multiple (physical) boxes? To me the separate physical web server on one box and database is a better idea, but some people keep on recommending this to me.

#4 Comment By peter On October 17, 2006 @ 1:14 am

Michael,

I think using VMWare and other virtualization techniques are good for two cases - testing and if you share same server among different people (in this case not VMWare but other techniques should be used of course)

Some people also use virtualization to ease with cloning as well as configuration moving to other server - I think it is easy enough to do standard way.

Also sharing any way limits you to resources of single server - dedicated physical web and database box will surely have more power.

#5 Pingback By Zedomax Server Upgrade Complete! | zedomax.com - blog about DIYs and Review on reviews of gadgets and technologies… On February 22, 2007 @ 5:55 pm

[…] If you want to know about running a more efficient web server, check out this article on mysql and web server on different boxes. […]

#6 Pingback By smalls blogger » Blog Archive » links for 2007-07-12 On July 11, 2007 @ 6:11 pm

[…] MySQL Performance Blog » Should MySQL and Web Server share the same box ? Should MySQL and Web Server share the same box ? (tags: mysql apache server performance scaling web architecture php) […]

#7 Comment By Dedicated Hosting Provider On March 24, 2008 @ 10:27 am

I had an infrastructure class during my undergraduate studies and we used VMWare for the entire course. VMWare was very good for simulation of different types of issues but also had a lot of problems. It is hard to simulate real systems using virtual machines and virtual machines are very easily corrupted so you need to ensure you backup your information very frequently.

---

Article printed from MySQL Performance Blog: http://www.mysqlperformanceblog.com

URL to article: http://www.mysqlperformanceblog.com/2006/10/16/should-mysql-and-web-server-share-the-same-box/

MySQL 4 to MySQL 5 Upgrade performance regressions

MySQL 4 to MySQL 5 Upgrade performance regressions

This week I already had two serious performance regression cases when upgrading from MySQL 4.0 and 4.1 to MySQL 5.0. By serious I mean several times performance difference not just 5-10% you often see for simple queries due to generally fatter code.

The problem in both cases was MySQL 5.0 broken group commit bug.

First I should note I am extremely unhappy how MySQL handled this problem. While working for MySQL we spotted this problem early in MySQL 5.0 release cycle as it was introduced and reported it to everyone we could inside the company - this was over 2 years ago. Few months later I created a bug for this issue to get more public attention to the problem and giving extra motivation to MySQL to fix it. Few months later I blogged about this problem with more performance results but as we can see the bug is still in Verified stage and there is no indication any work is going to have it fixed.

I can agree this may be fundamental issue which is not easy to fix, But why is not it mentioned in MySQL 4.1 to 5.0 upgrade notes ? ?

Furthermore if there were no good ideas how to make XA to work with group commit why would not you keep old working code path if XA is disabled ? Many customers do not flush binary log anyway and use single transactional storage engine so they do not care anyway.

Anyway. Enough complains. We have this problem and we have to live with it, most likely MySQL 5.0 and 5.1 would not get any fixes for this problem, so lets see who is affected, how to check you’re affected and how to fix it.

Who is affected ? The good thing is only cheap guys which care about their data are typically affected, meaning you have to have innodb_flush_logs_at_trx_commit=1 so transactions are truly durable. You have to have log-bin enabled to get replication or point in time recovery, but at the same time you should not have hardware RAID or have one without battery backed up cache (BBU unit). I guess this is one of the reason why this bug did not get so much traction inside MySQL - because paying customers would normally have enough money to get BBU unit which is great for performance anyway. Of course you also have to have plenty of concurrent transactions so group commit would trigger in MySQL 4.0 and large number of transactions in total so serializing them would make MySQL unable to keep up. Disks can do 80-150 single page fsyncs per second to get you an idea bout number.

How to spot you’re affected This one is interesting. If you have update prevailing load you would see very strange behavior on 5.0 of MySQL being slow but few queries being “inside innodb” and potentially even empty queue. This is because bottleneck happens in commit phase which is not counted as “inside innodb”. I wish there would be some stats for number of queries waiting to be committed but there is not one easily readable. You can see it from other symptoms though. You would see queries in “SHOW PROCESSLIST” stuck in “end” stage or have “commit” queries in the processlist for multiple statement transactions. Looking at SHOW INNODB STATUS you would notice large amount of log writes and fsyncs per second which matches your hard drive capacity. Plus you would normally see single outstanding log write all the time. There are other ways you can spot the problem as well but these are probably most obvious and easy to use.

How to fix the problem ? Assuming getting back to MySQL 4.1 is not the option you can do one of 3 things to get some of your performance back. XA support has its overhead anyway so you may not get same performance as with MySQL 4.1

1. Disable Binary Logging This allows to get group commit back but obviously you loose point in time recovery and replication.
2. Use innodb_flush_log_at_trx_commit=2 This is probably best solution. In many cases this would be good change to do with MySQL 4 also because 100% durable transactions are not required anyway and it would allow to get some extra speed. It is often left at default value without a good reason.
3. Get BBU If you can’t use any of first two workarounds you better to get battery backed up cache unit and make sure you set your RAID cache policy to “write back”. One of the customers I worked with indeed had battery backed up cache on their system… it just was in “write through” cache policy, so was basically disabled. Note getting BBU is often good idea anyway so you can use this together with other workarounds. Also it is worth to mention BBU does not fix the problem but dramatically raises number of update transactions per second needed to trigger the problem. Without BBU 200 per sec may be well enough with BBU you may only see it if you have 2000 update transactions per second or so, which few people reach.

The other way to approach the problem is of course to work on the application - in large amount of cases the problem happens in case there are away to many updates outside of transactions in auto_commit mode. Wrap them in transactions and reduce number of commits if you can. It is great optimization idea anyway.

Also please do not read this post as MySQL 5.0 is junk in terms of performance and you should stay on MySQL 4.1 until MySQL has taken it away from you as already happened with MySQL 4.0. MySQL 5.0 can offer substantial performance improvements in variety of cases as well as has other benefits. This is simply important regression which you better to know about.

VOIP Advice

VOIP Advice requested

Posted By peter On September 10, 2007 @ 8:27 am In site | 9 Comments

As one of my last posts about [1] issues with hosting provider got great response and We got a lot of good advice and offers I decided to ask for advice another problem we have as we’re growing [2] our company - organizing good phone communications.

Our goals are rather simple though the fact we’re globally distributed may put us a bit aside from typical small business needs.

Hosting Advice

Posted By peter On August 31, 2007 @ 5:36 pm In site

During last one and a half year we had pretty good track record with MySQL Performance Blog - there were times when site was slow (especially when backup was running) but I do not remember significant downtime, until today we went down for few hours.

All this time the site was running on dedicated server which I rented from [1] APLUS about 3 years ago. It is rather slow Celeron box with single disk and 512MB running Fedora Core 2. Despite its age (it was used “Value” server even when I got it) the server had very good track record with basically zero failures during this time - there were some network disruptions at Aplus but this is about all problems we had.

So How can you bypass these caches?

Query Profiling with MySQL: Bypassing caches

Quite frequently I run into question like this “I’m using SQL_NO_CACHE but my query is still much faster second time I run it, why is that ?

The answer to this question is simple - because SQL_NO_CACHE only bypasses query cache but it has no change on other caches, which are
MySQL Caches - Innodb Buffer Pool and Key Buffer are best example though Falcon, PBXT and other storage engines have similar buffers. There is also table_cache both MySQL side and Internal Innodb one which can affect query execution speed.
OS Caches Operation Systems typically cache file IO unless you explicitely bypass it by using O_DIRECT flag or mounting file system in direct IO mode.
Hardware Caches State of CPU cache may affect query execution speed but only lightly, the hardware IO cache may however cause dramatic difference. Hardware RAID cache is the one but more important SAN caches which can be pretty big.

So How can you bypass these caches?

For MySQL Caches you can restart MySQL and this is the only way to clean all of the caches. You can do FLUSH TABLES to clean MySQL table cache (but not Innodb table meta data) or you can do “set global key_buffer_size=0; set global key_buffer_size=DEFAULT” to zero out key buffer but there is no way to clean Innodb Buffer Pool without restart.

For OS Caches on Linux you can use drop caches control available in new Linux Kernels. You could also remount file system in question and the safest thing is of course to reboot.

For Hardware Caches it is more hardware specific. Typically doing some IO will flush cashes but you can’t be sure as you do not know what policies exactly do they employ. For RAID hardware caches reboot of the box is also enough however SAN caches may survive longer. Though few of us have SAN available for performance benchmarking

MySQL Performance - eliminating ORDER BY function

MySQL Performance - eliminating ORDER BY function

Posted By peter On October 17, 2007 @ 5:24 am In optimizer | 5 Comments

One of the first rules you would learn about MySQL Performance Optimization is to avoid using functions when comparing constants or order by. Ie use indexed_col=N is good. function(indexed_col)=N is bad because MySQL Typically will be unable to use index on the column even if function is very simple such as arithmetic operation. Same can apply to order by, if you would like that to use the index for sorting. There are however some interesting exception.

RAID System performance surprises

RAID System performance surprises

Implementing MySQL database in 24/7 environments we typically hope for uniform component performance, or at least would like to be able to control it. Typically this is indeed the case, for example CPU will perform with same performance day and night (unless system management software decides to lower CPU frequency due to overheating).

This is also almost the case with Hard Drives - there are possible performance differences based on where data is stored on the disk, amount of remapped sectors etc. There is also database and file system fragmentation however these also tend to accumulate in predictable fashion.

If you have RAID controller this well may not be the case - to protect your data RAID controller may implement bunch of algorithms which can affect performance dramatically.

For example speaking about PERC5 (LSI MegaRaid) - Pretty typical controller from Dell installed on PowerEdge 1950, 2950 etc you should be aware of couple of things

Battery Learning and Charging Build in Battery has to pass through learning cycle every 3 months or so and this cycle takes about 7 hours according to the docs. During this time battery backed up cache will be disabled and system will operate with write through cache possibly slowing down write performance several times.

Patrol Read This is feature which should discover bad sectors before it is too late and it is doing so by doing disk read checks periodically. When it wakes up it will some IO resources (30% by default) which will affect your performance some way.

Consistency Checks This is another thing which I’ve seen initiated by controller (though I’m not sure on this one) - which pretty much checks the disks are in sync - this also can slow down performance dramatically.

So what you can do about these ?

First you should not have any of this to come as surprise for you when you discovered your server just stopped performance during the time you planned investor show case or other important event. Learn what cron jobs does your RAID card have and see how they can be controlled - may be schedule them during least busy intervals or something similar.

Also you should be ready for degraded and rebuild RAID mode - when one of the disk fails and you replace it with another one which needs to be rebuilt. This means you already should leave some slack of the system. It often would be enough for consistency check and patrol read but not for battery backed up cache being temporary disabled.

Another thing you can do is of course switch to another server and take this down for maintenance if this learning process can’t be scheduled when it is non intrusive. To do this properly however you need to know when it is about to happen.

Configuring SSH on Windows 2003

When you configure OpenSSH on the Tivoli^® Intelligent Orchestrator server on a Windows 2003 environment, SSH is installed, when Cygwin is installed.

To configure SSH on Windows 2003:

1. Log on as tioadmin.
2. Invoke a Cygwin window.
3. Type the following command in the Cygwin window: /usr/bin/ssh-host-config -y. After typing this command, the following functions occur:
   • Host keys are generated. This command generates three different keys-DSA, RSA, RSA1, each corresponding to a different encryption algorithm. These keys allow a system to establish SSH sessions with systems requiring any one of these encryption algorithms.
   • Privilege separation is enabled.
   • sshd is installed as a service.
   • Passwordless logon is enabled.
   • New user account is created.
4. You will then be prompted for a password for the new user account that has been created. Enter a password for the new user and ensure that this password matches the password rules given on your system. Output should be similar to this:

   $ ssh-host-config -y
   Generating /etc/ssh_host_key
   Generating /etc/ssh_host_rsa_key
   Generating /etc/ssh_host_dsa_key
   Overwrite existing /etc/ssh_config file? (yes/no) yes
   Generating /etc/ssh_config file
   Overwrite existing /etc/sshd_config file? (yes/no) yes
   Privilege separation is set to yes by default since OpenSSH 3.3.
   However, this requires a non-privileged account called 'sshd'.
   For more info on privilege separation read
   /usr/share/doc/openssh/README.privsep.
   Should privilege separation be used? (yes/no) yes
   Generating /etc/sshd_config file Added ssh to /etc/inetd.conf Warning:
   The following functions require administrator privileges!
   Do you want to install sshd as service?
   (Say "no" if it's already installed as service) (yes/no) yes
   
   You appear to be running Windows 2003 Server or later. 
   On 2003 and later systems, it's not possible to use the
   LocalSystem account if sshd should allow passwordless logon
   (e. g. public key authentication). If you want to enable that functionality,
   it's required to create a new account sshd_server' with special privileges,
   which is then used to run the sshd service under.
   
   Should this script create a new local account 'sshd_server' which has
   the required privileges? (yes/no) yes
   Please enter a password for new user 'sshd_server'. Please be sure that this
   password matches the password rules given on your system. Entering no password
   will exit the configuration. 
   
   PASSWORD=
   
   User 'sshd_server' has been created with password 'xxxxx'.
   
   If you change the password, please keep in mind to change the password
   for the sshd service, too. Also keep in mind that the user sshd_server
   needs read permissions on all users'.ssh/authorized_keys file to allow
   public key authentication for these users!. 
   
   (Re-)running ssh-user-config for each user will set the required
   permissions correctly.

5. Accept the default value for the environment variable CYGWIN, when sshd is started and press Enter. Output should be similar to this:

   Which value should the environment variable CYGWIN have when sshd starts?
   It's recommended to set at least "ntsec" to be able to change user context
   without password.
   Default is "ntsec".  CYGWIN=
   
   The service has been installed under sshd_server account.
   To start the service, call `net start sshd' or `cygrunsrv -S sshd'.
   
   Host configuration finished. Have fun!

6. If the user account created by Cygwin was not automatically added to the Administrators group, then you will get a warning similar to this:

   WARNING: Adding user sshd_server to local group root Administrators failed!
   Please add sshd_server to local group root Administrators before starting
   the sshd service!

7. To manually add the user account created by Cygwin to the Administrators group, go to My Computers > Manage and then add the user account to the Administrators group.
8. Ensure that the newly created user account has been added to the Administrators group by typing the following command in DOS window:

   net localgroup administrators

9. Start the sshd service by running the following command (it will automatically start when rebooted) :

   cygrunsrv -S sshd

10. Verify that tioadmin has a password in /etc/passwd. If no password is found, then you must generate one by running the command:

    cat /etc/passwd | grep tioadmin || mkpasswd -l > /etc/passwd

11. Configure SSH for tioadmin, by typing the following command:

    ssh-user-config

12. When prompted to created the identity files, choose your option for the ssh version that will be used. This is a configuration example for using SSH2 (which is recommended). Press Enter when prompted for a passphrase. Output should be similar to this:

    Shall I create an SSH1 RSA identity file for you? (yes/no) no
    Shall I create an SSH2 RSA identity file for you? (yes/no)  (yes/no) yes
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Do you want to use this identity to login to this machine? (yes/no) yes
    Shall I create an SSH2 DSA identity file for you? (yes/no)  (yes/no) no
    
    Configuration finished. Have fun!

13. Switch to the .ssh directory by typing: cd .ssh. Press Enter.
14. The user key must be put into the authorized_keys file of the user account on the server. To perform this task, run the command:

    cat id_rsa.pub >>authorized_keys

15. To configure SSH to accept connections from new hosts without prompting for confirmation, create a file in /home/tioadmin/.ssh called config.
    Note: This directory depends on how you have set up your system. To confirm where /home directory is, look at /etc/passwd.
    Run the command:

    echo "StrictHostKeyChecking no" > config

    The file should contain the value of

    StrictHostKeyChecking no

16. To verify that SSH is configured properly, do the following:
    1. Ensure the Cygwin service is started.
    2. To log on to the local host through SSH, type ssh tioadmin@, where is your host name. If SSH is properly configured you will see the following message:

       Fanfare!!!
       You are successfully logged in to this server!!!

    3. Exit the sshd session by typing exit.
       Note: Ensure that the sshd service is set to automatic startup. To verify that the service is set to automatic:

       1. Open the Services window.
       2. Select sshd.
       3. Right-click Properties then Start Mode. Automatic should be selected.
17. Copy the id_rsa.pub file, which contains the public keys, into the authorized keys file of the administrative account of any server in the data center that the Tivoli Intelligent Orchestrator server must communicate with or manage. Include any servers in the data center that Tivoli Intelligent Orchestrator is managing.
    1. Ensure that the managed server has an administrative account for which the SSH RSA keys (id_rsa, id_rsa.pub, and authorized_keys) have already been generated and should be contained into the .ssh directory of the respective administrative account home directory.
    2. Append the content (a single line of text) of the id_rsa.pub file which contains the public key from the server that will initiate the SSH session to the authorized_keys file of the administrative account of any target server in the data center that the Tivoli Intelligent Orchestrator server must communicate with or manage. Include any servers in the data center that Tivoli Intelligent Orchestrator will be managing.
    3. To verify, on the Tivoli Intelligent Orchestrator server, type:

       ssh @

       There should be no password prompt, followed by the prompt on the remote machine. After a successful logon, an entry for the communication partner will be created into a known_hosts file. As a troubleshooting step, sometimes this file may contain old or invalid entries associated with the managed server IP address or name. Deleting that entry should fix the connection problem.

SSH is configured for Windows 2003.

A Comparison of Free SSH and SCP Programs for Windows 9x, NT, ME, 2000 and XP

Source: http://www.jfitz.com/tips/ssh_for_windows.html

Introduction

If you want to access remote servers securely from Windows 9x, NT, ME, 2000 or XP and don't want to pay for programs that are freely available for UNIX-like platforms, you may find this document useful. It describes free SSH implementations for Windows.

Although this document concentrates on Windows-specific implementations, much of the content, particularly relating to Port Forwarding and Authorized Keys, applies to any SSH implementation, regardless of the platform. VPN technologies, (including IPSEC), are discussed in passing, but these are not the main focus.

Email me, fitz@jfitz.com, if you find better implementations, or if you think there's anything I can add here. Please note that I will only include references to products that are available free of charge, and I have a strong preference for open source implementations.

For more links to SSH tools for Windows, (and Mac), I'd recommend trying the OpenSSH Windows/Mac pages: http://www.openssh.com/windows.html, or http://www.openssh.com/macos.html.

What is SSH?

SSH is a protocol that uses public key cryptography to transfer data securely over insecure networks. At the core of most SSH implementations are the "ssh" and "scp" commands.

"ssh" and "scp" are "secure" versions of the UNIX "rsh" and "rcp" commands, (which in turn are a bit like the more familiar "telnet" and "ftp" programs). The "ssh" command provides command-line, (non-GUI), access to a remote server. "scp" allows files to be copied to/from a remote server. Both programs use the SSH protocol to establish a secure connection and to encrypt all data passing between the client and the server.

What is "Port Forwarding" or "Tunneling"?

In addition to the direct access provided by the "ssh" and "scp" commands, the SSH protocol includes a feature called "Port Forwarding", or "Tunneling". This can be used to provide secure access to other services that do not normally encrypt data during transmission.

For example, to receive email, many email programs use the (unencrypted) POP3 protocol to connect to a mailserver on port 110. If we can SSH to the mailserver before downloading our mail, we can configure the SSH session to "forward" port 110 through the encrypted SSH link. Now, if we configure our email client to connect to port 110 on our local machine it will in fact be connecting to the remote mailserver, but all the data, (i.e. our incoming email), will be transferred over the encrypted SSH link.

If we also forward port 25, (which normally handles SMTP, the outgoing email protocol), then both incoming and outgoing email will be encrypted.

If you are mainly interested in securing your Windows email client, you could try jumping straight to my step-by-step guide to configuring a PuTTY SSH client for port forwarding email traffic.

Any number of ports can be forwarded in a single SSH session. For example, we could also forward port 80, (HTTP), to provide secure access to a corporate webserver. In effect, it is usually possible to create a pretty good approximation to a VPN, (Virtual Private Network), just by forwarding a handful of common ports, and using scp, (or WinSCP), to transfer files. In fact, a VPN is roughly equivalent to an SSH session where everything is being forwarded over a single secure channel.

Should I use SSH or a VPN?

If you are trying to decide between SSH and a VPN, I find that SSH is generally appropriate in situations where occasional remote access is needed to specific services, (for example, checking email, downloading files). If persistent access to a wide range of services is required, (for example, working exclusively on a corporate network while telecommuting), then a VPN is probably a better choice.

One of the problems with a VPN is that both the client and server need to be customized for the VPN to work, whereas most modern UNIX-like servers will provide SSH access by default when a user account is established, (so only the client PC needs configuring). A second problem is that traditional VPNs tended to be difficult to configure correctly, particularly on the server-side. However, solutions such as OpenVPN, (http://openvpn.net), are much easier to install and configure than many of the more traditional, IPSEC-based VPN solutions, so if you have administrative (root) privileges on the server it may be worth exploring OpenVPN. A VPN may require more work initially to get running on the server, but it will ultimately provide more flexibility on the client.

PuTTY

PuTTY, (and its file transfer utility, pscp), are excellent Windows ssh/scp implementations, and are a piece of cake to get up and running.

Download putty.exe and pscp.exe here: http://www.chiark.greenend.org.uk/~sgtatham/putty/, and you're ready to go. These programs are not zipped and do not require any installation.

When you run PuTTY, a configuration screen is presented first. It's a good idea to play with the configuration until you get colors, fonts and other settings that suit you. You can then give the session a name and save it, so it's easy to restore these settings next time you run PuTTY, (just double-click the stored session name). This process can be a little awkward because you need to exit and restart PuTTY to update a stored session, but once you have a good set of settings saved, you can use that session as a template to create other stored sessions for different host machines.

To help you get up and running with PuTTY, I've created a step-by-step guide to configuring a "typical" PuTTY session, (including port forwarding for incoming/outgoing email). You can check it out here: http://www.jfitz.com/tips/putty_config.html

When you first connect to a new machine using PuTTY or pscp, they will give you a message to indicate that they don't recognize the machine. Choose "OK" to add the machine to the list of known hosts and to continue connecting.

PuTTY works a bit like an xterm. If you highlight text with the mouse it is automatically copied to the clipboard. A right-click of the mouse will paste the copied text at the command-line cursor position. Right-click anywhere on PuTTY's Title Bar, (or left-click on the PuTTY icon in the top left corner), to access a menu where the current session settings can be changed.

To remove PuTTY automatically, run "putty -cleanup", (without the quotes). To do the job manually, delete the files you downloaded and use regedit to remove all the keys under
HKEY_CURRENT_USER->Software->SimonTatham->PuTTY.

WinSCP

WinSCP provides an easy-to-use graphical interface to "scp" functionality. It creates an SSH link to the remote server, then displays local and remote files in "Explorer-like" windows. Copying files securely to/from the remote machine is as simple as dragging and dropping, (or cutting and pasting), files in the WinSCP windows.

As of version 3, WinSCP also supports SFTP, ("secure ftp"). Unless you specifically need to use SFTP, it's probably as easy to stick with SCP -- the basic WinSCP functionality handles many of the issues that SFTP was designed to address.

You can download WinSCP here: http://winscp.net/eng/

WinSCP starts with a configuration screen and can save session settings in a manner that is very similar to PuTTY.

As of version 3, WinSCP includes a "Cleanup" option that allows you to remove any possibly sensitive settings that may have been created on your computer. With version 2, to remove WinSCP delete WinSCP2.exe and, to clean up the registry, use regedit to remove all the keys under
HKEY_CURRENT_USER->Software->Martin Prikryl->WinSCP 2.

WinSCP is released under the GPL. The source code is available for download with the application. WinSCP is partly based on PuTTY code, (it is usually pretty up-to-date -- check the WinSCP site for version details).

pscp

pscp is a companion program of PuTTY. It is a pretty straightforward Windows implementation of scp. It's a command-line program only, so you need to run it in an MS-DOS window.

The basic format of a pscp command, (or any scp command for that matter), is:

pscp myusername@remotehost:remotefilespec localfilespec

...to download from remotehost to your local machine, or:

pscp localfilespec myusername@remotehost:remotefilespec

...to upload to remotehost from your local machine.

Type pscp with no arguments for a list of other parameters that can be supplied on the command line.

Tera Term Pro/TTSSH

Another good terminal emulation package that supports SSH is Tera Term Pro. You can download it here: http://hp.vector.co.jp/authors/VA002416/teraterm.html

Tera Term Pro comes with a regular Windows setup program. As with PuTTY, I'd recommend changing the terminal settings to suit your preferences. Save the setup as teraterm.ini to replace the defaults.

By default, Tera Term Pro does not include SSH. For this, you need to download the SSH extension, TTSSH. The original download site was here: http://www.zip.com.au/~roca/ttssh.html, but this doesn't seem to be accessible any more. You can try the following mirror instead: http://public.planetmirror.com/pub/ttssh/, (the required file is ttssh154.zip).

Unzip this file in the same directory that Tera Term Pro is installed in. It adds a program called ttssh.exe. Run this program in place of Tera Term Pro itself, (ttermpro.exe), and you should now have SSH available as an option during connect. Additionally, you will find that some SSH options are added to the Setup menu. You may wish to change the defaults so that SSH, (on port 22), is the default connection method.

Tera Term Pro with TTSSH used to be the best free SSH implementation for Windows, but, in my opinion at least, recent versions of PuTTY have surpassed them for ease of use and improved functionality. You may wish to try both and draw your own conclusions.

Command line/X Windows - Cygwin/OpenSSH

If you're happiest working in a UNIX-like command-line environment, and are already familiar with the "ssh" and "scp" commands, you may want to try Cygwin, (http://www.cygwin.com/). Cygwin is a pretty complete GNU based, (i.e. UNIX-like), environment for Windows.

If you don't want/need the full Cygwin UNIX-like environment, OpenSSH for Windows, (http://sshwindows.sourceforge.net/), is basically a "cut-down" version of Cygwin which includes command line ssh/scp/sftp functionality. It also includes ssh/sftp servers, and it has a fairly straightforward Windows installation program. (Note that OpenSSH for Windows replaces the older, and no longer maintained, Network Simplicity OpenSSH server solution, http://www.networksimplicity.com/openssh/.)

If you do choose Cygwin, and you are comfortable working in the X-Windows environment, in addition to the rich selection of GNU command-line programs, (including sshd, ssh, scp and sftp), there are also Cygwin ports of XFree86, (http://cygwin.com/xfree/), KDE, (http://kde-cygwin.sourceforge.net/), and Gnome, (http://cygnome.sourceforge.net/). While this is no substitute for dual-booting with a good Linux distro or BSD release, it is handy for doing quick X-Windows work on a server without needing to reboot your Windows machine. I haven't used Cygwin's X-Windows tools extensively, but I have noticed that the quality and stability seems to be improving with each new release.

Known Hosts

The "known hosts" list contains the public keys of host machines that you "trust" when using ssh or scp. All implementations of ssh/scp, (including PuTTY, WinSCP and TTSSH), will give warnings if you are connecting to a new machine that is not in the list of known hosts. In addition, they will refuse to connect if the host's public key does not match the stored value.

PuTTY stores the list of known hosts in the registry, under the key:

HKEY_CURRENT_USER->Software->SimonTatham->PuTTY->SshHostKeys.

WinSCP also stores the known hosts in the registry, under the key:

HKEY_CURRENT_USER->Software->Martin Prikryl->WinSCP 2->SshHostKeys.

Tera Term Pro stores its list of known hosts in the following file:

C:\"Program Files"\ttermpro\ssh_known_hosts

(assuming the program was installed in the default folder).

If a host machine's public key is legitimately changed, (as part of an upgrade for example), you will need to remove the old key from the file, (or the registry in the case of PuTTY or WinSCP), to successfully connect to that host. You should, of course, be 100% sure that the key was genuinely changed, and that this is not a hacker's machine masquerading as the host -- this is why these keys are stored and checked in the first place.

Automatic login

With any of the installations covered so far we can only establish an SSH connection provided we supply a password each time. It is possible to automate SSH connections by generating "passphrase-less" secure keys and modifying our connection settings to use the new keys. In general, I would only recommend this procedure if you have a specific requirement for automating file transfers, and you clearly understand the security implications.

Keys can be generated on the Windows client-side, using PuTTYGen, or on the server-side, using ssh-keygen. Either program will generate a public key file and a private key file. Different key file combinations can be generated for different SSH protocol versions. If you specify a blank "passphrase", then only the key files will be required to authenticate the connection, thereby allowing unattended connections. (If you include a passphrase you will benefit from a "doubly-secure" authentication, based on both the key and the passphrase.)

To use the keys, save the key files in a secure location on the client machine. Then copy the contents of the public key file into the file $HOME/.ssh/authorized_keys, (SSH protocol version 1), or $HOME/.ssh/authorized_keys2, (SSH protocol version 2), on the server machine. Finally, modify the PuTTY, WinSCP or TTSSH session details so that it uses the saved private key file, (making sure it also uses the correct protocol for the specified key). You will find the option to specify the Private Key File under the SSH settings.

Assuming you've got the right files in the right places, and the correct session settings, when you attempt to connect you should find that you can connect without supplying a password or passphrase.

If you are using Cygwin and the command-line ssh/scp versions, you can check the man pages for ssh to determine where you need to save your key files so that the ssh and scp commands will connect without requiring a password/passphrase.

This "passphrase-less" approach is reasonably secure, provided access to the client machine is restricted. However, if someone manages to steal your private key file, (which might not be that difficult on most Windows machines), your server account will be fully compromised.

SSH Servers

The following is a brief introduction to SSH server software for Windows.

Cygwin, (http://www.cygwin.com), includes a port of the OpenSSH server software. Cygwin is basically a GNU, (UNIX-like), subsystem that runs on 32-bit Windows. To get SSH support for Cygwin, install the "openssh" package in addition to the basic Cygwin packages.

If you have any problems installing and configuring Cygwin "by hand", you could also try http://sshwindows.sourceforge.net/. This implementation basically wraps Cygwin's OpenSSH port in a package that includes a standard Windows installation program.

If you have concerns about running open source secure services, such as OpenSSH or OpenVPN, on Windows, the main alternative is the IPSEC functionality that comes built-in to Windows 2000 servers. IPSEC is fully supported by Microsoft, and Windows-compatible implementations are also available for most *nix. This seems intuitively more secure, since it involves fewer components that are not integral parts of the Windows O/S.

You could also use Windows Terminal Services. This has built-in encryption and has the advantage that it gives you GUI access. You might want to use this in conjunction with IPSEC if you have any concerns about the Terminal Services encryption algorithms.

If you prefer open source solutions, it is still possible to get secure remote GUI access to a Windows server using freely available software. VNC, (http://www.realvnc.com/), provides remote GUI access to Windows machines, and it uses predictable port assignment, so a VNC client session can be port-forwarded over an SSH link provided the Windows server is running an SSH server, (in addition to the VNC server).

I believe it is also possible to port-forward a Terminal Services client session over SSH -- the server appears to use port 3389 by default according to the following Microsoft support article: http://support.microsoft.com/default.aspx?scid=kb;en-us;150543.

Old Versions of this Document

The previous version of this document, (which you may find useful if you are using older versions of Windows SSH utilities), is available here: http://www.jfitz.com/tips/ssh_for_windows_doc_version2.html.

Summary of Links in this Document

• OpenSSH Windows page: http://www.openssh.com/windows.html
• OpenSSH Mac page: http://www.openssh.com/macos.html
• OpenVPN, an open source, SSL VPN solution: http://openvpn.net
• PuTTY, pscp and related tools: http://www.chiark.greenend.org.uk/~sgtatham/putty/
• Configuring PuTTY: http://www.jfitz.com/tips/putty_config.html
• WinSCP: http://winscp.net/eng/
• Tera Term Pro: http://hp.vector.co.jp/authors/VA002416/teraterm.html
• TTSSH, (Tera Term Pro SSH extension): http://www.zip.com.au/~roca/ttssh.html
• TTSSH, (mirror site): http://public.planetmirror.com/pub/ttssh/
• Cygwin: http://www.cygwin.com/
• XFree86 for Cygwin: http://cygwin.com/xfree/
• KDE for Cygwin: http://kde-cygwin.sourceforge.net/
• Gnome for Cygwin: http://cygnome.sourceforge.net/
• OpenSSH for Windows: http://sshwindows.sourceforge.net/
• Microsoft's Terminal Services Advanced Client FAQs: http://www.microsoft.com/windows2000/techinfo/administration/terminal/tsacfaq.asp
• Microsoft Terminal Services TCP port support article: http://support.microsoft.com/default.aspx?scid=kb;en-us;150543
• VNC - Remote GUI access for Windows: http://www.realvnc.com/

Revision History

Version 1.0 22 June 1999 - Initial document
Version 2.0 22 January 2001 - Revised format, rewrote much of the content
Version 2.1 7 February 2001 - Added section on ssh servers
Version 2.2 16 November 2001 - Added notes on PuTTY port forwarding and Cygwin
Version 2.3 3 December 2001 - Removed link to commercial software
Version 2.4 6 December 2001 - Added section on WinSCP
Version 2.5 28 January 2002 - Added link to PuTTY configuration guide
Version 3.0 30 January 2002 - Rewrote much of the content
Version 3.1 6 March 2002 - Started Questions and Answers section, (ftp solution)
Version 3.2 27 March 2002 - Added psftp question/answer
Version 3.3 8 April 2002 - Added tunneling HTTP question/answer
Version 4.0 23 April 2002 - Revised format
Version 4.1 11 May 2002 - Expanded SSH server section, reintroduced Revision History
Version 4.1.1 15 May 2002 - Added links to Cygwin packages: XFree86, KDE and Gnome
Version 4.1.2 17 July 2003 - Updated WinSCP links
Version 4.2 25 May 2004 - Edited and updated links
Version 4.2.1 8 July 2004 - Mentions PuTTY's "-cleanup" option
Version 4.2.2 21 March 2005 - Updated WinSCP links
Version 4.2.3 29 March 2006 - Added link to OpenVPN

Questions and Answers

Question:
Subject: FTP solution
I have a school PC running Windows NT terminal server on Citrix on Red Hat and Windows NT server 4.0 at home. I have next to no access at school, and can't install anything, but I can access HTTP (and SSH via a computer on the edge of the firewall). What I want to do is copy files from home, through the firewall, to and from school. Is this possible using a GUI? If not is it possible another way? How?

Answer:
You can use WinSCP to do exactly what you describe. It uses the SSH protocol to transfer files, and provides an "Explorer-like" GUI for dragging/dropping files.

If you can SSH directly to the internal box through the firewall, then WinSCP will work "as-is".

If not, then you need a little bit of port-forwarding to get where you want to go:

• Log onto the firewall, (or the machine that is directly accessible via the firewall), using SSH, forwarding an arbitrary local port to port 22 on the internal machine.
• When you run WinSCP connect to "localhost" on the arbitrary port you selected.

For example:

• Say "schoolfirewall.myschool.edu" is your school's firewall, (or the machine that is visible outside the firewall).
• From "schoolfirewall.myschool.edu", you are trying to access a machine called mypc.myschool.edu", (which is not visible outside the firewall).
• You (randomly) pick port 2200 as the local port to forward.
• Create, (and save), a PuTTY session that connects to "schoolfirewall.myschool.edu" using SSH, with Local Forwarding of port "2200" to "mypc.myschool.edu:22", (specified in the "SSH Tunnels" section).
• Before running WinSCP, run PuTTY using the saved session, (this will connect to the firewall with port 2200 forwarded - you must leave PuTTY connected for the duration of the WinSCP session).
• Run WinSCP. Connect to "localhost" on port 2200. This will in fact be connecting to the SSH server on port 22 of "mypc.myschool.edu".
• Provide the appropriate login details for "mypc.myschool.edu", and you should now be able to drag/drop files to/from the internal machine.

Notes:

• "mypc.myschool.edu", (the internal server), needs to be running an SSH server. Red Hat should have that installed by default. You can also get a version for Windows if you need it, (see the section on SSH Servers).
• If Local Forwarding to "mypc.myschool.edu:22" doesn't work, try using the internal IP address of the machine instead - it's possible that your client or the firewall can't resolve the internal address. For example, if "mypc.myschool.edu" is at IP address "10.0.2.100", try forwarding to "10.0.2.100:22".

Question:
Subject: psftp
Why not use psftp, (PuTTY's implementation of sftp)?

Answer:
Note: sftp is a "secure ftp" program. It works like a regular ftp client, but uses SSH to secure the traffic passing between the client and the server. As of version 3, WinSCP also supports sftp functionality.

I have a couple of niggling concerns with sftp:

• sftp requires sftp-server on the server side. Some sites don't enable sftp-server, so it can be annoying to become overly dependant on sftp. All the functionality of sftp can be achieved with ssh/scp, without the need for new server-side programs.
• sftp adds a new layer of protocol, which introduces new possibilities for security vulnerabilities.

My preference is to stick with tools that are implemented using the "basic" SSH protocols, but I accept the point that sftp would also get the job done, and may be easier to work with in many situations.

Question:
Subject: tunneling HTTP
Can I tunnel all my HTTP traffic through an SSH connection to a friend's Linux host?

Answer:
Tunneling only works from one host to another (single) host.

To tunnel all HTTP requests, you would need to establish tunnels between your machine and every other machine on the internet. Tunneling HTTP requests to a handful of webservers behind a firewall is feasible, but directly securing all HTTP traffic is not.

However, it is possible to tunnel HTTP requests indirectly:

You could install a HTTP proxy on the Linux machine, (such as squid, http://www.squid-cache.org/), then tunnel communications with the proxy, (port 3128 by default for squid). On the client machine, the HTTP proxy would appear to be running on localhost:3128. It would in fact be talking to squid on port 3128 on the Linux box.

This will secure traffic between your machine and the Linux box. However, the actual HTTP requests from squid to the rest of the Internet are not, (and cannot), be secured, (unless you somehow had SSH or SSL access to every other machine on the Internet). Given this fact, there's little to be gained from tunneling HTTP access through a proxy, unless you specifically want to encrypt the data between your machine and the proxy.