tag:blogger.com,1999:blog-69309184284851062652024-02-05T02:26:55.638-08:00Software/HardwareUnknownnoreply@blogger.comBlogger62125tag:blogger.com,1999:blog-6930918428485106265.post-46888885602938587812008-11-24T16:42:00.000-08:002008-11-24T16:43:23.395-08:00the Belgian eID and PDF: Forms and Digital SignaturesSource: http://itext.ugent.be/articles/eid-pdf/index.php?page=4#start<br />===============<br /><h1>the Belgian eID and PDF:<br />Forms and Digital Signatures</h1> <div class="backlight"><p>These are my notes for my <a href="http://govcamp.org/index.php/GovCamp_Brussels">GovCamp</a> presentation in Brussels (September 21, 2006; between OSCon and DrupalCon).</p> <p align="center"><a href="http://govcamp.org/index.php/GovCamp_Brussels"><img src="http://drupalcon.org/files/brussels-2006-timeline.png" border="0" /></a></p> <p>I was asked to talk about the Belgian eID; the eID is the new Belgian identity card in the form of a smartcard. I am not an eID expert, so I decided to change the focus of the presentation to a technology I know rather well: the Portable Document Format (PDF).<br /><br />In this article you will learn:</p> <ul><li>how to create a PDF document containing an interactive form (an AcroForm)</li><li>how to fill this form with data retrieved from the eID</li><li>how to use your eID to add an ordinary digital signature (recipient signature) as well as a certifying signature (author signature) to a PDF document</li></ul> <p>The conclusion will be that you really don't need to be an eID expert to achieve all this. If I can do it, so can you ;-)</p></div><br /><div><a href="http://itext.ugent.be/articles/eid-pdf/index.php?page=3#start">previous page</a> | <a href="http://itext.ugent.be/articles/eid-pdf/index.php?page=1#start">page 1</a> | <a href="http://itext.ugent.be/articles/eid-pdf/index.php?page=2#start">page 2</a> | <a href="http://itext.ugent.be/articles/eid-pdf/index.php?page=3#start">page 3</a> | <a href="http://itext.ugent.be/articles/eid-pdf/index.php?page=4#start">page 4</a> </div><a name="start"></a> <h2>Summary</h2> <p>In this presentation we have learned more about the following topics:</p> <h3>Forms in a PDF document</h3> <p>We have learned:</p> <ul><li>How to create a form using iText</li><li>How to use a PDF form in a web application</li><li>How to use a PDF form as a template</li><li>How to flatten a form</li><li>How to retrieve data from a form</li></ul> <p>Note that XFA forms are not yet supported in iText, and that special usage rights can only be added using official Adobe software.</p> <h3>The Belgian eID and PDF</h3> <p>We have used an eID to:</p> <ol><li>Fill in a form with data retrieved from the eID</li><li>Learn about the different certificates on an eID</li><li>Add one or more recipient (or ordinary) signature to a PDF document</li><li>Add one author (or certifying) signature to a PDF document</li></ol> <p>Note that we have use the qualified (or non-repudiation) certificate to add the signatures. The hash of the PDF document content was generated on a Vasco DigiPass 850 smartcard reader.</p> <p>We didn't go into details. I hope you agree you didn't have to be an eID specialist to understand the principles of using an eID in combination with PDF documents. You can find out more about PDF in my book. At the bottom of this page you'll find some interesting links to pages about the eID.</p> <h2>Extra links</h2> <h3>Resources</h3> <ul><li><a href="http://itext.ugent.be/articles/eid-pdf/eIDpdf.ppt">eIDpdf.ppt:</a> a powerpoint presentation</li><li><a href="http://itext.ugent.be/articles/eid-pdf/eid-pdf.zip">eid-pdf.zip:</a> the source code, jars and PDF files that accompany this article</li></ul> <p>All examples in this article were created using the Java version of iText. There are also some .NET ports of iText available. If you have .NET eID software, it should be possible to adapt the examples so that they work in a .NET environment too.</p> <h3>Other articles on this subject</h3> <ul><li><a href="http://segraves.tripod.com/">Bill Segraves pages:</a> <a href="http://segraves.tripod.com/index4.htm">Using iText to Save PDF Form Data</a></li><li><a href="http://www.codeproject.com/">The Code Project:</a> <a href="http://www.codeproject.com/useritems/Esignature.asp">E-signing PDF documents with iTextSharp</a> by Alaa-eddine Kaddouri </li></ul> <h3>Relevant mailinglist questions regarding signatures</h3> <ul><li>A digital signatures example to get you started in .NET: <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/17455">GMANE</a> <a href="http://www.nabble.com/iTextSharp-tf218061.html#a609655">Nabble</a></li><li>How to use an image for the signature appearance: <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/23250">GMANE</a> <a href="http://www.nabble.com/Image-in-Signature-tf1709085.html#a4640254">Nabble</a></li><li>Appearances created by iText: <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/13396">GMANE</a></li><li>How to find the names of the signature fields: <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/23223">GMANE</a> <a href="http://www.nabble.com/Find-name-of-field-tf1703120.html#a4625010">Nabble</a></li><li>Can I remove a digital signature? <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/16986">GMANE</a> <a href="http://www.nabble.com/Remove-digital-signature-tf154077.html#a428686">Nabble</a></li><li>How to remove a digital signature? <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/24885">GMANE</a> <a href="http://www.nabble.com/Re-2%3A--Signature-on-last-page-of-pdf-tf2110107.html#a5863492">Nabble</a></li><li>How to avoid the removal of a digital signature? <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/24871">GMANE</a> <a href="http://www.nabble.com/Re-2%3A--Signature-on-last-page-of-pdf-tf2110107.html#a5829403">Nabble</a></li><li>Multiple signatures: why don't they work? A very interesting extension of this short article, explaining the different levels of Certifying Signatures: <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/26437">GMANE</a> <a href="http://www.nabble.com/Multiple-signatures---why-don%27t-they-work---tf2520269.html">Nabble</a> </li></ul> <p>Note that some questions can't be answered because some requirements depend on wrong assumptions:</p> <ul><li>You can't change a document without invalidating the signature: <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/24063">GMANE</a> <a href="http://www.nabble.com/Watermark-and-Signatures-tf1905401.html#a5214433">Nabble</a></li><li>You can't change the appearance of an existing digital signature: <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/24053">GMANE</a> <a href="http://www.nabble.com/How-to-edit-the-appearance-of-an-existing-digital-signature--tf1903148.html#a5208365">Nabble</a></li><li>You can't set the Usage Rights of a PDF file with iText: <a href="http://article.gmane.org/gmane.comp.java.lib.itext.general/24071">GMANE</a> <a href="http://www.nabble.com/Changing-document-properties-of-pdf-tf1905464.html#a5218994">Nabble</a></li></ul> <h3>About the book</h3> <p>This article was made from the notes of my presentation for GovCamp Brussels. It is an extended version of <a href="http://itext.ugent.be/itext-in-action/chapter.php?chapter=22">Appendix D</a> of the book <a href="http://itext.ugent.be/itext-in-action/">iText in Action</a>.</p> <p>You will need this book if you want to know more about creating and manipulating PDF documents.</p> <p>For instance:</p> <ul><li><a href="http://itext.ugent.be/itext-in-action/chapter.php?chapter=6">How to create tables</a></li><li><a href="http://itext.ugent.be/itext-in-action/chapter.php?chapter=14">How to create PDF documents in an automated process</a></li><li><a href="http://itext.ugent.be/itext-in-action/chapter.php?chapter=15">How to create forms</a></li><li><a href="http://itext.ugent.be/itext-in-action/chapter.php?chapter=16">How to fill a form programmatically</a></li><li><a href="http://itext.ugent.be/itext-in-action/chapter.php?chapter=16">How to sign and verify PDF documents</a></li><li>and much more besides...</li></ul> <h3>About the eID</h3> <p>This is a list of useful links if you want to know more about the eID</p> <ol><li><a class="ext" target="_blank" href="http://homes.esat.kuleuven.be/%7Edecockd/wiki/bin/view.cgi">GoDot.be:</a> the website of Danny De Cock</li><li><a class="ext" target="_blank" href="http://eid.belgium.be/">eid.belgium.be:</a> Belgium's eID portal</li><li><a class="ext" target="_blank" href="http://www.cosic.esat.kuleuven.be/publications/article-769.pdf">The Belgian Identity Card (Overview):</a> a short introduction to the eID written by Danny De Cock, Christopher Wolf, and Bart Preneel.</li><li><a class="ext" target="_blank" href="http://www.fidis.net/fileadmin/fidis/deliverables/fidis-wp3-del3.6.study_on_id_documents.pdf">FIDIS:</a> a study on id documents</li><li><a class="ext" target="_blank" href="http://www.rijksregister.fgov.be/">rijksregister.fgov.be:</a> the National Registry</li><li><a class="ext" target="_blank" href="http://repository.eid.belgium.be/EN/index.html">Certipost:</a> check and download certificates</li><li><a class="ext" target="_blank" href="http://www.microsoft.com/belux/nl/eid/">MicroSoft:</a> the eID page</li><li><a class="ext" target="_blank" href="http://www.lexlewis.be/thesis/thesis_eID_Alexander_Goossens.pdf">Thesis Alexander Goossens (in Dutch):</a> eID; Wat is het? Hoe werkt het? Wat zijn de mogelijkheden?</li></ol> <h3>Further reading on cryptography and digital signatures</h3> <ol><li>Wikipedia: <a class="ext" target="_blank" href="http://en.wikipedia.org/wiki/Digital_signature">digital signature</a>, <a class="ext" target="_blank" href="http://en.wikipedia.org/wiki/Certificate_authority">Certificate Authority</a>, <a class="ext" target="_blank" href="http://en.wikipedia.org/wiki/Public_key_infrastructure">PKI</a>,...</li><li><a class="ext" target="_blank" href="http://www.pki-page.org/">The PKI Page</a></li><li><a class="ext" target="_blank" href="http://www.bouncycastle.org/">Bouncy Castle:</a> Java cryptography resources and open source code</li><li><a class="ext" target="_blank" href="http://www.swiss.ai.mit.edu/6.805/student-papers/fall97-papers/fillingham-sig.html">Legal issues:</a> A Comparison of Digital and Handwritten Signatures (Paper for MIT 6.805/STS085: Ethics and Law on the Electronic Frontier, Fall 1997)</li><li><a class="ext" target="_blank" href="http://www.cs.dartmouth.edu/%7Esws/papers/cms02.pdf">Digital Signatures and Electronic Documents: A Cautionary Tale</a> (IFIP Conference on Communications and Multimedia Security, September 2002)</li></ol> <h2>Acknowledgements</h2> <p>I would like to thank the following people:</p> <ul><li><b>Paulo Soares</b> wrote the code to provide support for ordinary signatures in iText. He also wrote the article <a class="ext" target="_blank" href="http://itextpdf.sourceforge.net/howtosign.html">How to sign a PDF using iText</a></li><li><b>Antonio Iacono</b> provided source code to support certifying signatures in iText</li><li><b>Danny De Cock</b> wrote the <a class="ext" target="_blank" href="http://homes.esat.kuleuven.be/%7Edecockd/wiki/bin/view.cgi/Main/AppletEidCardsUtilityTOC">GoDot tool</a> and maintains a <a href="http://godot.be/">site</a> full of useful information on the eID.</li><li><b>Philippe Frankinet</b> wrote the first code sample on <a class="ext" target="_blank" href="http://article.gmane.org/gmane.comp.java.lib.itext.general/21374">how to sign a PDF using iText</a></li><li><b>Bart Van Herreweghe</b> for the invitation to present iText at <a class="ext" target="_blank" href="http://govcamp.org/index.php/GovCamp_Brussels">GovCamp Brussels</a></li></ul>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6930918428485106265.post-89950313606720243232008-11-24T16:12:00.000-08:002008-11-24T16:14:26.654-08:00iText digital signature exampleSource: http://article.gmane.org/gmane.comp.java.lib.itext.general/21374<br />==========================<br /><br /><p> </p><span class="432044609-03032006">Dear Danny, Bruno and Paulo</span> <div><span class="432044609-03032006">I've also tested iText, the EID card and IAIK (<a href="http://jce.iaik.tugraz.at/">http://jce.iaik.tugraz.at</a>) without any problem.</span></div> <div> <span class="432044609-03032006"></span> <br /></div> <div><span class="432044609-03032006">Here is the code :</span></div> <div> <span class="432044609-03032006"></span> </div> <div><span class="432044609-03032006">package be.nsi.security.signature.pdf;</span></div> <div> </div> <div><span class="432044609-03032006">import java.io.ByteArrayOutputStream;<br />import java.io.File;<br />import java.io.FileNotFoundException;<br />import java.io.FileOutputStream;<br />import java.io.InputStream;<br />import java.security.MessageDigest;<br />import java.security.cert.Certificate;</span></div> <div> </div> <div><span class="432044609-03032006">import com.lowagie.text.*;<br />import com.lowagie.text.pdf.*;</span></div> <div> </div> <div><span class="432044609-03032006">import be.godot.sc.engine.BelpicCard;</span></div> <div> </div> <div><span class="432044609-03032006">public class PDFSign<br />{<br /> String[] args;<br /><br /> public PDFSign(String[] args)<br /> {<br /> super();<br /> this.args = args;<br /> }<br /><br /> public void selfSignMode() throws Throwable<br /> {<br /> File file = new File(args[0]);<br /> if (!file.exists())<br /> throw new FileNotFoundException("File "+ file.getPath() + "doesn't exist");<br /> <br /> PdfReader reader = new PdfReader(file.getPath());<br /> FileOutputStream fout = new FileOutputStream(args[0]+".selfsignmode.pdf");<br /> PdfStamper stamper = PdfStamper.createSignature(reader, fout, '\0');<br /> PdfSignatureAppearance sap = stamper.getSignatureAppearance();<br /> <br /> Certificate[] certs = new Certificate[1];<br /> <br /> BelpicCard scd = new BelpicCard("");<br /> certs[0] = scd.getNonRepudiationCertificate();</span></div> <div> </div> <div><span class="432044609-03032006"> //sap.setCrypto(iKeyStore.getPrivateKey(eidCertificate.getAlias(),null), certs, null, PdfSignatureAppearance.WINCER_SIGNED);<br /> sap.setCrypto(null, certs, null, PdfSignatureAppearance.SELF_SIGNED);<br /> sap.setReason("How to use iText with the new belgian electronic identity card");<br /> sap.setLocation("Belgium");<br /> // comment next line to have an invisible signature<br /> sap.setVisibleSignature(new Rectangle(100, 100, 200, 200), 1, null);<br /> sap.setExternalDigest(new byte[128], new byte[20], "RSA");<br /> sap.preClose();<br /> <br /> // Self-Sign mode<br /> PdfPKCS7 sig = sap.getSigStandard().getSigner(); <br /> <br /> byte[] content = streamToByteArray(sap.getRangeStream());<br /> byte[] hash= MessageDigest.getInstance("SHA-1").digest(content);<br /> byte[] signatureBytes = scd.generateNonRepudiationSignature(hash);<br /> <br /> sig.setExternalDigest(signatureBytes, null, "RSA");<br /> PdfDictionary dic = new PdfDictionary();<br /> dic.put(PdfName.CONTENTS, new PdfString(sig.getEncodedPKCS1()).setHexWriting(true));<br /> sap.close(dic); <br /> }<br /><br /> public static byte[] streamToByteArray(InputStream stream) throws Throwable<br /> { <br /> if (stream == null)<br /> {<br /> return null;<br /> }<br /> else<br /> {<br /> ByteArrayOutputStream byteArray = new ByteArrayOutputStream();<br /> byte buffer[] = new byte[1024];<br /> int c = 0;<br /> while ( (c = stream.read(buffer)) > 0)<br /> {<br /> byteArray.write(buffer, 0, c);<br /> }<br /> byteArray.flush();<br /> return byteArray.toByteArray();<br /> } <br /> }</span></div> <div> </div> <div><span class="432044609-03032006"><br /> public static void main(String[] args)<br /> {<br /> try<br /> {<br /> PDFSign pdfsign = new PDFSign(args);<br /> pdfsign.selfSignMode(); <br /> }<br /> catch(Throwable t)<br /> {<br /> t.printStackTrace(); <br /> }<br /> }<br />}</span></div> <div> </div> <table align="left" border="0" cellpadding="0" cellspacing="0" width="401"><tbody><tr><td> <p class="MsoNormal"> </p> </td></tr> <tr><td> <p class="MsoNormal"><span lang="FR-BE"> </span></p> </td></tr> <tr><td> <p class="MsoNormal"><span lang="FR-BE"> Philippe Frankinet<br /> </span><span lang="FR-BE">Technical Analyst (p.frankinet<a href="mailto:e.mail%20%3Cat%3E%20nsi-sa.be"><span> <at> nsi-sa.be</span></a></span><span>)</span></p> </td></tr> <tr><td> <p class="MsoNormal"><span lang="FR-BE"> NSI S.A. </span><span lang="FR-BE"> </span><span lang="FR-BE"><a href="http://www.nsi-sa.be%29e/"><span>(www.nsi-sa.be)</span></a></span><span lang="FR-BE"><br /></span><span lang="FR-BE"> Chaussée de Bruxelles, 174 A<br /> B-4340 Awans - Belgique<br /> Tél. : +32 (0)4 239 91 50 - Fax : +32 (0)4 246 13 08</span></p></td></tr></tbody></table>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6930918428485106265.post-35400765701412496942008-11-24T16:11:00.001-08:002008-11-24T16:11:59.422-08:00How to sign a PDF using iText and iTextSharpSource: http://itextpdf.sourceforge.net/howtosign.html<br />====================<br /><h1><center>How to sign a PDF using iText and iTextSharp</center></h1> <hr /> <h2>Index</h2> <ul><li><a href="http://itextpdf.sourceforge.net/howtosign.html#introduction">Introduction</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#howtosign">How to sign</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#howtocertify">How to certify</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#howtosignmult">How to do multiple signatures</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#howtoverify">How to verify</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#keysself">How to generate the keys for the self signed mode</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#signveriwin">How to sign with VeriSign and Microsoft Windows Certificate</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#keyfromca">How to acquire a permanent key from a CA</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#signextstd">How to sign with an external signature and a standard filter</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#signextdic">How to sign with an external signature dictionary (DSE200 example)</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#signextdiccms">How to sign with an external signature dictionary using BouncyCastle CMS</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#signextdicattrib">How to sign with an external signature dictionary (authenticatedAttributes example)</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#howtoverifycs">How to verify with iTextSharp</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#signextitextsharp1">How to sign with a smartcard using an external signature dictionary with iTextSharp, CAPICOM and .NET 1.1</a> </li><li><a href="http://itextpdf.sourceforge.net/howtosign.html#signextitextsharp2">How to sign with a smartcard using an external signature dictionary with iTextSharp and .NET 2.0</a> </li></ul> <hr /> <h2><a name="introduction">Introduction</a></h2> <p> iText supports visible and invisible signing using the following modes: </p><p> </p><ul><li>Self signed (Adobe.PPKLite) </li><li>VeriSign plug-in (VeriSign.PPKVS) </li><li>Windows Certificate Security (Adobe.PPKMS) </li></ul> <p> Signing and verifying with iText is easy, and it's always done the same way, the difficult part comes with the key and certificate generation. This is a quick guide and doesn't replace the know how you should have on cryptography to make the best use of the technology. </p><hr /> <h2><a name="howtosign">How to sign</a></h2> <p> Signing is done with this simple code: </p><p> </p><pre>KeyStore ks = KeyStore.getInstance("pkcs12");<br />ks.load(new FileInputStream("my_private_key.pfx"), "my_password".toCharArray());<br />String alias = (String)ks.aliases().nextElement();<br />PrivateKey key = (PrivateKey)ks.getKey(alias, "my_password".toCharArray());<br />Certificate[] chain = ks.getCertificateChain(alias);<br />PdfReader reader = new PdfReader("original.pdf");<br />FileOutputStream fout = new FileOutputStream("signed.pdf");<br />PdfStamper stp = PdfStamper.createSignature(reader, fout, '\0');<br />PdfSignatureAppearance sap = stp.getSignatureAppearance();<br />sap.setCrypto(key, chain, null, PdfSignatureAppearance.WINCER_SIGNED);<br />sap.setReason("I'm the author");<br />sap.setLocation("Lisbon");<br />// comment next line to have an invisible signature<br />sap.setVisibleSignature(new Rectangle(100, 100, 200, 200), 1, null);<br />stp.close();<br /></pre> <p> How to obtain the private key <code>my_private_key.pfx</code> and the certificates <code>my_certificates.p7b</code> will be explained later. </p><hr /> <h2><a name="howtocertify">How to certify</a></h2> <p>In addition to signing the documents can be certified. Any change to the document that goes against the certification level will make it lose the certified status. The certification level can be: <code>NOT_CERTIFIED</code>, <code>CERTIFIED_NO_CHANGES_ALLOWED</code>, <code>CERTIFIED_FORM_FILLING</code> and <code>CERTIFIED_FORM_FILLING_AND_ANNOTATIONS</code>. To certify a document sign it as in the previous example but add </p><p> </p><pre>sap.setCertificationLevel(PdfSignatureApperance.CERTIFIED_NO_CHANGES_ALLOWED);<br /></pre> before closing. This implementation of certification will only work in Acrobat 7 and later. <hr /> <h2><a name="howtosignmult">How to do multiple signatures</a></h2> <p> Signing as just described will invalidate any existing signatures in the document. To add another signature a new revision is needed. To create a new revision just create the <code>PdfStamper</code> the following way and do the rest as already explained: </p><p> </p><pre>PdfStamper stp = PdfStamper.createSignature(reader, fout, '\0', null, true);<br /></pre> <hr /> <h2><a name="howtoverify">How to verify</a></h2> <p> Verifying is a three step process: </p><ul><li>Was the document changed? </li><li>What revision does the signature cover? Does the signature cover all the document? </li><li>Can the signature certificates be verified in your trusted identities store? </li></ul> Here's an example on how to do it: <p> </p><pre>KeyStore kall = PdfPKCS7.loadCacertsKeyStore();<br />PdfReader reader = new PdfReader("my_signed_doc.pdf");<br />AcroFields af = reader.getAcroFields();<br />ArrayList names = af.getSignatureNames();<br />for (int k = 0; k < names.size(); ++k) {<br /> String name = (String)names.get(k);<br /> System.out.println("Signature name: " + name);<br /> System.out.println("Signature covers whole document: " + af.signatureCoversWholeDocument(name));<br /> System.out.println("Document revision: " + af.getRevision(name) + " of " + af.getTotalRevisions());<br /> // Start revision extraction<br /> FileOutputStream out = new FileOutputStream("revision_" + af.getRevision(name) + ".pdf");<br /> byte bb[] = new byte[8192];<br /> InputStream ip = af.extractRevision(name);<br /> int n = 0;<br /> while ((n = ip.read(bb)) > 0)<br /> out.write(bb, 0, n);<br /> out.close();<br /> ip.close();<br /> // End revision extraction<br /> PdfPKCS7 pk = af.verifySignature(name);<br /> Calendar cal = pk.getSignDate();<br /> Certificate pkc[] = pk.getCertificates();<br /> System.out.println("Subject: " + PdfPKCS7.getSubjectFields(pk.getSigningCertificate()));<br /> System.out.println("Document modified: " + !pk.verify());<br /> Object fails[] = PdfPKCS7.verifyCertificates(pkc, kall, null, cal);<br /> if (fails == null)<br /> System.out.println("Certificates verified against the KeyStore");<br /> else<br /> System.out.println("Certificate failed: " + fails[1]);<br />}<br /></pre> <p> Note that if you were using the self signed mode you would have to load <code>kall</code> with the certificate provided by whoever signed the document. The certificate could be loaded this way: </p><p> </p><pre>CertificateFactory cf = CertificateFactory.getInstance("X509");<br />Collection col = cf.generateCertificates(new FileInputStream("self.p7b"));<br />KeyStore kall = KeyStore.getInstance(KeyStore.getDefaultType());<br />kall.load(null, null);<br />for (Iterator it = col.iterator(); it.hasNext();) {<br /> X509Certificate cert = (X509Certificate)it.next();<br /> kall.setCertificateEntry(cert.getSerialNumber().toString(Character.MAX_RADIX), cert);<br />}<br /></pre> <hr /> <h2><a name="keysself">How to generate the keys for the self signed mode</a></h2> <p> To generate a key to self sign: </p><p> </p><pre>keytool -genkey -keyalg RSA -alias myname -keypass password -keystore keystore.ks -dname "cn=Paulo Soares, c=PT"<br /></pre> <p> Answer the questions and at the end you have a self signed certificate at <code>keystore.ks</code>. You can omit the -dname option but dont't forget to put the 2 letter country code when asked to. You can now sign the document with: </p><p> </p><pre>KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());<br />ks.load(new FileInputStream("keystore.ks"), "keystore_password".toCharArray());<br />String alias = (String)ks.aliases().nextElement();<br />PrivateKey key = (PrivateKey)ks.getKey(alias, "password".toCharArray());<br />Certificate[] chain = ks.getCertificateChain(alias);<br />PdfReader reader = new PdfReader("original.pdf");<br />FileOutputStream fout = new FileOutputStream("signed.pdf");<br />PdfStamper stp = PdfStamper.createSignature(reader, fout, '\0');<br />PdfSignatureAppearance sap = stp.getSignatureAppearance();<br />sap.setCrypto(key, chain, null, PdfSignatureAppearance.SELF_SIGNED);<br />sap.setReason("I'm the author");<br />sap.setLocation("Lisbon");<br />// comment next line to have an invisible signature<br />sap.setVisibleSignature(new Rectangle(100, 100, 200, 200), 1, null);<br />stp.close();<br /></pre> <p> To generate the certificate that can be validated at the receiving end do: </p><p> </p><pre>keytool -export -alias myname -file export.cer -keystore keystore.ks<br /></pre> <p> The generated certificate <code>export.cer</code> can be imported by iText or Acrobat for validation. </p><hr /> <h2><a name="signveriwin">How to sign with VeriSign and Microsoft Windows Certificate</a></h2> <p> To sign documents with VeriSign you need a key that must be certified by VeriSign. You can acquire a 60 day trial key or buy one at <a href="https://digitalid.verisign.com/client/class1MS.htm">https://digitalid.verisign.com/client/class1MS.htm</a>. </p><p> The Acrobat VeriSign plug-in (that you can get <a href="http://www.verisign.com/products-services/security-services/pki/pki-application/trusted-form-signing/index.html">here</a>) will only work with VeriSign certified keys but the Microsoft Windows Certificate will work with any trusted certificate and in addition to the VeriSign certificate you can also use a free Thawte certificate found at <a href="https://www.thawte.com/email/index.html">https://www.thawte.com/email/index.html</a>. </p><p> After going through all the CA instructions you'll end up with a certificate in your IE. You'll have to export the private key including all the certificated in the certification path. The result is a .pfx file with your private key that can be used to sign the documents. </p><hr /> <h2><a name="keyfromca">How to acquire a permanent key from a CA</a></h2> <p> To acquire a permanent key from a CA such as VeriSign you usually need to go through the following steps: </p><p> </p><ol><li>Generate a self signed key with<br /><code>keytool -genkey -keyalg RSA</code> </li><li>Generate a certificate signing request with<br /><code>keytool -certreq</code><br />the result is base64 data that you must cut and paste in the CA site when prompted </li><li>The CA will send a certificate reply chain that you must import with<br /><code>keytool -import</code><br />using the same alias </li></ol> <hr /> <h2><a name="signextstd">How to sign with an external signature and a standard filter</a></h2> <p> Some examples follow. Replace with your signing code in the appropriate places. </p><p> <b>An example with an external hash in Windows Certificate Mode</b> </p><p> </p><pre>KeyStore ks = KeyStore.getInstance("pkcs12");<br />ks.load(new FileInputStream("my_private_key.pfx"), "my_password".toCharArray());<br />String alias = (String)ks.aliases().nextElement();<br />PrivateKey key = (PrivateKey)ks.getKey(alias, "my_password".toCharArray());<br />Certificate[] chain = ks.getCertificateChain(alias);<br />PdfReader reader = new PdfReader("original.pdf");<br />FileOutputStream fout = new FileOutputStream("signed.pdf");<br />PdfStamper stp = PdfStamper.createSignature(reader, fout, '\0');<br />PdfSignatureAppearance sap = stp.getSignatureAppearance();<br />sap.setCrypto(key, chain, null, PdfSignatureAppearance.WINCER_SIGNED);<br />sap.setReason("I'm the author");<br />sap.setLocation("Lisbon");<br />// comment next line to have an invisible signature<br />sap.setVisibleSignature(new Rectangle(100, 100, 200, 200), 1, null);<br />sap.setExternalDigest(null, new byte[20], null);<br />sap.preClose();<br />MessageDigest messageDigest = MessageDigest.getInstance("SHA1");<br />byte buf[] = new byte[8192];<br />int n;<br />InputStream inp = sap.getRangeStream();<br />while ((n = inp.read(buf)) > 0) {<br /> messageDigest.update(buf, 0, n);<br />}<br />byte hash[] = messageDigest.digest();<br />PdfSigGenericPKCS sg = sap.getSigStandard();<br />PdfLiteral slit = (PdfLiteral)sg.get(PdfName.CONTENTS);<br />byte[] outc = new byte[(slit.getPosLength() - 2) / 2];<br />PdfPKCS7 sig = sg.getSigner();<br />sig.setExternalDigest(null, hash, null);<br />PdfDictionary dic = new PdfDictionary();<br />byte[] ssig = sig.getEncodedPKCS7();<br />System.arraycopy(ssig, 0, outc, 0, ssig.length);<br />dic.put(PdfName.CONTENTS, new PdfString(outc).setHexWriting(true));<br />sap.close(dic);<br /></pre> <p> <b>An example with an external hash and signature in Windows Certificate Mode</b> </p><p> </p><pre>KeyStore ks = KeyStore.getInstance("pkcs12");<br />ks.load(new FileInputStream("my_private_key.pfx"), "my_password".toCharArray());<br />String alias = (String)ks.aliases().nextElement();<br />PrivateKey key = (PrivateKey)ks.getKey(alias, "my_password".toCharArray());<br />Certificate[] chain = ks.getCertificateChain(alias);<br />PdfReader reader = new PdfReader("original.pdf");<br />FileOutputStream fout = new FileOutputStream("signed.pdf");<br />PdfStamper stp = PdfStamper.createSignature(reader, fout, '\0');<br />PdfSignatureAppearance sap = stp.getSignatureAppearance();<br />sap.setCrypto(key, chain, null, PdfSignatureAppearance.WINCER_SIGNED);<br />sap.setReason("I'm the author");<br />sap.setLocation("Lisbon");<br />// comment next line to have an invisible signature<br />sap.setVisibleSignature(new Rectangle(100, 100, 200, 200), 1, null);<br />sap.setExternalDigest(new byte[128], new byte[20], "RSA");<br />sap.preClose();<br />MessageDigest messageDigest = MessageDigest.getInstance("SHA1");<br />byte buf[] = new byte[8192];<br />int n;<br />InputStream inp = sap.getRangeStream();<br />while ((n = inp.read(buf)) > 0) {<br /> messageDigest.update(buf, 0, n);<br />}<br />byte hash[] = messageDigest.digest();<br />PdfSigGenericPKCS sg = sap.getSigStandard();<br />PdfLiteral slit = (PdfLiteral)sg.get(PdfName.CONTENTS);<br />byte[] outc = new byte[(slit.getPosLength() - 2) / 2];<br />PdfPKCS7 sig = sg.getSigner();<br />Signature sign = Signature.getInstance("SHA1withRSA");<br />sign.initSign(key);<br />sign.update(hash);<br />sig.setExternalDigest(sign.sign(), hash, "RSA");<br />PdfDictionary dic = new PdfDictionary();<br />byte[] ssig = sig.getEncodedPKCS7();<br />System.arraycopy(ssig, 0, outc, 0, ssig.length);<br />dic.put(PdfName.CONTENTS, new PdfString(outc).setHexWriting(true));<br />sap.close(dic);<br /></pre> <p> <b>An example with an external signature in Self Sign Mode</b> </p><p> </p><pre>KeyStore ks = KeyStore.getInstance("pkcs12");<br />ks.load(new FileInputStream("my_private_key.pfx"), "my_password".toCharArray());<br />String alias = (String)ks.aliases().nextElement();<br />PrivateKey key = (PrivateKey)ks.getKey(alias, "my_password".toCharArray());<br />Certificate[] chain = ks.getCertificateChain(alias);<br />PdfReader reader = new PdfReader("original.pdf");<br />FileOutputStream fout = new FileOutputStream("signed.pdf");<br />PdfStamper stp = PdfStamper.createSignature(reader, fout, '\0');<br />PdfSignatureAppearance sap = stp.getSignatureAppearance();<br />sap.setCrypto(key, chain, null, PdfSignatureAppearance.SELF_SIGNED);<br />sap.setReason("I'm the author");<br />sap.setLocation("Lisbon");<br />// comment next line to have an invisible signature<br />sap.setVisibleSignature(new Rectangle(100, 100, 200, 200), 1, null);<br />sap.setExternalDigest(new byte[128], null, "RSA");<br />sap.preClose();<br />PdfPKCS7 sig = sap.getSigStandard().getSigner();<br />Signature sign = Signature.getInstance("SHA1withRSA");<br />sign.initSign(key);<br />byte buf[] = new byte[8192];<br />int n;<br />InputStream inp = sap.getRangeStream();<br />while ((n = inp.read(buf)) > 0) {<br /> sign.update(buf, 0, n);<br />}<br />sig.setExternalDigest(sign.sign(), null, "RSA");<br />PdfDictionary dic = new PdfDictionary();<br />dic.put(PdfName.CONTENTS, new PdfString(sig.getEncodedPKCS1()).setHexWriting(true));<br />sap.close(dic);<br /></pre> <hr /> <h2><a name="signextdic">How to sign with an external signature dictionary (DSE200 example)</a></h2> <p> <b>An example with an external signature dictionary with the nCipher DSE 200 (this example is obsolete and should only be used with Acrobat 5)</b> </p><p> </p><pre>PdfReader reader = new PdfReader("original.pdf");<br />FileOutputStream fout = new FileOutputStream("signed.pdf");<br />PdfStamper stp = PdfStamper.createSignature(reader, fout, '\0');<br />PdfSignatureAppearance sap = stp.getSignatureAppearance();<br />// comment next line to have an invisible signature<br />sap.setVisibleSignature(new Rectangle(100, 100, 200, 200), 1, null);<br />sap.setLayer2Text("This is some custom made text.");<br />PdfDictionary dic = new PdfDictionary();<br />dic.put(PdfName.FT, PdfName.SIG);<br />dic.put(PdfName.FILTER, new PdfName("nCipher.TimeSeal"));<br />dic.put(PdfName.SUBFILTER, new PdfName("ntse.dse.1"));<br />sap.setCryptoDictionary(dic);<br />HashMap exc = new HashMap();<br />exc.put(PdfName.CONTENTS, new Integer(0x1802));<br />exc.put(PdfName.M, new Integer(0x19));<br />exc.put(PdfName.NAME, new Integer(0x1f));<br />sap.preClose(exc);<br /><br />MessageDigest messageDigest = MessageDigest.getInstance("MD5");<br />byte buf[] = new byte[8192];<br />int n;<br />InputStream inp = sap.getRangeStream();<br />while ((n = inp.read(buf)) > 0) {<br /> messageDigest.update(buf, 0, n);<br />}<br />byte hash[] = messageDigest.digest();<br />// The DataImprint object is the "data to be time-stamped"<br />DataImprint dataImprint = new DataImprint();<br />dataImprint.setHashAlgorithm(new AlgorithmIdentifier(AlgorithmIdentifier.md5));<br />dataImprint.setHashedData(hash);<br /><br />TimeStampRequest tsq = new TimeStampRequest();<br /><br />tsq.setDataImprint(dataImprint);<br /><br />// Set the nonce. It allows the client to verify the timeliness<br />// of the response when no trusted local clock is available.<br />Random rand = new Random(new Date().getTime());<br />tsq.setNonce(BigInteger.valueOf(rand.nextLong()));<br /><br />// Request that the TSA signer certificate be included in the TimeStampToken.<br />// This allows a TimeStampToken signature to be verified without an<br />// external TSA certificate.<br />tsq.setCertReq(true);<br /><br />// Set the request policy under which this TimeStampToken should be issued.<br />tsq.setReqPolicy(new PolicyIdentifier(PolicyIdentifier.serviceClass));<br /><br />// Create the encoded request.<br />byte[] encodedTSQ = tsq.encodeRequest();<br /><br />// Send the request to the Trusted Time StampServer<br />// and get the response (an encoded TimeStampToken)<br />TimeStampServerTCP tss = new TimeStampServerTCP("dse200 address");<br />byte[] encodedTST = tss.submitRequest(encodedTSQ, REQUEST_TIMEOUT);<br />// Decode the token to access the contained information<br />TimeStampToken tst = new TimeStampToken(encodedTST);<br /><br />if(!tst.verifySignature())<br /> throw new SignatureException("TimeStampToken signature invalid!");<br />TSTInfo tstInfo = tst.getTSTInfo();<br />GregorianCalendar cal = new GregorianCalendar(TimeZone.getTimeZone("GMT"));<br />cal.setTime(tstInfo.getDate());<br />StringTokenizer tk = new StringTokenizer(tstInfo.getTsaName(), ";");<br />String sn = "";<br />while (tk.hasMoreTokens()) {<br /> String x;<br /> if ((x = tk.nextToken()).startsWith("CN=")) {<br /> sn = x.substring(3);<br /> break;<br /> }<br />}<br /><br />PdfDictionary dic2 = new PdfDictionary();<br />dic2.put(PdfName.CONTENTS, new PdfString(encodedTST).setHexWriting(true));<br />dic2.put(PdfName.M, new PdfDate(cal));<br />dic2.put(PdfName.NAME, new PdfString(sn));<br />sap.close(dic2);<br /></pre> <hr /> <h2><a name="signextdiccms">How to sign with an external signature dictionary using BouncyCastle CMS</a></h2> <p> <b>An example with an external signature dictionary with the BouncyCastle CMS. The detached and encapsulated methods are presented.</b> </p><p> </p><pre>import com.lowagie.text.Rectangle;<br />import com.lowagie.text.pdf.PdfDate;<br />import com.lowagie.text.pdf.PdfDictionary;<br />import com.lowagie.text.pdf.PdfName;<br />import com.lowagie.text.pdf.PdfPKCS7;<br />import com.lowagie.text.pdf.PdfReader;<br />import com.lowagie.text.pdf.PdfSignature;<br />import com.lowagie.text.pdf.PdfSignatureAppearance;<br />import com.lowagie.text.pdf.PdfStamper;<br />import com.lowagie.text.pdf.PdfString;<br />import java.io.ByteArrayOutputStream;<br />import java.io.FileInputStream;<br />import java.io.FileOutputStream;<br />import java.io.IOException;<br />import java.io.InputStream;<br />import java.io.OutputStream;<br />import java.util.GregorianCalendar;<br />import java.util.HashMap;<br />import org.bouncycastle.cms.CMSException;<br />import org.bouncycastle.jce.provider.BouncyCastleProvider;<br />import java.security.cert.Certificate;<br />import java.security.PrivateKey;<br />import java.security.KeyStore;<br />import java.security.MessageDigest;<br />import java.security.Security;<br />import java.security.cert.CertStore;<br />import java.security.cert.CollectionCertStoreParameters;<br />import java.security.cert.X509Certificate;<br />import java.util.ArrayList;<br />import org.bouncycastle.cms.CMSProcessable;<br />import org.bouncycastle.cms.CMSProcessableByteArray;<br />import org.bouncycastle.cms.CMSSignedData;<br />import org.bouncycastle.cms.CMSSignedDataGenerator;<br /><br />public static void SignDetached() {<br /> try {<br /> Security.addProvider(new BouncyCastleProvider());<br /> KeyStore ks = KeyStore.getInstance("PKCS12");<br /> ks.load(new FileInputStream("my_private_key.pfx"), "password".toCharArray());<br /> String alias = (String)ks.aliases().nextElement();<br /> PrivateKey key = (PrivateKey)ks.getKey(alias, "password".toCharArray());<br /> Certificate[] chain = ks.getCertificateChain(alias);<br /><br /> PdfReader reader = new PdfReader("original.pdf");<br /> PdfStamper stp = PdfStamper.createSignature(reader, new FileOutputStream("signedDetached.pdf"), '\0');<br /> PdfSignatureAppearance sap = stp.getSignatureAppearance();<br /> sap.setVisibleSignature(new Rectangle(100, 100, 300, 200), 1, null);<br /> sap.setSignDate(new GregorianCalendar());<br /> sap.setCrypto(null, chain, null, null);<br /> sap.setReason("I like to sign");<br /> sap.setLocation("Universe");<br /> sap.setAcro6Layers(true);<br /> sap.setRender(PdfSignatureAppearance.SignatureRenderNameAndDescription);<br /> PdfSignature dic = new PdfSignature(PdfName.ADOBE_PPKLITE, PdfName.ADBE_PKCS7_DETACHED);<br /> dic.setDate(new PdfDate(sap.getSignDate()));<br /> dic.setName(PdfPKCS7.getSubjectFields((X509Certificate)chain[0]).getField("CN"));<br /> if (sap.getReason() != null)<br /> dic.setReason(sap.getReason());<br /> if (sap.getLocation() != null)<br /> dic.setLocation(sap.getLocation());<br /> sap.setCryptoDictionary(dic);<br /> int csize = 4000;<br /> HashMap exc = new HashMap();<br /> exc.put(PdfName.CONTENTS, new Integer(csize * 2 + 2));<br /> sap.preClose(exc);<br /><br /> CMSSignedDataGenerator generator = new CMSSignedDataGenerator();<br /> generator.addSigner(key, (X509Certificate)chain[0], CMSSignedDataGenerator.DIGEST_SHA1);<br /><br /> ArrayList list = new ArrayList();<br /> for (int i = 0; i < chain.length; i++) {<br /> list.add(chain[i]);<br /> }<br /> CertStore chainStore = CertStore.getInstance("Collection", new CollectionCertStoreParameters(list), "BC");<br /> generator.addCertificatesAndCRLs(chainStore);<br /> CMSProcessable content = new CMSProcessableRange(sap);<br /> CMSSignedData signedData = generator.generate(content, false, "BC");<br /> byte[] pk = signedData.getEncoded();<br /><br /> byte[] outc = new byte[csize];<br /><br /> PdfDictionary dic2 = new PdfDictionary();<br /><br /> System.arraycopy(pk, 0, outc, 0, pk.length);<br /><br /> dic2.put(PdfName.CONTENTS, new PdfString(outc).setHexWriting(true));<br /> sap.close(dic2);<br /> }<br /> catch (Exception e) {<br /> e.printStackTrace();<br /> }<br />}<br /><br />public static void signEncapsulated() {<br /> try {<br /> Security.addProvider(new BouncyCastleProvider());<br /> KeyStore ks = KeyStore.getInstance("PKCS12");<br /> ks.load(new FileInputStream("my_private_key.pfx"), "password".toCharArray());<br /> String alias = (String)ks.aliases().nextElement();<br /> PrivateKey key = (PrivateKey)ks.getKey(alias, "password".toCharArray());<br /> Certificate[] chain = ks.getCertificateChain(alias);<br /><br /> PdfReader reader = new PdfReader("original.pdf");<br /> PdfStamper stp = PdfStamper.createSignature(reader, new FileOutputStream("signedEncapsulated.pdf"), '\0');<br /> PdfSignatureAppearance sap = stp.getSignatureAppearance();<br /> sap.setVisibleSignature(new Rectangle(100, 100, 300, 200), 1, null);<br /> sap.setSignDate(new GregorianCalendar());<br /> sap.setCrypto(null, chain, null, null);<br /> sap.setReason("I like to sign");<br /> sap.setLocation("Universe");<br /> sap.setAcro6Layers(true);<br /> sap.setRender(PdfSignatureAppearance.SignatureRenderNameAndDescription);<br /> PdfSignature dic = new PdfSignature(PdfName.ADOBE_PPKMS, PdfName.ADBE_PKCS7_SHA1);<br /> dic.setDate(new PdfDate(sap.getSignDate()));<br /> dic.setName(PdfPKCS7.getSubjectFields((X509Certificate)chain[0]).getField("CN"));<br /> if (sap.getReason() != null)<br /> dic.setReason(sap.getReason());<br /> if (sap.getLocation() != null)<br /> dic.setLocation(sap.getLocation());<br /> sap.setCryptoDictionary(dic);<br /> int csize = 4000;<br /> HashMap exc = new HashMap();<br /> exc.put(PdfName.CONTENTS, new Integer(csize * 2 + 2));<br /> sap.preClose(exc);<br /><br /> MessageDigest md = MessageDigest.getInstance("SHA1");<br /> InputStream s = sap.getRangeStream();<br /> int read = 0;<br /> byte[] buff = new byte[8192];<br /> while ((read = s.read(buff, 0, 8192)) > 0) {<br /> md.update(buff, 0, read);<br /> }<br /><br /> CMSSignedDataGenerator generator = new CMSSignedDataGenerator();<br /> generator.addSigner(key, (X509Certificate)chain[0], CMSSignedDataGenerator.DIGEST_SHA1);<br /><br /> ArrayList list = new ArrayList();<br /> for (int i = 0; i < chain.length; i++) {<br /> list.add(chain[i]);<br /> }<br /> CertStore chainStore = CertStore.getInstance("Collection", new<br /> CollectionCertStoreParameters(list), "BC");<br /> generator.addCertificatesAndCRLs(chainStore);<br /> CMSProcessable content = new CMSProcessableByteArray(md.digest());<br /> CMSSignedData signedData = generator.generate(content, true, "BC");<br /> byte[] pk = signedData.getEncoded();<br /><br /> byte[] outc = new byte[csize];<br /><br /> PdfDictionary dic2 = new PdfDictionary();<br /><br /> System.arraycopy(pk, 0, outc, 0, pk.length);<br /><br /> dic2.put(PdfName.CONTENTS, new PdfString(outc).setHexWriting(true));<br /> sap.close(dic2);<br /> }<br /> catch (Exception e) {<br /> e.printStackTrace();<br /> }<br />}<br /><br />public class CMSProcessableRange implements CMSProcessable {<br /> private PdfSignatureAppearance sap;<br /> private byte[] buf = new byte[8192];<br /><br /> public CMSProcessableRange(PdfSignatureAppearance sap) {<br /> this.sap = sap;<br /> }<br /><br /> public void write(OutputStream outputStream) throws IOException, CMSException {<br /> InputStream s = sap.getRangeStream();<br /> ByteArrayOutputStream ss = new ByteArrayOutputStream();<br /> int read = 0;<br /> while ((read = s.read(buf, 0, buf.length)) > 0) {<br /> outputStream.write(buf, 0, read);<br /> }<br /> }<br /><br /> public Object getContent() {<br /> return sap;<br /> }<br />}<br /></pre> <hr /> <h2><a name="signextdicattrib">How to sign with an external signature dictionary (authenticatedAttributes example)</a></h2> <p> <b>An example with an external signature dictionary with authenticatedAttributes</b> </p><p> </p><pre>KeyStore ks = KeyStore.getInstance("pkcs12");<br />ks.load(new FileInputStream("my_private_key.pfx"),<br />"my_password".toCharArray());<br />String alias = (String)ks.aliases().nextElement();<br />PrivateKey key = (PrivateKey)ks.getKey(alias, "password".toCharArray());<br />Certificate[] chain = ks.getCertificateChain(alias);<br /><br />PdfReader reader = new PdfReader("original.pdf");<br />FileOutputStream fout = new FileOutputStream("signed.pdf");<br />PdfStamper stp = PdfStamper.createSignature(reader, fout, '\0');<br />PdfSignatureAppearance sap = stp.getSignatureAppearance();<br />// comment next line to have an invisible signature<br />sap.setVisibleSignature(new Rectangle(100, 100, 200, 200), 1, null);<br />sap.setLayer2Text("This is some custom made text.\n\nDate: some date");<br />Calendar cal = Calendar.getInstance();<br />PdfDictionary dic = new PdfDictionary();<br />dic.put(PdfName.FT, PdfName.SIG);<br />dic.put(PdfName.FILTER, new PdfName("SAFE.PPKSF"));<br />dic.put(PdfName.SUBFILTER, new PdfName("adbe.pkcs7.detached"));<br />dic.put(PdfName.M, new PdfDate(cal));<br />dic.put(PdfName.NAME, new<br />PdfString(PdfPKCS7.getSubjectFields((X509Certificate)chain[0]).getField("CN")));<br />sap.setCryptoDictionary(dic);<br />HashMap exc = new HashMap();<br />exc.put(PdfName.CONTENTS, new Integer(0x2502));<br />sap.preClose(exc);<br /><br />PdfPKCS7 pk7 = new PdfPKCS7(key, chain, null, "SHA1", null, false);<br />MessageDigest messageDigest = MessageDigest.getInstance("SHA1");<br />byte buf[] = new byte[8192];<br />int n;<br />InputStream inp = sap.getRangeStream();<br />while ((n = inp.read(buf)) > 0) {<br /> messageDigest.update(buf, 0, n);<br />}<br />byte hash[] = messageDigest.digest();<br />byte sh[] = pk7.getAuthenticatedAttributeBytes(hash, cal);<br />pk7.update(sh, 0, sh.length);<br />PdfDictionary dic2 = new PdfDictionary();<br />byte sg[] = pk7.getEncodedPKCS7(hash, cal);<br />byte out[] = new byte[0x2500 / 2];<br />System.arraycopy(sg, 0, out, 0, sg.length);<br />dic2.put(PdfName.CONTENTS, new PdfString(out).setHexWriting(true));<br />sap.close(dic2);<br /></pre> <hr /> <h2><a name="howtoverifycs">How to verify with iTextSharp</a></h2> <p> How to verify a signature using the certificates in the Windows certificate store using WSE. </p><p> </p><pre>using wse = Microsoft.Web.Services.Security.X509;<br />using Org.BouncyCastle.X509;<br /><br />wse.X509CertificateStore store = new wse.X509CertificateStore(wse.X509CertificateStore.StoreProvider.System, wse.X509CertificateStore.StoreLocation.LocalMachine, wse.X509CertificateStore.RootStore);<br />store.Open();<br />wse.X509CertificateCollection c = store.Certificates;<br />System.Console.Out.WriteLine("There are " + c.Count + " certificates in the Windows certificate store");<br />X509CertificateParser parser = new X509CertificateParser();<br />ArrayList kall = new ArrayList();<br />foreach (wse.X509Certificate cert in c) {<br /> X509Certificate c2 = parser.ReadCertificate(cert.GetRawCertData());<br /> kall.Add(c2);<br />}<br />PdfReader reader = new PdfReader("signed.pdf");<br />AcroFields af = reader.AcroFields;<br />ArrayList names = af.GetSignatureNames();<br />for (int k = 0; k < names.Count; ++k) {<br /> String name = (String)names[k];<br /> System.Console.Out.WriteLine("Signature name: " + name);<br /> System.Console.Out.WriteLine("Signature covers whole document: " + af.SignatureCoversWholeDocument(name));<br /> System.Console.Out.WriteLine("Document revision: " + af.GetRevision(name) + " of " + af.TotalRevisions);<br /> PdfPKCS7 pk = af.VerifySignature(name);<br /> DateTime cal = pk.SignDate;<br /> X509Certificate[] pkc = pk.Certificates;<br /> System.Console.Out.WriteLine("Subject: " + PdfPKCS7.GetSubjectFields(pk.SigningCertificate).GetField("CN"));<br /> System.Console.Out.WriteLine("Document modified: " + !pk.Verify());<br /> Object[] fails = PdfPKCS7.VerifyCertificates(pkc, kall, null, cal);<br /> if (fails == null)<br /> System.Console.Out.WriteLine("Certificates verified against the KeyStore");<br /> else<br /> System.Console.Out.WriteLine("Certificate failed: " + fails[1]);<br />}<br /></pre> <hr /> <h2><a name="signextitextsharp1">How to sign with a smartcard using an external signature dictionary with iTextSharp, CAPICOM and .NET 1.1</a></h2> <p> <b>This example allows the signing with smartcards. The signing is done using the Windows mode and includes the hash in the signature. This example requires .NET 1.1 and CAPICOM 2.0 as a COM Interop (select the CAPICOM as a reference and Visual Studio will create the C# interfaces and classes). </b> </p><p> </p><pre>using System;<br />using System.IO;<br />using System.Collections;<br />using System.Security.Cryptography;<br />using System.Security.Cryptography.X509Certificates;<br />using CAPICOM;<br />using iTextSharp.text;<br />using iTextSharp.text.pdf;<br /><br />namespace CCapi {<br /> class Class1 {<br /> static void Main(string[] args) {<br /> Certificate cer = GetCertificate();<br /> Org.BouncyCastle.X509.X509Certificate[] chain = GetChain(cer);<br /> PdfReader reader = new PdfReader("c:\\hello.pdf");<br /> PdfStamper stp = PdfStamper.CreateSignature(reader, new FileStream("c:\\hello_hashedcapi.pdf", FileMode.Create), '\0');<br /> PdfSignatureAppearance sap = stp.SignatureAppearance;<br /> sap.SetVisibleSignature(new Rectangle(100, 100, 300, 200), 1, null);<br /> sap.SignDate = DateTime.Now;<br /> sap.SetCrypto(null, chain, null, null);<br /> sap.Reason = "I like to sign";<br /> sap.Location = "Universe";<br /> sap.Acro6Layers = true;<br /> sap.Render = PdfSignatureAppearance.SignatureRender.NameAndDescription;<br /> PdfSignature dic = new PdfSignature(PdfName.ADOBE_PPKMS, PdfName.ADBE_PKCS7_SHA1);<br /> dic.Date = new PdfDate(sap.SignDate);<br /> dic.Name = PdfPKCS7.GetSubjectFields(chain[0]).GetField("CN");<br /> if (sap.Reason != null)<br /> dic.Reason = sap.Reason;<br /> if (sap.Location != null)<br /> dic.Location = sap.Location;<br /> sap.CryptoDictionary = dic;<br /> int csize = 4000;<br /> Hashtable exc = new Hashtable();<br /> exc[PdfName.CONTENTS] = csize * 2 + 2;<br /> sap.PreClose(exc);<br /><br /> HashAlgorithm sha = new SHA1CryptoServiceProvider();<br /><br /> Stream s = sap.RangeStream;<br /> int read = 0;<br /> byte[] buff = new byte[8192];<br /> while ((read = s.Read(buff, 0, 8192)) > 0) {<br /> sha.TransformBlock(buff, 0, read, buff, 0);<br /> }<br /> sha.TransformFinalBlock(buff, 0, 0);<br /> byte[] pk = SigMsg(sha.Hash, cer, false);<br /> byte[] outc = new byte[csize];<br /><br /> PdfDictionary dic2 = new PdfDictionary();<br /><br /> Array.Copy(pk, 0, outc, 0, pk.Length);<br /><br /> dic2.Put(PdfName.CONTENTS, new PdfString(outc).SetHexWriting(true));<br /> sap.Close(dic2);<br /> }<br /><br /> public static byte[] SigMsg(byte[] msg, Certificate cert, bool detached) {<br /> SignerClass signer = new SignerClass();<br /><br /> // check for private key in the certificate<br /> if (cert.HasPrivateKey())<br /> signer.Certificate = cert;<br /> else<br /> throw new ArgumentException("Certificate has no private key");<br /> UtilitiesClass ut = new UtilitiesClass();<br /> SignedDataClass sd = new SignedDataClass();<br /> string gg = ut.ByteArrayToBinaryString(msg);<br /> sd.Content = gg;<br /> string signedData = sd.Sign(signer, detached, CAPICOM_ENCODING_TYPE.CAPICOM_ENCODE_BASE64);<br /> return Convert.FromBase64String(signedData);<br /> }<br /><br /> public static Certificate GetCertificate() {<br /> StoreClass store = new StoreClass();<br /> // can also be CAPICOM_STORE_LOCATION.CAPICOM_CURRENT_USER_STORE<br /> store.Open(CAPICOM_STORE_LOCATION.CAPICOM_SMART_CARD_USER_STORE,<br /> "My", CAPICOM_STORE_OPEN_MODE.CAPICOM_STORE_OPEN_READ_ONLY);<br /> Certificate cc = null;<br /> // choose the certificate. May also choose by serial id, etc.<br /> foreach (Certificate cert in store.Certificates) {<br /> if (cert.SubjectName.IndexOf("Paulo Soares") > 0) {<br /> cc = cert;<br /> break;<br /> }<br /> }<br /> return cc;<br /> }<br /><br /> public static Org.BouncyCastle.X509.X509Certificate[] GetChain(Certificate cer) {<br /> ICertContext context = cer as ICertContext;<br /> IntPtr ptr = new IntPtr(context.CertContext);<br /> X509Certificate xCert = new X509Certificate(ptr);<br /> Org.BouncyCastle.X509.X509CertificateParser cp =<br /> new Org.BouncyCastle.X509.X509CertificateParser();<br /> Org.BouncyCastle.X509.X509Certificate[] chain =<br /> new Org.BouncyCastle.X509.X509Certificate[]{cp.ReadCertificate(xCert.GetRawCertData())};<br /> return chain;<br /> }<br /> }<br />}<br /></pre> <hr /> <h2><a name="signextitextsharp2">How to sign with a smartcard using an external signature dictionary with iTextSharp and .NET 2.0</a></h2> <p> <b>This example allows the signing with smartcards. Two functions are provided: SignHashed() signs using the Windows mode and includes the hash in the signature and SignDetached() signs in detached mode but needs more memory to do it. This example requires .NET 2.0. </b> </p><p> </p><pre>using System;<br />using System.Collections;<br />using System.IO;<br />using System.Security.Cryptography.X509Certificates;<br />using System.Security.Cryptography.Pkcs;<br />using System.Security.Cryptography;<br />using iTextSharp.text;<br />using iTextSharp.text.pdf;<br /><br />public static void SignHashed() {<br /> X509Certificate2 card = GetCertificate();<br /> Org.BouncyCastle.X509.X509CertificateParser cp = new Org.BouncyCastle.X509.X509CertificateParser();<br /> Org.BouncyCastle.X509.X509Certificate[] chain = new Org.BouncyCastle.X509.X509Certificate[]{cp.ReadCertificate(card.RawData)};<br /> <br /> PdfReader reader = new PdfReader("c:\\hello.pdf");<br /> PdfStamper stp = PdfStamper.CreateSignature(reader, new FileStream("c:\\hello_hashed.pdf", FileMode.Create), '\0');<br /> PdfSignatureAppearance sap = stp.SignatureAppearance;<br /> sap.SetVisibleSignature(new Rectangle(100, 100, 300, 200), 1, null);<br /> sap.SignDate = DateTime.Now;<br /> sap.SetCrypto(null, chain, null, null);<br /> sap.Reason = "I like to sign";<br /> sap.Location = "Universe";<br /> sap.Acro6Layers = true;<br /> sap.Render = PdfSignatureAppearance.SignatureRender.NameAndDescription;<br /> PdfSignature dic = new PdfSignature(PdfName.ADOBE_PPKMS, PdfName.ADBE_PKCS7_SHA1);<br /> dic.Date = new PdfDate(sap.SignDate);<br /> dic.Name = PdfPKCS7.GetSubjectFields(chain[0]).GetField("CN");<br /> if (sap.Reason != null)<br /> dic.Reason = sap.Reason;<br /> if (sap.Location != null)<br /> dic.Location = sap.Location;<br /> sap.CryptoDictionary = dic;<br /> int csize = 4000;<br /> Hashtable exc = new Hashtable();<br /> exc[PdfName.CONTENTS] = csize * 2 + 2;<br /> sap.PreClose(exc);<br /><br /> HashAlgorithm sha = new SHA1CryptoServiceProvider();<br /> <br /> Stream s = sap.RangeStream;<br /> int read = 0;<br /> byte[] buff = new byte[8192];<br /> while ((read = s.Read(buff, 0, 8192)) > 0) {<br /> sha.TransformBlock(buff, 0, read, buff, 0);<br /> }<br /> sha.TransformFinalBlock(buff, 0, 0);<br /> byte[] pk = SignMsg(sha.Hash, card, false);<br /><br /> byte[] outc = new byte[csize];<br /><br /> PdfDictionary dic2 = new PdfDictionary();<br /><br /> Array.Copy(pk, 0, outc, 0, pk.Length);<br /><br /> dic2.Put(PdfName.CONTENTS, new PdfString(outc).SetHexWriting(true));<br /> sap.Close(dic2);<br />}<br /><br />public static void SignDetached() {<br /> X509Certificate2 card = GetCertificate();<br /> Org.BouncyCastle.X509.X509CertificateParser cp = new Org.BouncyCastle.X509.X509CertificateParser();<br /> Org.BouncyCastle.X509.X509Certificate[] chain = new Org.BouncyCastle.X509.X509Certificate[]{cp.ReadCertificate(card.RawData)};<br /> <br /> PdfReader reader = new PdfReader("c:\\hello.pdf");<br /> PdfStamper stp = PdfStamper.CreateSignature(reader, new FileStream("c:\\hello_detached.pdf", FileMode.Create), '\0');<br /> PdfSignatureAppearance sap = stp.SignatureAppearance;<br /> sap.SetVisibleSignature(new Rectangle(100, 100, 300, 200), 1, null);<br /> sap.SignDate = DateTime.Now;<br /> sap.SetCrypto(null, chain, null, null);<br /> sap.Reason = "I like to sign";<br /> sap.Location = "Universe";<br /> sap.Acro6Layers = true;<br /> sap.Render = PdfSignatureAppearance.SignatureRender.NameAndDescription;<br /> PdfSignature dic = new PdfSignature(PdfName.ADOBE_PPKLITE, PdfName.ADBE_PKCS7_DETACHED);<br /> dic.Date = new PdfDate(sap.SignDate);<br /> dic.Name = PdfPKCS7.GetSubjectFields(chain[0]).GetField("CN");<br /> if (sap.Reason != null)<br /> dic.Reason = sap.Reason;<br /> if (sap.Location != null)<br /> dic.Location = sap.Location;<br /> sap.CryptoDictionary = dic;<br /> int csize = 4000;<br /> Hashtable exc = new Hashtable();<br /> exc[PdfName.CONTENTS] = csize * 2 + 2;<br /> sap.PreClose(exc);<br /><br /> Stream s = sap.RangeStream;<br /> MemoryStream ss = new MemoryStream();<br /> int read = 0;<br /> byte[] buff = new byte[8192];<br /> while ((read = s.Read(buff, 0, 8192)) > 0) {<br /> ss.Write(buff, 0, read);<br /> }<br /> byte[] pk = SignMsg(ss.ToArray(), card, true);<br /><br /> byte[] outc = new byte[csize];<br /><br /> PdfDictionary dic2 = new PdfDictionary();<br /><br /> Array.Copy(pk, 0, outc, 0, pk.Length);<br /><br /> dic2.Put(PdfName.CONTENTS, new PdfString(outc).SetHexWriting(true));<br /> sap.Close(dic2);<br />}<br /><br />// Sign the message with the private key of the signer.<br />static public byte[] SignMsg(Byte[] msg, X509Certificate2 signerCert, bool detached) {<br /> // Place message in a ContentInfo object.<br /> // This is required to build a SignedCms object.<br /> ContentInfo contentInfo = new ContentInfo(msg);<br /><br /> // Instantiate SignedCms object with the ContentInfo above.<br /> // Has default SubjectIdentifierType IssuerAndSerialNumber.<br /> SignedCms signedCms = new SignedCms(contentInfo, detached);<br /><br /> // Formulate a CmsSigner object for the signer.<br /> CmsSigner cmsSigner = new CmsSigner(signerCert);<br /><br /> // Include the following line if the top certificate in the<br /> // smartcard is not in the trusted list.<br /> cmsSigner.IncludeOption = X509IncludeOption.EndCertOnly;<br /><br /> // Sign the CMS/PKCS #7 message. The second argument is<br /> // needed to ask for the pin.<br /> signedCms.ComputeSignature(cmsSigner, false);<br /><br /> // Encode the CMS/PKCS #7 message.<br /> return signedCms.Encode();<br />}<br /><br />public static X509Certificate2 GetCertificate() {<br /> X509Store st = new X509Store(StoreName.My, StoreLocation.CurrentUser);<br /> st.Open(OpenFlags.ReadOnly);<br /> X509Certificate2Collection col = st.Certificates;<br /> X509Certificate2 card = null;<br /> X509Certificate2Collection sel = X509Certificate2UI.SelectFromCollection(col, "Certificates", "Select one to sign", X509SelectionFlag.SingleSelection);<br /> if (sel.Count > 0) {<br /> X509Certificate2Enumerator en = sel.GetEnumerator();<br /> en.MoveNext();<br /> card = en.Current;<br /> }<br /> st.Close();<br /> return card;<br />}<br /></pre>Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-6930918428485106265.post-13395271807678252482008-05-23T14:06:00.001-07:002008-06-19T15:05:57.915-07:00Restarting a remote computerIf you want to restart a computer which you have connected via Windows Remote Desktop do this.<br /><br /><span style="color: rgb(255, 0, 0);">Start </span>>> <span style="color: rgb(255, 0, 0);">Run</span> >> <span style="color: rgb(255, 0, 0);">shutdown -r -f </span><br /><br />The machine would be restarted after 30 seconds automatically<br /><br />There are lots of other options available with shutdown command. You can view all the available options by typing "<span style="color: rgb(255, 0, 0);">shutdown /?</span>" in the command window.Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6930918428485106265.post-14506595987480850782008-04-19T07:34:00.000-07:002008-04-19T07:35:16.699-07:00The New Builder Pattern<h3 class="post-title entry-title"> <a href="http://xavimiro.blogspot.com/2008/04/new-builder-pattern.html">The New Builder Pattern</a> </h3> <h2>The idea</h2><div style="text-align: left;">I like to create immutable objects, especially after reading Josh Bloch's excellent "Effective Java" book. If an object is immutable, it has only one possible state and it is a stable one, so once you successfully build an object, you don't need to care about state transitions that can make your object unstable or corrupted. And immutable objects can be shared even in a multithreaded application. There are many other pros of immutability (you can read some of them <a href="http://www.javapractices.com/topic/TopicAction.do?Id=29">here</a>).<br /><br />There is a classical way of making immutable objects in Java which consists of making all fields final (and private, of course), using only constructors to modify them (so that the only moment when a field is modified is during its construction) and making the class final (to avoid adding "setter" methods to subclasses). When you only have a couple of fields, that's fine, but when you have many of them you end up with a constructor with many arguments, which is ugly and difficult to use. If you have optional parameters, you can have a constructor with all the parameters and some other shorter constructors that have the mandatory parameters and some optional ones, that invoke the big constructor, like this:<br /></div><br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code>public class Foo {<br /><br />private final String mandatoryOne;<br />private final String mandatoryTwo;<br />private final String optionalOne;<br />private final String optionalTwo;<br /><br />public Foo(String mOne, String mTwo, String optOne, String optTwo){<br />this.mandatoryOne = mOne;<br />this.mandatoryTwo = mTwo;<br />this.optionalOne = optOne;<br />this.optionalTwo = optTwo;<br />}<br /><br />public Foo(String mOne, String mTwo, String optOne){<br />this(mOne, mTwo, optOne, null);<br />}<br />...<br />}<br /></code></pre><br /><br />This can be a bit messy when you add more optional parameters, you end up with a lot of constructors like these and it has a lot of boilerplate code.The use of setters for the optional parameters is not an option, because this leads to non immutable objects (some object can change the state of your object with one of those setter methods).<br />Some time ago, thinking about this problem, I thought a solution could be to use a Javabean object, with one setter per field (even for the mandatory ones), but with a kind of "seal" method, that would "mark" the object as built and since that moment, an IllegalStateException would be thrown if a setter was called. Nevertheless, I wasn't very satisfied with this approach, because the setter methods that sometimes can be called and sometimes not would be confusing for the caller.<br /><br /><div style="text-align: left;">Then I found the New Builder pattern, explained by Josh Bloch in <a href="http://developers.sun.com/learning/javaoneonline/2007/pdf/TS-2689.pdf">this PDF presentation</a>, which is different from the original <a href="http://en.wikipedia.org/wiki/Builder_pattern">GoF Builder pattern</a>. This pattern uses a public inner static class as a builder. The constructor of the original class is made private, so the only way to build objects is with the inner builder class. The builder has a setter method for each optional parameter and uses a fluent idiom that allows chaining of these method calls. I like this pattern a lot, because it solves the problem elegantly and effectively.<br /></div><br /><h2>The implementation</h2><div style="text-align: left;">In Josh Bloch's presentation there wasn't a detailed implementation of the pattern, although it was very clear the idea and the intention so I have searched for it in the Internet.<br /><br />In <a href="http://rwhansen.blogspot.com/2007/07/theres-builder-pattern-that-joshua.html">Richard Hansen's blog </a>you can find an implementation that seems to be more close to what Josh explains: the builder is a static nested class of the class from which it has to make instances, the builder's constructor is public (so you invoke the builder with 'new'), and the builder has the same fields as its enclosing class. The 'build()' method copies the content of the builder's fields into a new instance of the enclosing class. What I don't like about this implementation is this duplication of fields (for each field in the original class you have a duplicate field in the builder).<br /></div><p style="text-align: left;">In <a href="http://garbagecollected.org/2007/07/12/builder-pattern-deluxe/">Robbie Vanbrabant's blog </a>there is a variation of this pattern, which avoids the boilerplate code using a base class for the builder and some reflection to build the object from the builder. I don't like the use of an interface for the builder, because that way you can't add a new optional parameter without breaking existing code that uses the builder (if you change the signature of a public interface the classes that use it have to change their code to implement the new methods). Also, I don't like the use of reflection because it's slower than the normal access to fields, but I do like the way it avoids duplication of fields in the builder.<br /></p><div style="text-align: left;">The implementation I like most is the one found in <a href="http://www.softonaut.com/2007/01/10/thoughts-about-builder-pattern/">Mario Hochreiter's blog</a>. The builder is a nested public static class, but it changes the fields of its enclosing class directly, it doesn't use duplicates. It doesn't use reflection and the builder is a class, not an interface. The only problem I see is that, in theory, with a reference to a builder, you can change the state of the object it built, so you don't have the guarantee that the object is immutable. So I would add a check before each "setter" of the builder that would throw an IllegalStateException if the object has been already built and a check before the 'build' method itself to ensure the object is not built more than once. Also, I would make the mandatory parameters final.<br /></div><p style="text-align: left;">So, with the example of Mario, I would implement this pattern this way:<br /></p><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code>public class ID3Tag {<br /><br />private final String title;<br />private final String artist;<br />private String album;<br />private int albumTrack;<br />private String comment;<br /><br />public static class Builder {<br /><br /> private boolean isBuilt = false;<br /> private ID3Tag id3tag;<br /><br /> public Builder(String title, String artist) {<br /> id3tag = new ID3Tag(title, artist);<br /> }<br /><br /> public Builder album(String val) {<br /> if (isBuilt){<br /> throw new IllegalStateException("The object cannot be modified after built");<br /> }<br /> id3tag.album = val;<br /> return this;<br /> }<br /><br /> public Builder albumTrack(int val) {<br /> if (isBuilt){<br /> throw new IllegalStateException("The object cannot be modified after built");<br /> }<br /> id3tag.albumTrack = val;<br /> return this;<br /> }<br /><br /> public Builder comment(String val) {<br /> if (isBuilt){<br /> throw new IllegalStateException("The object cannot be modified after built");<br /> }<br /> id3tag.comment = val;<br /> return this;<br /> }<br /> // ... a lot more optional parameters<br /><br /> public ID3Tag build() {<br /> if (isBuilt){<br /> throw new IllegalStateException("The object cannot be built twice");<br /> }<br /> isBuilt = true;<br /> return id3tag;<br /> }<br />}<br /><br />private ID3Tag(String title, String artist) {<br /> this.title = title;<br /> this.artist = artist;<br />}<br />}<br /><br /></code></pre><br />The usage of this class would be:<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code>ID3Tag tag = new ID3Tag.Builder("My Title", "My author")<br /> .comment("Great song").build();<br /><br /></code></pre><br /><br /><p style="text-align: left;">I have found a similar pattern, called the Essence pattern, described <a href="http://jerry.cs.uiuc.edu/%7Eplop/plop98/final_submissions/P10.pdf">here </a>and <a href="http://www.codeproject.com/KB/architecture/essencepattern.aspx">here</a> by Dr Herbie. This pattern uses direct access to the fields of the builder (like in a C++ structure) instead of using "setter" methods and it doesn't use "chaining" of modifications like in the New Builder Pattern ("...builder.option1(value1).option2(value2)...").</p>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6930918428485106265.post-77494946690578724832008-04-19T06:57:00.000-07:002008-04-19T06:59:34.290-07:00Initialization on demand holder idiom<h1 class="firstHeading"><a href="http://en.wikipedia.org/wiki/Initialization_on_demand_holder_idiom">Initialization on demand holder idiom</a></h1><!-- start content --> <table id="toc" class="toc" summary="Contents"> <tbody><tr> <td> </td> </tr> </tbody></table> <script type="text/javascript"> //<![CDATA[ if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); } //]]> </script> <p>In <a href="http://en.wikipedia.org/wiki/Software_engineering" title="Software engineering">software engineering</a>, the <b>Initialization on Demand Holder</b> idiom (<a href="http://en.wikipedia.org/wiki/Design_pattern_%28computer_science%29" title="Design pattern (computer science)">design pattern</a>) is a <a href="http://en.wikipedia.org/wiki/Lazy_initialization" title="Lazy initialization">lazy-loaded</a> <a href="http://en.wikipedia.org/wiki/Singleton_pattern" title="Singleton pattern">singleton</a>. The idiom can be implemented in both single-threaded/serial and concurrent environments, but care must be taken to correctly implement the idiom under concurrent conditions.<br /></p><p><span class="mw-headline">Example Java Implementation</span></p> <p>This implementation from Bill Pugh (see links below) is a well-performing and concurrent implementation valid in all versions of Java. The original implementation from Bill Pugh has been modified to reduce the scope of <b>LazyHolder.something</b> to package and to make the field final.</p> <div dir="ltr" style="text-align: left;"> <pre class="source-java"><span class="kw2">public</span> <span class="kw2">class</span> Something <span class="br0">{</span><br /> <span class="kw2">private</span> Something<span class="br0">(</span><span class="br0">)</span> <span class="br0">{</span><br /> <span class="br0">}</span><br /><br /> <span class="kw2">private</span> <span class="kw2">static</span> <span class="kw2">class</span> LazyHolder <span class="br0">{</span><br /> <span class="kw2">private</span> <span class="kw2">static</span> <span class="kw2">final</span> Something something = <span class="kw2">new</span> Something<span class="br0">(</span><span class="br0">)</span>;<br /> <span class="br0">}</span><br /><br /> <span class="kw2">public</span> <span class="kw2">static</span> Something getInstance<span class="br0">(</span><span class="br0">)</span> <span class="br0">{</span><br /> <span class="kw2">return</span> LazyHolder.<span class="me1">something</span>;<br /> <span class="br0">}</span><br /><span class="br0">}</span> </pre></div> <p><a name="How_it_works" id="How_it_works"></a></p> <h3><span class="editsection"></span><span class="mw-headline">How it works</span></h3> <p>The implementation relies on the well-specified initialization phase of execution within the Java Virtual Machine (JVM); see section <a href="http://java.sun.com/docs/books/jls/third_edition/html/execution.html#44557" class="external text" title="http://java.sun.com/docs/books/jls/third_edition/html/execution.html#44557" rel="nofollow">12.4 of Java Language Specification</a> (JLS) for details.</p> <p>When the class <b>Something</b> is loaded by the JVM, the class goes through initialization. Since the class does not have any static variables to initialize, the initialization completes trivially. The static class definition <b>LazyHolder</b> within it is <b>not</b> initialized until the JVM determines that <b>LazyHolder</b> must be executed. The static class <b>LazyHolder</b> is only executed when the static method <b>getInstance</b> is invoked on the class <b>Something</b>, and the first time this happens the JVM will load and initialize the <b>LazyHolder</b> class. The initialization of the <b>LazyHolder</b> class results in static variable <b>something</b> being initialized by executing the (private) constructor for the outer class <b>Something</b>. Since the class initialization phase is guaranteed by the JLS to be serial, i.e., non-concurrent, no further synchronization is required in the static <b>getInstance</b> method during loading and initialization. And since the initialization phase writes the static variable <b>something</b> in a serial operation, all subsequent concurrent invocations of the <b>getInstance</b> will return the same correctly initialized <b>something</b> without incurring any additional synchronization overhead.</p> <p><a name="When_to_use_it" id="When_to_use_it"></a></p> <h3><span class="editsection"></span> <span class="mw-headline">When to use it</span></h3> <p>Use this pattern if the initialization of the class is <i>expensive</i> and it cannot be done safely at <i>class-loading</i> time and the initialization is highly concurrent. The crux of the pattern is the safe removal of the synchronization overhead associated with accessing a singleton instance.</p>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6930918428485106265.post-89269861413702533002008-04-19T06:46:00.000-07:002008-04-19T06:47:28.436-07:00A Java Builder Pattern<h3 class="post-title entry-title"> <a href="http://rwhansen.blogspot.com/2007/07/theres-builder-pattern-that-joshua.html">A Java Builder Pattern</a> </h3> <p>There's a Builder pattern that Joshua Bloch has briefly described in a couple of his <a href="http://developers.sun.com/learning/javaoneonline/j1sessn.jsp?sessn=TS-2689&yr=2007&track=5">"Effective Java Reloaded"</a> sessions at Java One. This Builder is not necessarily a replacement for the <a href="http://en.wikipedia.org/wiki/Builder_pattern">original design pattern</a>. The problems this Builder pattern can solve are too many constructors, too many constructor parameters, and over use of setters to create an object.</p><br /><p>Here are some examples of the pattern in use. These examples create various Widgets with two required properties and several optional ones -</p><br /><pre><br /><span style=""><code><br />Widget x = new Widget.Builder("1", 1.0).<br /> model("1").build();<br />Widget y = new Widget.Builder("2", 2.0).<br /> model("2").manufacturer("222").<br /> serialNumber("12345").build();<br />Widget z = new Widget.Builder("3", 4.0).<br /> manufacturer("333").<br /> serialNumber("54321").build();<br /></code></span><br /></pre><br /><p>The basic idea behind the pattern is to limit the number of constructor parameters and avoid the use of setter methods. Constructors with too many parameters, especially optional ones, are ugly and hard to use. Multiple constructors for different modes are confusing. Setter methods add clutter and force an object to be mutable. Here is an class skeleton of the pattern -</p><br /><pre><br /><span style=""><code>public class Widget {<br />public static class Builder {<br /> public Builder(String name, double price) { ... }<br /> public Widget build() { ... }<br /> public Builder manufacturer(String value) { ... }<br /> public Builder serialNumber(String value) { ... }<br /> public Builder model(String value) { ... }<br />}<br /><br />private Widget(Builder builder) { ... }<br />}<br /></code></span><br /></pre><br /><p>Notice that Widget has no public constructor and no setters and that the only way to create a Widget is using the static inner class Widget.Builder. Widget.Builder has a constructor that takes the required properties of Widget. Widget's optional properties can be set using optional property methods on the Widget.Builder. The property methods of Widget.Builder return a reference to the builder so method calls can be chained.</p><br /><p>A really nice feature of this pattern is the ability to do pre-creation validation of an object state. When setters are used to set object state during creation it is virtually impossible to guarantee that object has been properly created.</p><br /><p>Here is the full source for Widget and its Builder -</p><br /><pre><br /><span style=""><code>public class Widget {<br /> public static class Builder {<br /> private String name;<br /> private String model;<br /> private String serialNumber;<br /> private double price;<br /> private String manufacturer;<br /><br /> public Builder(String name, double price) {<br /> this.name = name;<br /> this.price = price;<br /> }<br /><br /> public Widget build() {<br /> // any pre-creation validation here<br /> Widget result = new Widget(name, price);<br /> result.model = model;<br /> result.serialNumber = serialNumber;<br /> result.manufacturer = manufacturer;<br /> return result;<br /> }<br /><br /> public Builder manufacturer(String value) {<br /> this.manufacturer = value;<br /> return this;<br /> }<br /><br /> public Builder serialNumber(String value) {<br /> this.serialNumber = value;<br /> return this;<br /> }<br /><br /> public Builder model(String value) {<br /> this.model = value;<br /> return this;<br /> }<br /> }<br /><br />private String name;<br />private String model;<br />private String serialNumber;<br />private double price;<br />private String manufacturer;<br /><br />/**<br /> * Creates an immutable widget instance.<br /> */<br />private Widget(String name, double price) {<br /> this.name = name;<br /> this.price = price;<br />}<br /><br />public String toString() {<br /> return super.toString() + " {"<br /> + "name="<br /> + getName()<br /> + " model="<br /> + getModel()<br /> + " serialNumber="<br /> + getSerialNumber()<br /> + " price="<br /> + getPrice()<br /> + " manufacturer="<br /> + getManufacturer()<br /> + "}";<br />}<br /><br />public String getManufacturer() {<br /> return manufacturer;<br />}<br /><br />public String getModel() {<br /> return model;<br />}<br /><br />public String getName() {<br /> return name;<br />}<br /><br />public double getPrice() {<br /> return price;<br />}<br /><br />public String getSerialNumber() {<br /> return serialNumber;<br />}<br />}<br /></code></span><br /></pre><br /><p>Notice that Widget's private constructor takes the required properties and that the Builder sets the optional properties. Another thing to note is that widget is an immutable object as implemented.</p>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6930918428485106265.post-21251046139562710022008-04-18T08:14:00.000-07:002008-04-18T08:15:12.194-07:00Memory Monitoring with Java SE 5<h2 xmlns=""><a href="http://www.informit.com/guides/content.aspx?g=java&seqNum=249&rll=1">Memory Monitoring with Java SE 5</a></h2> Last updated Mar 10, 2006. <p>One of the benefits that Java introduced over C++ was the concept of automatic memory management. Specifically, Java Virtual Machines (JVM) provide a garbage collection thread that frees memory when the JVM's heap exhausts available memory. The implementation of the garbage collection strategy is dependent on the JVM vendor, but it suffices to say that the garbage collection process can be expensive. The Sun JVM defines garbage collection in two modes:</p> <ul><li>Minor "copy" collections</li><li>Major "Mark-Sweep-Compact" collections</li></ul> <p>A minor collection runs relatively quickly and involves moving live data around the heap in the presence of running threads. A major collection is a much more intrusive garbage collection that suspends all execution threads while it completes its task. In terms of performance tuning the heap, the primary goal is to reduce the frequency and duration of major garbage collections.</p> <p>The IBM JVM implements a different architecture and graduates its garbage collection behavior based off of the state of the heap. Garbage collections can range from minimal impacting sweeps (fueled by a mark phase executed concurrently to executing threads) to a worst case scenario of a full mark-sweep-compact, <em>stop-the-world</em> garbage collection.</p> <p>Because Java applications and their respective components all run inside the JVM and rely on objects instances loaded into the heap, tuning the heap is of paramount importance. But before the heap can be properly tuned, you need insight into the performance of the heap.</p> <p>Until the release of Java SE 5, monitoring options included:</p> <ul><li>Heap size and usage information obtained through calls to the <tt>java.lang.Runtime</tt> class</li><li>Verbose garbage collection log parsing, enabled by passing startup parameters to the JVM to instruct it to generate verbose logging information every time a garbage collection occurs</li><li>Proprietary JVM APIs, such as Sun's <a onclick="newwindow(this)" href="http://java.sun.com/performance/jvmstat/index.html">jvmstat</a> interface</li><li>Dangerous proprietary strategies such as injecting a DLL or shared object into the JVM process space to read process memory</li></ul> <p>Each of these strategies has advantages and disadvantages in terms of overhead and ease of implementation, but the result is that there is no single perfect strategy. </p> <p>One of the benefits of performance monitoring application servers is that all major vendors have adopted the Java Management Extensions (JMX) and exposed monitoring information in the form of managed beans (MBeans.) With the release of Java SE 5, a JMX registry and support for MBeans was added to the JVM itself, so now that same luxury is available to stand-alone applications as well as enterprise applications. In addition, the Java SE 5 specification has defined new classes and interfaces in the <tt>java.lang.management</tt> package that expose JVM runtime statistics. A handful of those new interfaces are implemented by managed beans that expose runtime information about the behavior of the JVM memory, both heap and non-heap process memory. In this article we look at three bean interfaces:</p> <ul><li>MemoryMXBean</li><li>MemoryPoolMXBean</li><li>GarbageCollectionMXBean</li></ul> <p>All management MBeans can be loaded through a helper class: <tt>java.lang.management.ManagementFactory</tt>. The <tt>ManagementFactory</tt> provides a set of static methods that return the requests MBeans:</p> <pre>MemoryMXBean memorymbean = ManagementFactory.getMemoryMXBean();<br />List<memorypoolmxbean> mempoolsmbeans = ManagementFactory.getMemoryPoolMXBeans();<br />List<garbagecollectormxbean> gcmbeans = ManagementFactory.getGarbageCollectorMXBeans();</garbagecollectormxbean></memorypoolmxbean></pre> <p>The <tt>MemoryMXBean</tt> provides information about memory usage, including both heap and non-heap memory. Specifically it provides the following two methods:</p> <pre>MemoryUsage getHeapMemoryUsage()<br />MemoryUsage getNonHeapMemoryUsage()</pre> <p>These methods return an instance of <tt>java.lang.management.MemoryUsage</tt> that defined four key attributes:</p> <ul><li>init: the initial amount of memory that the JVM requested from the operating system during startup</li><li>used: the amount of memory currently in use</li><li>committed: the amount of memory that is guaranteed to be available for use by the JVM; it can change over time and is always guaranteed to be greater than or equal to the used memory</li><li>max: the maximum amount of memory that can be used by the JVM in the specified area (heap or non-heap)</li></ul> <p>The <tt>MemoryPoolMXBean</tt> provides information about specific memory pools within the JVM memory spaces, both heap and non-heap. For those familiar with the Sun heap, these pools include the Eden Space, Survivor Space, Tenured Generation, and Permanent Generation.</p> <p>In addition to others that are presented later. But the point is that these are the logical partitions that memory is subdivided into. For each <tt>MemoryPoolMXBean</tt>, you can discover the following information: </p> <ul><li>Current Usage</li><li>Peak Usage</li><li>Usage at the last collection</li><li>The type of memory space (heap or non-heap)</li><li>The memory managers that operate on this space, for example "Copy" and "MarkSweepCompact"</li></ul> <p>Finally, the <tt>GarbageCollectorMXBean</tt> provides collection times and collection counts for each type of memory pool.</p> <p>Listing 1 uses each of the aforementioned MBeans in conjunction with some memory mismanagement to display information about the JVM memory.</p> <h4>Listing 1. Java5ManagementTest.java</h4> <pre>package com.javasrc.management;<br /><br />import java.lang.management.*;<br />import java.util.*;<br /><br />public class Java5ManagementTest<br />{<br />public static void dumpMemoryInfo()<br />{<br /> try<br /> {<br /> System.out.println( "\nDUMPING MEMORY INFO\n" );<br /> // Read MemoryMXBean<br /> MemoryMXBean memorymbean = ManagementFactory.getMemoryMXBean();<br /> System.out.println( "Heap Memory Usage: " + memorymbean.getHeapMemoryUsage() );<br /> System.out.println( "Non-Heap Memory Usage: " + memorymbean.getNonHeapMemoryUsage() );<br /><br /> // Read Garbage Collection information<br /> List<garbagecollectormxbean> gcmbeans = ManagementFactory.getGarbageCollectorMXBeans();<br /> for( GarbageCollectorMXBean gcmbean : gcmbeans )<br /> {<br /> System.out.println( "\nName: " + gcmbean.getName() );<br /> System.out.println( "Collection count: " + gcmbean.getCollectionCount() );<br /> System.out.println( "Collection time: " + gcmbean.getCollectionTime() );<br /> System.out.println( "Memory Pools: " );<br /> String[] memoryPoolNames = gcmbean.getMemoryPoolNames();<br /> for( int i=0; i<memorypoolnames.length; t="" i="" read="" pool="" information="" memory="" pools="" info="" list=""><memorypoolmxbean> mempoolsmbeans = ManagementFactory.getMemoryPoolMXBeans();<br /> for( MemoryPoolMXBean mempoolmbean : mempoolsmbeans )<br /> {<br /> System.out.println( "\nName: " + mempoolmbean.getName() );<br /> System.out.println( "Usage: " + mempoolmbean.getUsage() );<br /> System.out.println( "Collection Usage: " + mempoolmbean.getCollectionUsage() );<br /> System.out.println( "Peak Usage: " + mempoolmbean.getPeakUsage() );<br /> System.out.println( "Type: " + mempoolmbean.getType() );<br /> System.out.println( "Memory Manager Names: " ) ;<br /> String[] memManagerNames = mempoolmbean.getMemoryManagerNames();<br /> for( int i=0; i<memmanagernames.length; t="" n="" exception="" e="" public="" static="" void="" args="" tweak="" memory="" a="" little="" bit="" int="" i=""><1000000; s = "My String "></memorypoolmxbean></memorypoolnames.length;></garbagecollectormxbean></pre> <p>Listing 1 presents a class that displays JVM memory management information before and after allocating one million strings. It begins by retrieving and displaying the memory usage information through the <tt>MemoryMXBean</tt>. In my execution, the following are the states before and after the run:</p> <pre>// Before<br />Heap Memory Usage: init = 33554432(32768K)<br /> used = 241680(236K)<br /> committed = 33357824(32576K)<br /> max = 33357824(32576K)<br />Non-Heap Memory Usage: init = 29556736(28864K)<br /> used = 12055504(11772K)<br /> committed = 29851648(29152K)<br /> max = 121634816(118784K)<br /><br />// After<br />Heap Memory Usage: init = 33554432(32768K)<br /> used = 218656(213K)<br /> committed = 33357824(32576K)<br /> max = 33357824(32576K)<br />Non-Heap Memory Usage: init = 29556736(28864K)<br /> used = 12131600(11847K)<br /> committed = 29884416(29184K)<br /> max = 121634816(118784K)</pre> <p>In this case, the minimum and maximum values for the heap are set to 32MB, which is because I used the following parameters on startup:</p> <pre>-Xms32m –Xmx32m</pre> <p>The used memory dropped from 236K to 213K after the run, which we will discover is the result of several garbage collections.</p> <p>The garbage collection information is retrieved from the <tt>GarbageCollectorMXBean</tt>. The Sun JVM with its default configuration implements two garbage collectors: <tt>Copy</tt> and <tt>MarkSweepCompact</tt>. In my sample execution, the following are the states of garbage collection before and after the run:</p> <pre>// Before<br />Name: Copy<br />Collection count: 0<br />Collection time: 0<br />Memory Pools:<br /> Eden Space<br /> Survivor Space<br /><br />Name: MarkSweepCompact<br />Collection count: 0<br />Collection time: 0<br />Memory Pools:<br /> Eden Space<br /> Survivor Space<br /> Tenured Gen<br /> Perm Gen<br /> Perm Gen [shared-ro]<br /> Perm Gen [shared-rw]<br /><br />// After<br />Name: Copy<br />Collection count: 63<br />Collection time: 12<br />Memory Pools:<br /> Eden Space<br /> Survivor Space<br /><br />Name: MarkSweepCompact<br />Collection count: 0<br />Collection time: 0<br />Memory Pools:<br /> Eden Space<br /> Survivor Space<br /> Tenured Gen<br /> Perm Gen<br /> Perm Gen [shared-ro]<br /> Perm Gen [shared-rw]</pre> <p>From this output you can see that creating and discarding one million Strings resulted in 63 copy collections that accounted for 12 milliseconds to run. If we change the code to not discard the Strings between iterations then we will see MarkSweepCompact collections occur.</p> <p>Finally we display information about the various memory pools by accessing the <tt>MemoryPoolMXBeans</tt>. This returns several memory pools in the Sun JVM:</p> <ul><li>Code Cache: contains memory used for compilation and storage of native code</li><li>Eden Space: pool from which memory is initially allocated for most objects</li><li>Survivor Space: pool containing objects that have survived Eden space garbage collection</li><li>Tenured Gen: pool containing long-lived objects</li><li>Perm Gen: contains reflective data of the JVM itself, including class and memory objects</li><li>Perm Gen [shared-ro]: read-only reflective data</li><li>Perm Gen [shared-rw]: read-write reflective data</li></ul> <p>The following displays sample output for the four primary memory pools (Eden, Survivor Space, Tenured Generation, and Permanent Generation) after the test has completed:</p> <pre>Name: Eden Space<br />Usage: init = 2162688(2112K) used = 90784(88K) committed = 2162688(2112K) max = 2162688(2112K)<br />Collection Usage: init = 2162688(2112K) used = 0(0K) committed = 2162688(2112K) max = 2162688(2112K)<br />Peak Usage: init = 2162688(2112K) used = 2162688(2112K) committed = 2162688(2112K) max = 2162688(2112K)<br />Type: Heap memory<br />Memory Manager Names:<br /> MarkSweepCompact<br /> Copy<br /><br />Name: Survivor Space<br />Usage: init = 196608(192K) used = 16(0K) committed = 196608(192K) max = 196608(192K)<br />Collection Usage: init = 196608(192K) used = 16(0K) committed = 196608(192K) max = 196608(192K)<br />Peak Usage: init = 196608(192K) used = 127928(124K) committed = 196608(192K) max = 196608(192K)<br />Type: Heap memory<br />Memory Manager Names:<br /> MarkSweepCompact<br /> Copy<br /><br />Name: Tenured Gen<br />Usage: init = 30998528(30272K) used = 127856(124K) committed = 30998528(30272K) max = 30998528(30272K)<br />Collection Usage: init = 30998528(30272K) used = 0(0K) committed = 0(0K) max = 30998528(30272K)<br />Peak Usage: init = 30998528(30272K) used = 127856(124K) committed = 30998528(30272K) max = 30998528(30272K)<br />Type: Heap memory<br />Memory Manager Names:<br /> MarkSweepCompact<br /><br />Name: Perm Gen<br />Usage: init = 8388608(8192K) used = 127800(124K) committed = 8388608(8192K) max = 67108864(65536K)<br />Collection Usage: init = 8388608(8192K) used = 0(0K) committed = 0(0K) max = 67108864(65536K)<br />Peak Usage: init = 8388608(8192K) used = 127800(124K) committed = 8388608(8192K) max = 67108864(65536K)<br />Type: Non-heap memory<br />Memory Manager Names:<br /> MarkSweepCompact</pre> <p>From this output you can surmise that for a 32MB heap running on Windows with default configuration that Eden was allocated 2112KB, each of the two survivor spaces received 192KB, and the tenured generation received the remaining 30272KB, all adding up to the 32768KB heap. An interesting observation is that the permanent space is allocated an initial 8MB and can grow up to 64MB. By polling this information you can analyze the behavior of the entire heap. For those of you with a passion for this type of work, the MBeans also define a notification interface that you can find in the <a onclick="newwindow(this)" href="http://java.sun.com/j2se/1.5.0/docs/api/">Java SE 5 Javadocs</a>.</p> <p>Finally, if you would like to view this information at runtime, the Sun JVM provides the Java Monitoring and Management Console (JConsole) that can connect to a running JVM and present this information. You can read more about it in a Sun article by Mandy Chung: <a onclick="newwindow(this)" href="http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html">Using JConsole to Monitor Applications</a>.</p> <h3>Summary</h3> <p>Memory monitoring in Java 1.4.x and earlier was something of a black art, but with the introduction of Java SE 5 and the adoption of public interfaces such as JMX, the task has now been made simple. With the technical details of obtaining the information out of the way, you are free to focus on the real business value: analyzing those metrics.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-59189906697228168572008-04-15T14:24:00.001-07:002008-04-15T14:24:55.215-07:00Setting up a VPN in Windows Server 2003<div id="art-descr"><a href="http://www.windowsnetworking.com/kbase/WindowsTips/Windows2003/AdminTips/Network/SettingupaVPNontheGo.html"> Setting up a VPN in Windows Server 2003</a> or Windows XP is something that is fairly easy to do in no time at all. Here is a no fluff guide on how to get it done. </div> <!-- >>> --> A Virtual Private Network (VPN) is a method used for creating a secure encrypted tunnel between two network endpoints. Creating a VPN in Windows XP or Windows Server 2003 has many advantages in a variety of different network scenarios. Setting up a simply VPN is a very quick and painless process that can be achieved in no time at all. <ol><li>Open the Control Panel </li><li>Double-click the “Network Connections” icon </li><li>Click “Create New Connection” in the left hand side of the window. </li><li>Click “Next” and then select “Setup an advanced connection” and click “Next” again </li><li>Select “Accept incoming connections” and click “Next” </li><li>Click “Next” </li><li>Select “Allow virtual private connections” and click “Next” </li><li>Place a checkmark next to the users you wish to allow to connect and click “Next” </li><li>Click “Next” and then click “Finish” </li></ol>It’s as simple that! As an added note, if you are behind a router you will want forward port 1723 which is standard for PPTP. There are obviously several other ways to implement VPN but this is your basic setup.Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-6930918428485106265.post-54884923161317098382008-04-14T08:34:00.000-07:002008-04-14T08:35:47.263-07:00Innodb Performance Optimization Basics<p id="BlogTitle"><a href="http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/">Innodb Performance Optimization Basics</a></p> <p id="BlogDate">Posted By <u>peter</u> On November 1, 2007 @ 9:17 am In <u>Innodb</u> | <u><a href="http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/print/#comments_controls">30 Comments</a></u></p> <div id="BlogContent"><p>Interviewing people for our [1] <a href="http://www.mysqlperformanceblog.com/jobs/" rel="external">Job Openings</a> I like to ask them a basic question - if you have a server with 16GB of RAM which will be dedicated for MySQL with large Innodb database using typical Web workload what settings you would adjust and interestingly enough most people fail to come up with anything reasonable. So I decided to publish the answer I would like to hear extending it with basics of Hardware OS And Application optimization.<br />I call this <strong>Innodb Performance Optimization Basics</strong> so these are general guidelines which work well for wide range of applications, though the optimal settings of course depend on the workload.<br /><a name='more'></a></p> <p><strong>Hardware</strong><br />If you have large Innodb database size Memory is paramount. 16G-32G is the cost efficient value these days. From CPU standpoint 2*Dual Core CPUs seems to do very well, while with even just two Quad Core CPUs scalability issues can be observed on many workloads. Though this depends on the application a lot. The third is IO Subsystem - directly attached storage with plenty of spindles and RAID with battery backed up cache is a good bet. Typically you can get 6-8 hard drives in the standard case and often it is enough, while sometimes you may need more. Also note new 2.5″ SAS hard drives. They are tiny but often faster than bigger ones. RAID10 works well for data storage and for read-mostly cases when you still would like some redundancy RAID5 can work pretty well as well but beware of random writes to RAID5.</p> <p><strong>Operating System </strong><br />First - run 64bit operating system. We still see people running 32bit Linux on 64bit capable boxes with plenty of memory. Do not do this. If using Linux setup LVM for database directory to get more efficient backup. EXT3 file system works OK in most cases, though if you’re running in particular roadblocks with it try XFS. You can use noatime and nodiratime options if you’re using innodb_file_per_table and a lot of tables though benefit of these is minor. Also make sure you wrestle OS so it would not swap out MySQL out of memory. </p> <p><strong>MySQL Innodb Settings </strong><br />The most important ones are:<br /><strong>innodb_buffer_pool_size</strong> 70-80% of memory is a safe bet. I set it to 12G on 16GB box.<br /><strong>UPDATE:</strong> If you’re looking for more details, check out detailed guide on[2] <a href="http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/" rel="external"> tuning innodb buffer pool</a><br /><strong>innodb_log_file_size</strong> - This depends on your recovery speed needs but 256M seems to be a good balance between reasonable recovery time and good performance<br /><strong>innodb_log_buffer_size=4M</strong> 4M is good for most cases unless you’re piping large blobs to Innodb in this case increase it a bit.<br /><strong>innodb_flush_log_at_trx_commit=2</strong> If you’re not concern about ACID and can loose transactions for last second or two in case of full OS crash than set this value. It can dramatic effect especially on a lot of short write transactions.<br /><strong>innodb_thread_concurrency=8</strong> Even with current Innodb Scalability Fixes having limited concurrency helps. The actual number may be higher or lower depending on your application and default which is 8 is decent start<br /><strong>innodb_flush_method=O_DIRECT</strong> Avoid double buffering and reduce swap pressure, in most cases this setting improves performance. Though be careful if you do not have battery backed up RAID cache as when write IO may suffer.<br /><strong>innodb_file_per_table</strong> - If you do not have too many tables use this option, so you will not have uncontrolled innodb main tablespace growth which you can’t reclaim. This option was added in MySQL 4.1 and now stable enough to use. </p> <p>Also check if your application can run in READ-COMMITED isolation mode - if it does - set it to be default as <strong>transaction-isolation=READ-COMITTED</strong>. This option has some performance benefits, especially in locking in 5.0 and even more to come with MySQL 5.1 and row level replication.</p> <p>There are bunch of other options you may want to tune but lets focus only on Innodb ones today. You can check about [3] <a href="http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-in-mysql-server-after-installation/" rel="external">tuning other options</a> here or read one of our [4] <a href="http://www.mysqlperformanceblog.com/mysql-performance-presentations/" rel="external">MySQL Presentations</a>.</p> <p><strong>Application tuning for Innodb</strong><br />Especially when coming from MyISAM background there would be some changes you would like to do with your application. First make sure you’re using transactions when doing updates, both for sake of consistency and to get better performance. Next if your application has any writes be prepared to handle deadlocks which may happen. Third you would like to review your table structure and see how you can get advantage of Innodb properties - clustering by primary key, having primary key in all indexes (so keep primary key short), fast lookups by primary keys (try to use it in joins), large unpacked indexes (try to be easy on indexes). </p> <p>With these basic innodb performance tunings you will be better of when majority of Innodb users which take MySQL with defaults run it on hardware without battery backed up cache with no OS changes and have no changes done to application which was written keeping MyISAM tables in mind. </p> </div> <hr class="Divider" style="text-align: center;"> <span style="float: right;" id="comments_controls">30 Comments (<a href="http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/print/#" onclick="javascript:document.getElementById('comments_box').style.display = 'block'; return false;">Open</a> | <a href="http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/print/#" onclick="javascript:document.getElementById('comments_box').style.display = 'none'; return false;">Close</a>)</span> <div id="comments_box"> <p id="CommentTitle">30 Comments To "Innodb Performance Optimization Basics"</p> <p class="CommentDate"> <strong>#1 Comment</strong> By <u>Jeffrey Gilbert</u> On November 1, 2007 @ 11:21 am </p> <div class="CommentContent"> <p>I’m happy to say that through reading this site regularly and getting suggestions from the forums I’ve been able to consistently shave off seconds of load time from my site over the past year bringing page load times to an almost instant state. It does take patience in testing new settings, especially when dealing with older slower 32bit hardware, but the payoffs are there and the lessons learned are priceless. My old slow query log was filled with thousands of unsolvable mysteries every day and the slow query time was only set to 10 seconds! Now that I’ve tuned everything up in the settings and have a better understanding of what each setting does in the my.cnf, I have it set to 3 seconds and only find that just around 100-200 queries a day are slower than that (usually because i dont have a failover server during backups which are causing locks that slow things down. working on it!)</p> <p>I’ve seen great speed improvements using just these tips alone. What I don’t see here which is something that many novice administrators or tuners may not know is that if you set your buffers and settings too high and restart your mysql server, mysql wont instantly complain. What I think happens is it either ignores these settings completely and uses defaults or it uses them, discovers that they dont work for the session, reverts to the defaults or recovers in some other way which is slow. This can seriously impair your performance! </p> <p>My only wishes for mysql would be that they would allow you to log queries which trigger counters of things like sort_merge_pass, full joins and tmp tables on disk so you could actually better find the queries causing slowdowns or poorly written queries in your applications, AS WELL AS a tool that would allow you to see how your buffers were being used in a visual way rather than just guessing through examining the raw numbers. These two changes would make administration lightyears more advanced than it is now for novice or intermediate developers/admins. Out of 801,000 tmp tables created, only 3,762 of those were on disk. It still bugs me that I can’t just look at a log and find them to fix them. I do have 0 Select_full_join and 0 Sort_merge_passes though finally.</p> <p>What is most confidence inspiring is thinking about the day when i can take the kid gloves off and run my database on a 64bit machine with a more acceptable amount of ram. After being hamstrung this long with 32bit chips, I can’t wait to see how things perform with the newest tech out there!</p> </div> <p class="CommentDate"> <strong>#2 Comment</strong> By <u>Jay Janssen</u> On November 1, 2007 @ 11:57 am </p> <div class="CommentContent"> <p>I have to disagree with the 70-80% of RAM usage for the buffer pool. When I asked Heikki about it at yours and his talk during the conference he admitted that was based on his test box with 1G of RAM. I’ve seen people with 64G of RAM blindly following the 80% rule and only using about 50G of RAM for the buffer poll, leaving 14G unused!</p> <p>I tend to tell people to leave a few GB for the operating system, and let the buffer pool use the rest. 4G might not be too unreasonable on a 16G box, depending on what else is going on, but I’d probably start with 2G and work up if needed. It’s super important to use O_DIRECT when tuning this, otherwise the OS will snatch up all of your free RAM for fs caching.</p> </div> <p class="CommentDate"> <strong>#3 Comment</strong> By <u>Jay Janssen</u> On November 1, 2007 @ 11:58 am </p> <div class="CommentContent"> <p>P.S.</p> <p>Good post though <img src="http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_smile.gif" alt=":)" class="wp-smiley" /> Agrees with much of what I tell people at Yahoo.</p> </div> <p class="CommentDate"> <strong>#4 Comment</strong> By <u>Xaprb</u> On November 1, 2007 @ 12:22 pm </p> <div class="CommentContent"> <p>I’d just like to point out that Peter is giving you a sneak peek at the upcoming second edition of High Performance MySQL here. This post is like the cliff notes version of the InnoDB tuning advice in the book. So if you like Peter’s posts, get the book when it comes out.</p> </div> <p class="CommentDate"> <strong>#5 Comment</strong> By <u>Jeremy Cole</u> On November 1, 2007 @ 12:22 pm </p> <div class="CommentContent"> <p>Howdy,</p> <p>Echoing what Jay says, I wouldn’t suggest a percentage for the buffer pool, rather a relatively fixed size, as the percentage doesn’t scale well as memory sizes have grown. I usually go for 14G on a 16G box, potentially reducing it if more than normal amounts of memory are needed for other things (say, a very high number of temp tables).</p> <p>Regards,</p> <p>Jeremy</p> </div> <p class="CommentDate"> <strong>#6 Comment</strong> By <u>peter</u> On November 1, 2007 @ 12:47 pm </p> <div class="CommentContent"> <p>Jay, Jeremy </p> <p>I guess “how much to use for Innodb Buffer Pool” is the question answer to which may depend a lot. As I mentioned I provide some basic guidelines in this post which I would like to be simple and 70-80% is a good answer in this case. It works for most typical range of boxes, say 4GB-32GB and it is safe even though you’re not getting the every single penny of performance. </p> <p>Your advice of leave a bit for MySQL and OS needs and give the rest to Innodb Buffer Pool is good but how one would know how much memory is needed for these ? </p> <p>Also note not everything may work as you would expect it in theory. For example even with O_DIRECT OS may be swapping out portions of MySQL due to IO pressure which may come from logs, disk based sorts or disk based temporary table.</p> <p>Another thing you need to keep into account is caching Innodb logs. As IO to Innodb logs is unaligned you better have them fit in the cache otherwise you will be getting read-around-write stalls every so often.</p> <p>But you’re right of course for 64GB you would want the buffer pool to be significantly higher than 50G</p> </div> <p class="CommentDate"> <strong>#7 Comment</strong> By <u>Keith Murphy</u> On November 1, 2007 @ 12:51 pm </p> <div class="CommentContent"> <p>Great posting. Can you do me a favor and expand on this please??? “Also make sure you wrestle OS so it would not swap out MySQL out of memory” I know what you mean by this..just don’t know how to do it..We run 64-bit Linux (debian actually).</p> <p>thanks,</p> <p>Keith</p> </div> <p class="CommentDate"> <strong>#8 Comment</strong> By <u>peter</u> On November 1, 2007 @ 1:01 pm </p> <div class="CommentContent"> <p>First. Check “si so” columns in VMSTAT - if you have some swap used but there is no swapping activity I would not worry, it is when these values are significant (sometimes in burst) you’re in trouble. </p> <p>O_DIRECT is a great if you’re using Innodb. You also can use large pages to make MyISAM key buffer and Query Cache not swapable (and get some other benefits) there are some instructions here:<br />[5] <a href="http://www.mysqlperformanceblog.com/2006/06/08/mysql-server-variables-sql-layer-or-storage-engine-specific/" rel="external">http://www.mysqlperformanceblog.com/2006/06/08/mysql-server-variables-sql-layer-or-storage-engine-specific/</a></p> <p>you can use –memlock with varying success - a lot seems to be dependent on Linux Kernel version if it works properly. You can also try to echo 0 > /proc/sys/vm/swappiness though in my experience it does not really work well for preventing swapping.</p> </div> <p class="CommentDate"> <strong>#9 Comment</strong> By <u>peter</u> On November 1, 2007 @ 4:00 pm </p> <div class="CommentContent"> <p>Jeffrey,</p> <p>You should have been looking at another post:<br />[6] <a href="http://www.mysqlperformanceblog.com/2007/10/31/new-patch-for-mysql-performance/" rel="external">http://www.mysqlperformanceblog.com/2007/10/31/new-patch-for-mysql-performance/</a></p> <p>We just created the patch which allow to log query flags with queries so you can see which queries caused on disk temporary tables and which required file sort. Now you just need small script to filter through the log.</p> <p>We surely will modify data aggregation scripts so they can use this log format.</p> </div> <p class="CommentDate"> <strong>#10 Comment</strong> By <u>Don MacAskill</u> On November 1, 2007 @ 8:25 pm </p> <div class="CommentContent"> <p>I’ve been doing all of this stuff for years… or so I thought. <img src="http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_smile.gif" alt=":)" class="wp-smiley" /> Buried in there, you say ‘having primary key in all indexes’. Can you elaborate more?</p> <p>Let’s take a sample table:</p> <p>CREATE TABLE `users` (<br />`UserID` smallint(4) unsigned NOT NULL auto_increment,<br />`Email` varchar(255) NOT NULL,<br />PRIMARY KEY (`UserID`),<br />KEY `Email` (`Email`)<br />) ENGINE=InnoDB;</p> <p>Are you saying that this would be better when doing queries for UserID based on Email:</p> <p>CREATE TABLE `users` (<br />`UserID` smallint(4) unsigned NOT NULL auto_increment,<br />`Email` varchar(255) NOT NULL,<br />PRIMARY KEY (`UserID`),<br />KEY `Email` (`Email`, `UserID`)<br />) ENGINE=InnoDB;</p> <p>?</p> <p>If so, it looks like I (wrongly?) assumed that the Primary Key was always referenced by other indexes. I’ve never seen this be a problem, that I know of, but now I’m wondering…</p> <p>Thanks!</p> </div> <p class="CommentDate"> <strong>#11 Comment</strong> By <u>Ben Schwarz</u> On November 1, 2007 @ 9:24 pm </p> <div class="CommentContent"> <p>These kinds of posts are great; really helpful to get some insight to the mysteries of innodb and mysql tuning.<br />However, my only gripe is that it all feels a bit like random ‘lets tweak this and see’, rather than putting a test suite behind it with your own hardware.</p> </div> <p class="CommentDate"> <strong>#12 Comment</strong> By <u>peter</u> On November 2, 2007 @ 1:20 am </p> <div class="CommentContent"> <p>Ben,</p> <p>Of course to get last percent of performance out of your system you need to setup benchmarks (which well match your real workload) and do experiments. However you’re better to start somewhere other than default MySQL configuration to get results fast and also you do not always have time to spend a lot of time on this. So view this as starting point for Innodb configuration from which you tune it further.</p> </div> <p class="CommentDate"> <strong>#13 Comment</strong> By <u>peter</u> On November 2, 2007 @ 1:24 am </p> <div class="CommentContent"> <p>Don,</p> <p>What I’m saying is if UserID is primary key in Innodb table the key on (Email) is internally (Email,UserID) because PK value is always stored in the index and rows are stored by it for same key value. </p> <p>This means the UserID key part of id also can be used for covering index, where clause and I think it is being fixed for filesort now. See this post for examples:<br />[7] <a href="http://www.mysqlperformanceblog.com/2006/10/03/mysql-optimizer-and-innodb-primary-key/" rel="external">http://www.mysqlperformanceblog.com/2006/10/03/mysql-optimizer-and-innodb-primary-key/</a></p> </div> <p class="CommentDate"> <strong>#14 Comment</strong> By <u>Mike</u> On November 2, 2007 @ 4:48 am </p> <div class="CommentContent"> <p>Are there any rules when specifying a server’s RAM based on the database size? Is 16GB still useful if your database is 6GB? 12GB?</p> </div> <p class="CommentDate"> <strong>#15 Comment</strong> By <u>peter</u> On November 2, 2007 @ 6:14 am </p> <div class="CommentContent"> <p>Mike,<br />Good question. Of course if your database is 6GB and you have 16GB of memory you will likely have more memory than you can efficiently use. You can allocate it as Innodb buffer pool and it will be as “free pages” or you can set buffer pool to lower value, say 7GB and let it be Free on OS side. Over time OS will find something to cache where but in practice that would not be efficient use anyway. If you plan your data size to growth I would set it to higher value so you do not have to revisit it many times adjusting as your database growths.<br />Of course if there is a mix between MyISAM and Innodb it is other story.</p> </div> <p class="CommentDate"> <strong>#16 Comment</strong> By <u>Don MacAskill</u> On November 2, 2007 @ 8:13 am </p> <div class="CommentContent"> <p>Peter,</p> <p>Oh, great, that’s how I always assumed it was. Whew. Thanks for clarifying!</p> </div> <p class="CommentDate"> <strong>#17 Pingback</strong> By <u>Choosing innodb_buffer_pool_size | MySQL Performance Blog</u> On November 3, 2007 @ 4:41 pm </p> <div class="CommentContent"> <p>[…] last post about Innodb Performance Optimization got a lot of comments choosing proper innodb_buffer_pool_size and indeed I oversimplified things a […]</p> </div> <p class="CommentDate"> <strong>#18 Comment</strong> By <u>Charlie Arehart</u> On November 3, 2007 @ 8:25 pm </p> <div class="CommentContent"> <p>No one else has commented, so maybe some think it’s self-evident, but I could some casual (new) readers being confused or misled. Where you said, “We still see people running 32bit Linux or 64bit capable boxes with plenty of memory. Do not do this”, I’m assuming you meant “on”, not “or”. <img src="http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_smile.gif" alt=":-)" class="wp-smiley" /></p> </div> <p class="CommentDate"> <strong>#19 Comment</strong> By <u>peter</u> On November 4, 2007 @ 2:51 am </p> <div class="CommentContent"> <p>Thanks Charlie, </p> <p>Fixed now.</p> </div> <p class="CommentDate"> <strong>#20 Comment</strong> By <u>Jeffrey Gilbert</u> On November 4, 2007 @ 7:50 am </p> <div class="CommentContent"> <p>peter, re #9</p> <p>That’s great news!! I didn’t expect to see something materialize so quickly. I will definitely check that out and appreciate the heads up and effort.</p> <p>best regards<br />– Jeff</p> </div> <p class="CommentDate"> <strong>#21 Comment</strong> By <u>Matthew Kent</u> On November 5, 2007 @ 2:00 pm </p> <div class="CommentContent"> <p>Trivial: but the atime stuff reminded me that nodiratime isn’t required, see [8] <a href="http://lwn.net/Articles/245097/" rel="external">http://lwn.net/Articles/245097/</a></p> </div> <p class="CommentDate"> <strong>#22 Comment</strong> By <u>peter</u> On November 5, 2007 @ 3:27 pm </p> <div class="CommentContent"> <p>Thank you Matt, </p> <p>Honestly I typically did not use it either but I got it somewhere and added is as this is one of the thing which should not hurt.</p> </div> <p class="CommentDate"> <strong>#23 Pingback</strong> By <u>» The Links » roarin’ reporter</u> On November 20, 2007 @ 9:05 pm </p> <div class="CommentContent"> <p>[…] InnoDB Performance Optimization Basics […]</p> </div> <p class="CommentDate"> <strong>#24 Comment</strong> By <u>ajay singh</u> On November 28, 2007 @ 11:24 pm </p> <div class="CommentContent"> <p>hi,<br />just wanted to know the role of mmap in innodb and how is it set … also if anyone can help in the same regard with MyISAM….<br />thank you very much ..<br />take care…<br />ajay.</p> </div> <p class="CommentDate"> <strong>#25 Comment</strong> By <u>Kirby</u> On March 4, 2008 @ 6:44 am </p> <div class="CommentContent"> <p>First off I love the blog and would like to thank all of those who contribute.</p> <p>I did want to point out though that the innodb_flush_logs_at_trx_commit setting you have listed is spelled incorrectly. If I’m not mistaken the setting is innodb_flush_log_at_trx_commit (log should not pluralized). Thought I would make an effort to point this out given the recent posting on the about checking MySQL Config files.</p> <p>Keep up the fantastic work.<br />Kirby</p> </div> <p class="CommentDate"> <strong>#26 Comment</strong> By <u>peter</u> On March 4, 2008 @ 9:59 am </p> <div class="CommentContent"> <p>Kirby,</p> <p>Thank you - fixed.</p> </div> <p class="CommentDate"> <strong>#27 Comment</strong> By <u>Thiru</u> On March 12, 2008 @ 6:41 am </p> <div class="CommentContent"> <p>“We still see people running 32bit Linux on 64bit capable boxes with plenty of memory. Do not do this.”</p> <p>Could you please explain why. </p> <p>Thanks,<br />Thiru.</p> </div> <p class="CommentDate"> <strong>#28 Comment</strong> By <u>Thiru</u> On March 12, 2008 @ 6:45 am </p> <div class="CommentContent"> <p>Oh, thank you for the many excellent posts! <img src="http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_smile.gif" alt=":)" class="wp-smiley" /></p> </div> <p class="CommentDate"> <strong>#29 Comment</strong> By <u>peter</u> On March 12, 2008 @ 12:08 pm </p> <div class="CommentContent"> <p>If you run 32bit Linux you will be limited to 32bit address space for MySQL which will limit how much memory you can use. </p> <p>Plus it will be slower for kernel to access large memory.</p> </div> <p class="CommentDate"> <strong>#30 Comment</strong> By <u>Patrick</u> On April 13, 2008 @ 7:50 pm </p> <div class="CommentContent"> <p>[..]Of course if there is a mix between MyISAM and Innodb it is other story.[…]<br />Do you still recommand thoses settings for a 65% INNODB, 35% MyISAM database ? Does MyISAM performance will be affected ? I’ll soon be switching for a MySQL dedicated server with 16Go of Ram, this post is really interesting to me.</p> </div> <hr class="Divider" style="text-align: center;"> </div> <p style="text-align: left;">Article printed from MySQL Performance Blog: <strong>http://www.mysqlperformanceblog.com</strong></p> <p style="text-align: left;">URL to article: <strong>http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/</strong></p> <p style="text-align: left;">URLs in this post:<br />[1] Job Openings: <b>http://www.mysqlperformanceblog.com/jobs/</b><br />[2] tuning innodb buffer pool: <b>http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/</b><br />[3] tuning other options: <b>http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-in-mysql-server-after-installation/</b><br />[4] MySQL Presentations: <b>http://www.mysqlperformanceblog.com/mysql-performance-presentations/</b><br />[5] http://www.mysqlperformanceblog.com/2006/06/08/mysql-server-variables-sql-layer-or-storage-engine-specific/: <b>http://www.mysqlperformanceblog.com/2006/06/08/mysql-server-variables-sql-layer-or-storage-engine-sp<br />ecific/<br /></b><br />[6] http://www.mysqlperformanceblog.com/2007/10/31/new-patch-for-mysql-performance/: <b>http://www.mysqlperformanceblog.com/2007/10/31/new-patch-for-mysql-performance/</b><br />[7] http://www.mysqlperformanceblog.com/2006/10/03/mysql-optimizer-and-innodb-primary-key/: <b>http://www.mysqlperformanceblog.com/2006/10/03/mysql-optimizer-and-innodb-primary-key/</b><br />[8] http://lwn.net/Articles/245097/: <b>http://lwn.net/Articles/245097/</b></p>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6930918428485106265.post-29230675114295125992008-04-14T04:29:00.000-07:002008-04-14T04:30:12.508-07:00MySQL Crash Recovery<h3 class="post-title"><a ca_clicked="0" href="http://www.mysqlperformanceblog.com/2006/07/30/mysql-crash-recovery/" rel="bookmark">MySQL Crash Recovery</a></h3> <div class="post-content"> <p>MySQL is known for its stability but as any other application it has bugs so it may crash sometime. Also operation system may be flawed, hardware has problems or simply power can go down which all mean similar things - MySQL Shutdown is unexpected and there could be various inconsistences. And this is not only problem as we’ll see. </p> <p>MySQL has angel process <em>mysqld_safe</em> which will restart MySQL Server in most cases. It is great, unless you have run into some bug which causes it to crash again - such crashes qucikly following one another are kind of worse because they explore many less tested code paths in MySQL and so problem potential is larger. </p> <p>So lets look at the problem which happen during the crash which might need to take care of or which may seriously affect MySQL Performance.</p> <p><strong>MyISAM Corruption </strong> - If you’re writting to MyISAM tables there is very large chance of them becoming corrupted during the crash. Note corruption may be hidden and do not expose itself instantly - you may notice wrong query results days after crash. Sometimes corrupted tables may be reason for further crashes or hangs, and corruption may spread itself further in the table. You probably do not want any of these so it is very good idea to run MySQL with<strong> myisam_recover </strong> option which will make sure all improperly closed MyISAM tables are checked first time it is accessed. This option is however rather painful to use with web applications - users may issue different queries which may trigger check/repair running for many tables at onces, which typically make system extremely slow and also can use up all allowed connections or run out of memory ( <em>myisam_sort_buffer_size</em> is normally set pretty lage). If this becomes the problem I use tiny script which moves out all MyISAM tables out of MySQL database directory, checks them with MyISAMchk and moves them back to running server. This looks scary but it works great - until table is checked and ready application gets error rather than stalling forever which allows application to become partially functional as soon as possible. This hack is needed only in some cases - in most cases using Innodb for tables which you need to be recovered fast is better solution.</p> <p><strong>Innodb Recovery</strong> - Unless you have some hardware problems (99%) or found new Innodb bug (1%) Innodb recovery should be automatic and bring your database to consistent state. Depending on<em> innodb_flush_lot_at_trx_commit </em>setting you may lose few last committed transactions but it is it. It is Performance of this process which may cause the problems. As I <a ca_clicked="0" href="http://www.mysqlperformanceblog.com/2006/07/03/choosing-proper-innodb_log_file_size/">already wrote</a> innodb_log_file_size and innodb_buffer_pool_size affect recovery time significantly as well as your workload. I should also mention if you have <em>innodb_file_per_table=1</em> your recovery speed will depend on number of Innodb tables you have, as well as many other operations, so beware. </p> <p><strong>Binary log corruption</strong> - Binary log may become corrupted and out of sync with database content. This will sometimes break replication but if you’re just planning on using binary log for point in time recovery it can go unnoticed. <strong>sync_binlog</strong> Is helping by syncing binary log, but at performance penalty. If using Innodb you also might with to use <strong>innodb-safe-binlog</strong> option in MySQL 4.1 so your Innodb log and binary log are synchronized. In MySQL 5.0 XA is taking care of this synchronization. </p> <p><strong>.frm Corruption</strong> - Few people know MySQL is not really ACID even with Innodb tables, at least not for DDL statements.<br />There is a chance of failing for example during CREATE statement with table created in Innodb dictionary but .frm not created or not completely written. Partially written .frm files or .frm being unsync with internal Innodb dictionary may cause MySQL to fail with wierd error messages. In MySQL 4.1 <strong>sync_frm</strong> option was added which reduces this problem as time window when it can happen is much less. Still if failure happens just during writting .frm file nasty things may happen, not to mention such potentially multiple operation DDL statements as<em> RENAME TABLE</em> - these are most vulnerable.</p> <p><strong>master.info corruption</strong> - If slave happens to crash you can also have relay logs corruption and master.info being corrupted. Not to mention MyISAM tables can contain partially completed statements as well as some of updates totally lost. The safe approach it to reclone the slaves if they crash or you can take the risks and try to continue. Sometimes you might be able to manually find appropriate position even if master.info file is out of sync but I would not be basing my failure handling scenarios. </p> <p><strong>Cold Start</strong> - If you restart MySQL server its caches (<em>key_buffer</em>, <em>innodb_buffer_pool</em>, <em>query_cache</em>,<em>table_cache</em>) are cleaned, so may be OS caches. This may reduce performance dramatically. So if you’re bringing server back after crash you might want to populate caches. For MyISAM key_cache this can be done by using <em>LOAD INDEX INTO CACHE</em> statement, for other storage engines it can be done by issuing large index scan queries. Full table scan queries allow to preload table data ether in storage engine caches or in OS cache. You can save these into .sql file and use –init-file to make sure it is run on startup. The other approach is to prime server with real servers (ie clone queries from other slave) before putting traffic to it.<br />In case application is not highly available so there is only one server you might with to start serving only some users initially (returning error to others) and gradually increase the load as server warms up. This may sound strange but makes a lot of sense as not only waiting for pages which never load is more frustrating for users than getting honest “try again later” message, but also - warmup takes longer time on extreme load. </p> <p><strong>Innodb statistics </strong> - Unlike MyISAM Innodb does not store index cardinality in tables, instead it computes them on first table access after startup. This may take significant time if you have very large number of tables (Some users have hundreds of thousands of tables per database host). This one is pretty much part of cold start problems but I wanted to point out it separately. To warmup this data you might run <strong>select 1 from _table_ limit 1</strong> for each table or any other statement - it is table open which is important. </p> <p>There are other problems which you may experience related to MySQL Crash Recovery - Restoring data from backup, corrupted Innodb tablespace recovery etc but I should write about them some other time.</p> </div>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6930918428485106265.post-88889984828597591292008-04-14T03:42:00.000-07:002008-04-14T04:08:13.225-07:00How To Hack A Windows XP Password1. start cmd<br />2. 'net user': it will list users/accounts of the computer, for example 'Administrator', 'cesar', etc....<br />2. 'net user <account> *': it will ask for password for <account><br />4. enter new passwordUnknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-15850794759604338572008-04-14T03:38:00.001-07:002008-04-14T03:38:38.000-07:00Should MySQL and Web Server share the same box ?<p id="BlogTitle">Should MySQL and Web Server share the same box ?</p> <p id="BlogDate">Posted By <u>peter</u> On October 16, 2006 @ 6:09 am In <u>lamp</u>, <u>production</u>, <u>tips</u> | <u><a href="http://www.mysqlperformanceblog.com/2006/10/16/should-mysql-and-web-server-share-the-same-box/print/#comments_controls">7 Comments</a></u></p> <div id="BlogContent"><p>This is interesting question which I thought it would be good to write about. There are obviously benefits and drawbacks for each of methods.</p> <p>Smaller applications usually start with single server which has both MySQL and Web server on it. In this case it is not usually the question but once application growths larger and you need to have multiple servers you may decide ether to grow system in MySQL+Apache pairs or split MySQL And Web Server and place them on different boxes.</p> <p>Generally using separate boxes for MySQL and Web Servers is rather good practice. </p> <p><strong>It is more secure</strong> - Compromising your web server does not directly give access to your database, even though most applications have enough database access permissions to be allow intruder to trash/dump data.</p> <p><strong>It is easier to analyze</strong> - Troubleshooting bottlenecks on shared boxes is more complicated compared to systems running only MySQL or only Web server. In this case you already know who is troublemaker by simply looking at system wide stats.</p> <p><strong>Easier to maintain </strong> - Same thing if box happens to run multiple things it is harder to maintain. I would not call the difference significant in this case though.</p> <p><strong>Easier to balance </strong> - Lets say you have Web application and just added some new feature, ie chat application which increases load on your web server but does not really affect database part of load. If you can operate database farm and web server farms separately you can simply increase number of web servers.</p> <p><strong>It is less expensive</strong> - You typically want database boxes to be secure, using good hardware with ECC memory to avoid database corruption, use RAID to avoid loosing database with any hard drive loss etc. Database boxes also generally require more monitoring and maintainence such as backups so you end up using some serious hardware for this boxes to keep their<br />number manageable. With Web boxes it is different - you’re quite OK using crappy hardware for them as all you need is CPU power. If box starts to misbehave it is easy to shut it down without affecting site operations. Also you rarely would have data corruption due to web boxes memory failure, more likely you’ll have web server crashes and this sort of things. You can ether clone web servers from template hard drive or even have them disk less booting by NFS.</p> <p>So if using dedicated boxes is so great why to think about sharing MySQL and Web server at all ? Well mostly it is for cheap guys.<br />In many applications you will find database servers to be IO bound so CPUs are doing virtually nothing and you’re wasting resources. This is the reason for some cheap environments to have Web servers also on database boxes, might be only handling partial load etc. </p> <p>I would however only use it in last resort - placing some data crunching scripts on database server is often better use of their free CPU time.</p> <p>Second thing you may feel bad about it is Web Servers memory. Getting certain amount of memory is pretty cheap, ie 4GB of memory per box costs very close to 2GB, while jump from 16GB to 32GB may be much more expensive (even in price by GB).<br />So you can get Web boxes with relatively plenty of memory cheap but unless you’re running 500 Apache children with mod_P (php,perl,python) per box (which is probably bad idea anyway). </p> <p>The good use for such extra memory is probably caching - Web page caching, if you do not have separate layer for it, local memory or cacheing type of caching (depending on your application needs) is very good idea.</p> <p>One more benefit of local access to MySQL is latency. This was problem many years ago with 10Mbit network but with 1Gbit networks being commodity these days you should not worry too much about it, unless you have each page generated by 1000+ queries, which is bad idea already.</p> <p>One case I should mention when shared MySQL and Web server makes sense is Web Services architecture when you can have certain boxes providing you with some simple “Services” - these could be small enough to be single shared box (or pair of shared boxes for HA). In such cases I would think about Web Server mainly being provider of different protocol to access your data - it is typically simple and would not require much of CPU and other resources itself.</p> <p>For example you can see this “Shared” kind of architecture in CNET systems using ATOMICS component to talk to MySQL over HTTP. (not what I’m great fan of this idea though).</p> </div> <hr class="Divider" style="text-align: center;"> <span style="float: right;" id="comments_controls">7 Comments (<a href="http://www.mysqlperformanceblog.com/2006/10/16/should-mysql-and-web-server-share-the-same-box/print/#" onclick="javascript:document.getElementById('comments_box').style.display = 'block'; return false;">Open</a> | <a href="http://www.mysqlperformanceblog.com/2006/10/16/should-mysql-and-web-server-share-the-same-box/print/#" onclick="javascript:document.getElementById('comments_box').style.display = 'none'; return false;">Close</a>)</span> <div id="comments_box"> <p id="CommentTitle">7 Comments To "Should MySQL and Web Server share the same box ?"</p> <p class="CommentDate"> <strong>#1 Comment</strong> By <u>John Latham</u> On October 16, 2006 @ 7:58 am </p> <div class="CommentContent"> <p>Special case for shared box: circular multi-master replication topology, with each node running web & db, datasources point to localhost. This will (in principle) scale linearly with number of boxes, until propagation delays become problematic (but less of an issue if using sticky sessions). Useful for read-heavy apps.</p> </div> <p class="CommentDate"> <strong>#2 Comment</strong> By <u>peter</u> On October 16, 2006 @ 8:18 am </p> <div class="CommentContent"> <p>John, </p> <p>Thank you for comment. </p> <p>It scales linearly only from the first glance. In reality it has problems with scaling writes (you mention it already) second as database size grows it may change from CPU bound workload to IO bound workload which slows things down dramatically. </p> <p>This is not to mention conflicting updates and complicated failure recovery for circular replication. </p> <p>In general I can only see it used then conflicting updates are not an issue and application can’t be made aware of multi server configuration.</p> </div> <p class="CommentDate"> <strong>#3 Comment</strong> By <u>Michael</u> On October 16, 2006 @ 2:43 pm </p> <div class="CommentContent"> <p>Do you have any comments on using VMWare to partition your web servers / databases as virtual machines on one or multiple (physical) boxes? To me the separate physical web server on one box and database is a better idea, but some people keep on recommending this to me.</p> </div> <p class="CommentDate"> <strong>#4 Comment</strong> By <u>peter</u> On October 17, 2006 @ 1:14 am </p> <div class="CommentContent"> <p>Michael,</p> <p>I think using VMWare and other virtualization techniques are good for two cases - testing and if you share same server among different people (in this case not VMWare but other techniques should be used of course)</p> <p>Some people also use virtualization to ease with cloning as well as configuration moving to other server - I think it is easy enough to do standard way. </p> <p>Also sharing any way limits you to resources of single server - dedicated physical web and database box will surely have more power.</p> </div> <p class="CommentDate"> <strong>#5 Pingback</strong> By <u>Zedomax Server Upgrade Complete! | zedomax.com - blog about DIYs and Review on reviews of gadgets and technologies…</u> On February 22, 2007 @ 5:55 pm </p> <div class="CommentContent"> <p>[…] If you want to know about running a more efficient web server, check out this article on mysql and web server on different boxes. […]</p> </div> <p class="CommentDate"> <strong>#6 Pingback</strong> By <u>smalls blogger » Blog Archive » links for 2007-07-12</u> On July 11, 2007 @ 6:11 pm </p> <div class="CommentContent"> <p>[…] MySQL Performance Blog » Should MySQL and Web Server share the same box ? Should MySQL and Web Server share the same box ? (tags: mysql apache server performance scaling web architecture php) […]</p> </div> <p class="CommentDate"> <strong>#7 Comment</strong> By <u>Dedicated Hosting Provider</u> On March 24, 2008 @ 10:27 am </p> <div class="CommentContent"> <p>I had an infrastructure class during my undergraduate studies and we used VMWare for the entire course. VMWare was very good for simulation of different types of issues but also had a lot of problems. It is hard to simulate real systems using virtual machines and virtual machines are very easily corrupted so you need to ensure you backup your information very frequently.</p> </div> <hr class="Divider" style="text-align: center;"> </div> <p style="text-align: left;">Article printed from MySQL Performance Blog: <strong>http://www.mysqlperformanceblog.com</strong></p> <p style="text-align: left;">URL to article: <strong>http://www.mysqlperformanceblog.com/2006/10/16/should-mysql-and-web-server-share-the-same-box/</strong></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-42027851225132199972008-04-14T03:15:00.001-07:002008-04-14T03:15:58.447-07:00MySQL 4 to MySQL 5 Upgrade performance regressions<h3 class="post-title"><a ca_clicked="0" href="http://www.mysqlperformanceblog.com/2007/06/06/mysql-4-to-mysql-5-upgrade-performance-regressions/" rel="bookmark">MySQL 4 to MySQL 5 Upgrade performance regressions</a></h3> <div class="post-content"> <p>This week I already had two serious performance regression cases when upgrading from MySQL 4.0 and 4.1 to MySQL 5.0. By serious I mean several times performance difference not just 5-10% you often see for simple queries due to generally fatter code. </p> <p>The problem in both cases was <a ca_clicked="0" href="http://www.mysqlperformanceblog.com/2006/05/19/group-commit-and-xa/">MySQL 5.0 broken group commit</a> <a ca_clicked="0" href="http://bugs.mysql.com/bug.php?id=13669" onclick="javascript:pageTracker._trackPageview ('/outbound/bugs.mysql.com');"> bug</a>. </p> <p>First I should note I am extremely unhappy how MySQL handled this problem. While working for MySQL we spotted this problem early in MySQL 5.0 release cycle as it was introduced and reported it to everyone we could inside the company - this was over 2 years ago. Few months later I created a bug for this issue to get more public attention to the problem and giving extra motivation to MySQL to fix it. Few months later I blogged about this problem with more performance results but as we can see the bug is still in Verified stage and there is no indication any work is going to have it fixed. </p> <p>I can agree this may be fundamental issue which is not easy to fix, But why is not it mentioned in <a ca_clicked="0" href="http://dev.mysql.com/doc/refman/5.0/en/upgrading-from-4-1.html" onclick="javascript:pageTracker._trackPageview ('/outbound/dev.mysql.com');">MySQL 4.1 to 5.0 upgrade notes ? </a> ? </p> <p>Furthermore if there were no good ideas how to make XA to work with group commit why would not you keep old working code path if XA is disabled ? Many customers do not flush binary log anyway and use single transactional storage engine so they do not care anyway. </p> <p>Anyway. Enough complains. We have this problem and we have to live with it, most likely MySQL 5.0 and 5.1 would not get any fixes for this problem, so lets see who is affected, how to check you’re affected and how to fix it.</p> <p><strong>Who is affected ? </strong> The good thing is only cheap guys which care about their data are typically affected, meaning you have to have <strong>innodb_flush_logs_at_trx_commit=1</strong> so transactions are truly durable. You have to have <strong>log-bin</strong> enabled to get replication or point in time recovery, but at the same time you should not have hardware RAID or have one without battery backed up cache (BBU unit). I guess this is one of the reason why this bug did not get so much traction inside MySQL - because paying customers would normally have enough money to get BBU unit which is great for performance anyway. Of course you also have to have plenty of concurrent transactions so group commit would trigger in MySQL 4.0 and large number of transactions in total so serializing them would make MySQL unable to keep up. Disks can do 80-150 single page fsyncs per second to get you an idea bout number. </p> <p><strong>How to spot you’re affected </strong> This one is interesting. If you have update prevailing load you would see very strange behavior on 5.0 of MySQL being slow but few queries being “inside innodb” and potentially even empty queue. This is because bottleneck happens in commit phase which is not counted as “inside innodb”. I wish there would be some stats for number of queries waiting to be committed but there is not one easily readable. You can see it from other symptoms though. You would see queries in <em>“SHOW PROCESSLIST” </em> stuck in “end” stage or have “commit” queries in the processlist for multiple statement transactions. Looking at <em>SHOW INNODB STATUS</em> you would notice large amount of log writes and fsyncs per second which matches your hard drive capacity. Plus you would normally see single outstanding log write all the time. There are other ways you can spot the problem as well but these are probably most obvious and easy to use. </p> <p><strong>How to fix the problem ?</strong> Assuming getting back to MySQL 4.1 is not the option you can do one of 3 things to get some of your performance back. XA support has its overhead anyway so you may not get same performance as with MySQL 4.1</p> <ol><li> <strong>Disable Binary Logging</strong> This allows to get group commit back but obviously you loose point in time recovery and replication. </li><li> <strong>Use innodb_flush_log_at_trx_commit=2</strong> This is probably best solution. In many cases this would be good change to do with MySQL 4 also because 100% durable transactions are not required anyway and it would allow to get some extra speed. It is often left at default value without a good reason. </li><li> <strong>Get BBU</strong> If you can’t use any of first two workarounds you better to get battery backed up cache unit and make sure you set your RAID cache policy to “write back”. One of the customers I worked with indeed had battery backed up cache on their system… it just was in “write through” cache policy, so was basically disabled. Note getting BBU is often good idea anyway so you can use this together with other workarounds. Also it is worth to mention BBU does not fix the problem but dramatically raises number of update transactions per second needed to trigger the problem. Without BBU 200 per sec may be well enough with BBU you may only see it if you have 2000 update transactions per second or so, which few people reach. </li></ol> <p>The other way to approach the problem is of course to work on the application - in large amount of cases the problem happens in case there are away to many updates outside of transactions in auto_commit mode. Wrap them in transactions and reduce number of commits if you can. It is great optimization idea anyway. </p> <p>Also please do not read this post as MySQL 5.0 is junk in terms of performance and you should stay on MySQL 4.1 until MySQL has taken it away from you as already happened with MySQL 4.0. MySQL 5.0 can offer substantial performance improvements in variety of cases as well as has other benefits. This is simply important regression which you better to know about. </p> </div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-3004650464446128302008-04-14T02:48:00.001-07:002008-04-14T02:48:56.803-07:00VOIP Advice<p id="BlogTitle"><a href="http://www.mysqlperformanceblog.com/2007/09/10/voip-advice-requested/#more-263">VOIP Advice requested</a></p> <p id="BlogDate">Posted By <u>peter</u> On September 10, 2007 @ 8:27 am In <u>site</u> | <u><a href="http://www.mysqlperformanceblog.com/2007/09/10/voip-advice-requested/print/#comments_controls">9 Comments</a></u></p> <div id="BlogContent"><p>As one of my last posts about [1] <a href="http://www.mysqlperformanceblog.com/2007/08/31/site-was-down-today-support-and-web-hosting/" rel="external">issues with hosting provider</a> got great response and We got a lot of good advice and offers I decided to ask for advice another problem we have as we’re growing [2] <a href="http://www.mysqlperformanceblog.com/mysql-consulting/" rel="external">our company</a> - organizing good phone communications. </p> <p>Our goals are rather simple though the fact we’re globally distributed may put us a bit aside from typical small business needs.<br /><a name='more'></a></p> <p><strong>Normal Phone Line</strong> We’re looking to get normal business phone line which would be routed to me and Vadim withing our normal business hours and voicemail outside of them. The same line could offer extensions to reach other team members plus. Plus we’d like to get direct numbers for me and Vadim at least so people do not have to pass through choice system each time. And Yes… Voicemail should be sent by email so it is easy to read it. We would like to keep using normal phones for this kind of service, while being able to use software or hardware VOIP calls for dial out. Phone conferencing features would be good to have to add people to the line.</p> <p><strong>Emergency Phone Line</strong> Providing Emergency Support to our clients we need to put better systems in place. Gurrently we give away few land lines and mobile numbers to reach us and it works well but does not look professional and neither scales as we get more people. We would like instead to get On Call schedule so calls are automatically routed to this person land and mobile numbers and if he does not replies in time followed escalation procedure to get at least anyone to the phone. The alternative is of course to use external on call center and make them to take the callers number reach someone and make him to call back, but I do not like this. From my personal experience if I have an issue I want to get to someone who can help rather than promise for someone to call back.</p> <p>Of course both lines should be multi-line ie “never busy” and phone numbers should be ours so we can switch solutions while keeping phone numbers.</p> <p>We considered two ways to implement it. First one is to use Asterisk or some derivatives as TrixBox. This way is flexible and geeky while I do not think we want to get into all aspects to get it usable as business critical solution.</p> <p>The other one is to use a hosted service so we do not need to support solution ourselves and can simplify setup. However this means this someone should be good both in providing great service quality - uptime, call quality for international calls, support and flexible enough to accomodate our needs.</p> <p>Any advice you could give us on this topic ?</p> </div> <hr class="Divider" style="text-align: center;"> <span style="float: right;" id="comments_controls">9 Comments (<a href="http://www.mysqlperformanceblog.com/2007/09/10/voip-advice-requested/print/#" onclick="javascript:document.getElementById('comments_box').style.display = 'block'; return false;">Open</a> | <a href="http://www.mysqlperformanceblog.com/2007/09/10/voip-advice-requested/print/#" onclick="javascript:document.getElementById('comments_box').style.display = 'none'; return false;">Close</a>)</span> <div id="comments_box"> <p id="CommentTitle">9 Comments To "VOIP Advice requested"</p> <p class="CommentDate"> <strong>#1 Comment</strong> By <u>Brian Aker</u> On September 10, 2007 @ 9:52 am </p> <div class="CommentContent"> <p>Hi!</p> <p>If you are going to be in Heidelberg I can sit down with you and explain the basics of setting up what you need with Asterisk.</p> <p>Cheers,<br /> -Brian</p> </div> <p class="CommentDate"> <strong>#2 Comment</strong> By <u>Troy</u> On September 10, 2007 @ 10:53 am </p> <div class="CommentContent"> <p>Asterisk is powerful, but it is a lot of work to get everything just right. We still have some echo issues with polycom phones, but at the end of the day we are in control of how our phone system works, and exactly what it does. I contacted several VOIP hosting companies and at our size, I could not find a single one that made me feel like we were important to them.</p> <p>Troy</p> </div> <p class="CommentDate"> <strong>#3 Comment</strong> By <u>zen</u> On September 10, 2007 @ 2:56 pm </p> <div class="CommentContent"> <p>Hi,<br />Asterisk seems to be the only option, and more important to you, it can grow with your company easily.<br />Re VoIP providers, it’s not really customer service problem. Just chooose one with good quality and availability. After all, you can still use your landline with Asterisk for incoming connections (with some hardware support).</p> </div> <p class="CommentDate"> <strong>#4 Comment</strong> By <u>Michael</u> On September 10, 2007 @ 4:35 pm </p> <div class="CommentContent"> <p>We’re using packet8 for the business lines, and then using angel.com for our emergency paging. A customer can dial our number, and then an emergency extension, and it goes to the on-call engineers pager via email.</p> <p>Quite useful.</p> </div> <p class="CommentDate"> <strong>#5 Comment</strong> By <u>Joel Strellner</u> On September 11, 2007 @ 11:23 am </p> <div class="CommentContent"> <p>Personally I would do it this way:</p> <p>I would use kall8.com for what you describe. They can do the PBX stuff that you want (Follow Me, scheduling, escalation, voicemail, and a bunch of other things).</p> <p>They have a pretty sweet feature for the forwarding, where you can pick how and to who calls are routed. For example, you could have your big clients provide you with a phone number, and when they call from that number it can be routed directly to you so they don’t go through the on call person or you can do some other action with it.</p> <p>They are pretty affordable to.</p> <p>Then, I would get a service like Vonage for each of you that need to be on the phone system and have the kall8 stuff point to those numbers. The reason I’d use Vonage or other voip provider is because anywhere you have internet you can get calls (you can make them too if you’d like - much cheaper than a regular telco call from another country to the US because it would be just like you were in the US).</p> <p>Then, I’d setup Vonage to have a ring list for all of your personal phones.</p> <p>Essentially, Kall8 would decide who in your team gets the phone call and Vonage would decide where you get it (Vonage phone, cell phone, etc).</p> <p>You can use normal phones with Vonage too. No special handsets needed.</p> <p>To an extent, this is what we do and it has worked very well for us.</p> </div> <p class="CommentDate"> <strong>#6 Comment</strong> By <u>peter</u> On September 12, 2007 @ 8:27 am </p> <div class="CommentContent"> <p>Thanks Brian,</p> <p>This would be great to pick your brain.</p> </div> <p class="CommentDate"> <strong>#7 Comment</strong> By <u>peter</u> On September 12, 2007 @ 8:30 am </p> <div class="CommentContent"> <p>Thanks Joel,</p> <p>This is indeed good idea. I have Vonage so I should check if it can forward the call to my UK mobile well.</p> <p>The problem with Vonage though it may not work great with forwarding calls to some countries. At least when I call to rural Russian areas I may not get through from the first time.</p> </div> <p class="CommentDate"> <strong>#8 Comment</strong> By <u>Call Center Management</u> On September 14, 2007 @ 4:49 am </p> <div class="CommentContent"> <p><b><a>Call Center Management</a></b><br />Packet8 offers the first fully integrated iPBX call center management solution.</p> </div> <p class="CommentDate"> <strong>#9 Comment</strong> By <u>Duane</u> On September 23, 2007 @ 1:29 pm </p> <div class="CommentContent"> <p>anyone that thinks asterisk is the only option and more to the point that it can scale is kidding themselves especially on the last point, go check up on google for the number of VSPs that actively moan about the instability of asterisk.</p> <p>Software Options:</p> <p> * Freeswitch - Former asterisk devs peeved about the direction/instability/crap code in Asterisk<br />* CallWeaver (formerly OpenPBX) - More peeved asterisk devs/users, tired of the dual licensing (and digium making money from their efforts) forked asterisk into a GPLv2 base and have been working heavily to fix all the problems digium won’t and it works on BSD a 1000x better then asterisk apparently<br />* SER/OpenSER - Most companies that use asterisk for a voicemail/queues/confs etc put SER or OpenSER infront to cope with the real grunt work that asterisk falls over with</p> <p>etc</p> <p>As for VSP companies, there are lots, and it really depends where you are calling etc.</p> </div> <hr class="Divider" style="text-align: center;"> </div> <p style="text-align: left;">Article printed from MySQL Performance Blog: <strong>http://www.mysqlperformanceblog.com</strong></p> <p style="text-align: left;">URL to article: <strong>http://www.mysqlperformanceblog.com/2007/09/10/voip-advice-requested/</strong></p> <p style="text-align: left;">URLs in this post:<br />[1] issues with hosting provider: <b>http://www.mysqlperformanceblog.com/2007/08/31/site-was-down-today-support-and-web-hosting/</b><br />[2] our company: <b>http://www.mysqlperformanceblog.com/mysql-consulting/</b></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-81995557656430602562008-04-14T02:42:00.000-07:002008-04-14T02:44:40.774-07:00Hosting Advice<a href="http://www.mysqlperformanceblog.com/2007/08/31/site-was-down-today-support-and-web-hosting/">Posted By <u>peter</u> On August 31, 2007 @ 5:36 pm In <u>site</u></a> <div id="BlogContent"><p>During last one and a half year we had pretty good track record with MySQL Performance Blog - there were times when site was slow (especially when backup was running) but I do not remember significant downtime, until today we went down for few hours. </p> <p>All this time the site was running on dedicated server which I rented from [1] <a href="http://www.aplus.net/" rel="external">APLUS</a> about 3 years ago. It is rather slow Celeron box with single disk and 512MB running Fedora Core 2. Despite its age (it was used “Value” server even when I got it) the server had very good track record with basically zero failures during this time - there were some network disruptions at Aplus but this is about all problems we had.<br /><a name='more'></a></p> <p>As the OS on the box become rather outdated plus server was old and had no RAID and I did not want to rely just on daily backups we got new server from [2] <a href="http://www.1and1.com/" rel="external">1and1</a> couple of weeks ago. Some people recommended them to me plus they had good price for hardware with decent base specs - 64bit CPUs, RAID remote reboot and serial console, backup etc. Plus we had one server hosted with them in Europe for couple of months for tracking European traffic in [3] <a href="http://clickaider.com/" rel="external">ClickAider</a> and it work reasonably well.</p> <p>This time I got less lucky and in about a week after we moved MySQL Performance Blog to the new server, it stopped responding. I went to the control panel and reboot the system - it does not come back with no messages at serial console at all. Worst of all it gets into situation when It thinks reboot is in progress forever so I can’t even boot it again to rescue mode.</p> <p>I call 1and1 and explain them the problem. The guy checks the system by rebooting it in rescue mode and back to normal and as it does not boot in normal mode he tells me his dedicated server team has to take a look at it and I should expect answer from them in 3-4 hours. Come one! 3-4 hours before anyone even starts looking at your problem this is as good as never for any passionate online business. The guy also tells me I can write to the Server Support team by email and they should get back to me quickly - as you may guess I’m yet still to get reply from them. </p> <p>I call again in a few hours and get to another guy. This one he tells me he can’t reboot the server and I should try to boot it in “Last known good configuration” for which I point him it is not Windows. He tells me “OK you’ve got to run hardware diagnostics when while in rescue mode” ok, I ask him what command should I run - he tells me to run “fdisk -l” (which lists partitions) I ask him to spell that to me carefully and then politely ask him to pass me to someone less clueless then he is. He refuses to do that (silly as I just can call again) I ask him to talk to his manager and he also refuses sending me to nameless complain service (which I would imagine goes directly to trash). </p> <p><strong>A side note: </strong> This actually may be the worse part. In organization your stuff members can be wrong or there may be misunderstanding with the customer so passing to the different guy or to the manager is a must for any reasonable customer service. This was the case in MySQL and Tom Basil had a magic of calming down most of rare offended customers. In [4] <a href="http://www.mysqlperformanceblog.com/mysql-consulting/" rel="external">our Consulting Work</a> we also follow the same principle - if customer requests second opinion he always gets it. </p> <p>So what is the best way to deal with clueless support stuff working for big companies/big call centers ? Of course call again. I call again and get to the different guy. This person is not so clueless which is good. Though now they can’t find any record of a call I made 3 hours ago. Anyway he goes ahead and comes back with same result - Server Team has to take a look at the server and this time I get even better time estimate - tomorrow and there is nothing he can do other than escalating the case in the system.<br />As you may guess my blood is boiling at this point. </p> <p>The next joke comes just 10 minutes ago - An email from 1and1 about closed case asking me to tell them how happy I was with service provided. How does one suppose to feel having his case solved without problem being resolved and which feedback one would provide. Of course there may be internal ticket created for mysterious “Server Team”.</p> <p>At this point I did not expect any help to match my timing so I went ahead and returned web site back to old server which happily was still available. Now I will wait just for sake of interest how long will it take 1and1 to finally solve my problem before canceling their service. </p> <p><strong>Request for Advice:</strong> As this move did not work I’m looking for other hosting location in US I would either rent 2-3 servers or best get some rack space and buy my own servers instead.</p> <p><strong>P.S </strong> Some may tell me I’m just paying for being cheap. Well it is true I’m trying to get good value at good price,<br />and it really works in most cases. I’m very happy with [5] <a href="http://www.aplus.net/" rel="external">Aplus</a> Value Server which we’ve been used as well as with our racks with [6] <a href="http://www.he.net/" rel="external">Hurricane Electric</a> and [7] <a href="http://www.blacklotus.net/" rel="external">Black Lotus</a> both are not perfect but pretty good value for the price. [8] <a href="http://www.hostik.com/" rel="external">Hostik</a> on other hand was absolutely horrible.</p> </div> <hr class="Divider" style="text-align: center;"> <span style="float: right;" id="comments_controls">27 Comments (<a href="http://www.mysqlperformanceblog.com/2007/08/31/site-was-down-today-support-and-web-hosting/print/#" onclick="javascript:document.getElementById('comments_box').style.display = 'block'; return false;">Open</a> | <a href="http://www.mysqlperformanceblog.com/2007/08/31/site-was-down-today-support-and-web-hosting/print/#" onclick="javascript:document.getElementById('comments_box').style.display = 'none'; return false;">Close</a>)</span> <div id="comments_box"> <p id="CommentTitle">27 Comments To "Site was down today, support and web hosting."</p> <p class="CommentDate"> <strong>#1 Comment</strong> By <u>James</u> On August 31, 2007 @ 9:10 pm </p> <div class="CommentContent"> <p>I have two servers at Layered Technologies (http://www.layeredtech.com/). They have excellent prices and frequent sales.</p> <p>One of the servers that I have, an AMD Athlon 2800 with 1 gig of RAM and an 80 gig EIDE drive is $80/month. I’ve had it for 18 months with no downtime whatsoever save when I had the OS reloaded with the latest version of CentOS.</p> <p>The second, an AMD Athlon 64 4000 with 1 gig of RAM and two 160 gig EIDE drives runs about $110/month or so. It had a minor issue in June that they eventually traced to a “loose network cable” (how they had a lose network cable, I don’t know), but it was resolved quickly.</p> <p>The support is a bit slow (an hour, typically, for a response), but aside from the loose network cable issue, I’ve yet to have had to contact them.</p> <p>They’re having a big sale this month with daily server deals. Frankly, I don’t think you can get a better price for this quality of service.</p> </div> <p class="CommentDate"> <strong>#2 Comment</strong> By <u>Troy</u> On August 31, 2007 @ 9:38 pm </p> <div class="CommentContent"> <p>My brother-in-law is liquidating some nice servers from a major search engine company. They are rackable (that’s the brand) c2400’s with 2 dual core xeons, 4GB ram and 4 36GB 15k scsi drives and a raid card. They are nice, I bought a bunch for my startup company (if you get 12 or more they give you the rack). I like them because they are 2U, but half depth so you can mount them back-to-back. I’m pretty sure they had mysql on them, but they come reformatted w/ no OS. I think he is selling them for $625, but I got a better deal on a group of them (plus I’m related, so I’d like to think he’d hook me up!)</p> <p>His name is Josh, you can call him at 916-226-9245 (PST)</p> </div> <p class="CommentDate"> <strong>#3 Comment</strong> By <u>Jad</u> On August 31, 2007 @ 10:14 pm </p> <div class="CommentContent"> <p>For dedicated servers, the best company for us so far has been SoftLayer. I have 3 servers there - prices may not be the lowest but support is lightning fast, which, IMO, is key when selecting a hosting provider. </p> <p>Less than a month ago, I still had servers at LayeredTech, which, like James mentioned, has good uptime but slow support. What I didn’t like at LayeredTech and was ground to taking the decision of moving my servers from there was their billing system/team.</p> <p>Support and reliability not taken into account, SL is better than LT: Last month, our bank had put a stop on our credit card because of some unusual charges they were seeing. To make the story short, recurrent billing failed at both providers - LT shut down the server right away while SL was so cooperative and ready to wait a couple days if need be (it’s good to mention that I had been with LT for over a year at that time and only less than 3 months at SL). Well guess what? As soon as the bank re-activated the credit card number, I called SL to replace my LT’ servers. They can be quick at pulling the plug, but I can be quicker <img src="http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_wink.gif" alt=";)" class="wp-smiley" /></p> </div> <p class="CommentDate"> <strong>#4 Comment</strong> By <u>James</u> On August 31, 2007 @ 10:28 pm </p> <div class="CommentContent"> <p>Perhaps I should add that my comments come with no warranty, express or implied. <img src="http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_biggrin.gif" alt=":-D" class="wp-smiley" /> </p> <p>YMMV. Some people get lucky, others get extremely unlucky.</p> <p>I’ve also heard great things about SoftLayer. You might also check out the forums at webhostingtalk.com. I usually search through the forums before considering a new provider.</p> </div> <p class="CommentDate"> <strong>#5 Comment</strong> By <u>Jeremy Cole</u> On August 31, 2007 @ 11:46 pm </p> <div class="CommentContent"> <p>Hi Peter,</p> <p>I had great success with dedicated servers from ServerBeach (owned by Peer1) for several years, and we’re now using colo space from Peer1 directly. I never had any problem whatsoever with ServerBeach, they even responded very quickly to anything I ever asked. Peer1 has been very good so far as well.</p> <p>Regards,</p> <p>Jeremy</p> </div> <p class="CommentDate"> <strong>#6 Comment</strong> By <u>michael</u> On September 1, 2007 @ 12:43 am </p> <div class="CommentContent"> <p>You should try Slicehost, best hosting experience I have ever had. Their 1gig VPS is only $70 a month. Just google them for reviews.</p> </div> <p class="CommentDate"> <strong>#7 Comment</strong> By <u>kmike</u> On September 1, 2007 @ 2:03 am </p> <div class="CommentContent"> <p>SoftLayer gets my vote, too. Good support and excellent infrastructure. Every customer gets his own private network with up to 1Gb speed - which is essential for any multi-server web site. Also you can connect directly to your VLAN via PPTP or VPN, bypassing public internet interfaces completely.</p> </div> <p class="CommentDate"> <strong>#8 Comment</strong> By <u>Marki</u> On September 1, 2007 @ 2:17 am </p> <div class="CommentContent"> <p>Lose network cables happen, even in big datacenters. Maybe when they were doing some maintenance in near server or so… Once we had issue in company I work for that the FC cables to disk array were disconnected by accident, but fortunately they realized it at the same time as we received a ticket for that…<br />Also 1 hour is not so bad (but I don’t know if that is expected resolution time or just response time). It also depends on the issue you have, of course 1 hour for manually rebooting server is too much.</p> </div> <p class="CommentDate"> <strong>#9 Comment</strong> By <u>Xaprb</u> On September 1, 2007 @ 5:04 am </p> <div class="CommentContent"> <p>I use pair’s shared hosting for about ten sites, most of them on one machine, but several customers have their own accounts. They are not the cheapest but they’ve treated me & my family extremely well. If there’s a problem I usually get an email back in a few minutes. Once there was an issue with a server suffering from another user and they moved my entire family’s account to another server. Other than needing to remember to SSH to another server, it was completely transparent.</p> <p>I’ve never had the problems people write about with dreamhost or other “the best shared hosting ever!” companies. Everyone seems to rave about how great these companies are, then get really mad and blast them for not being any good. Probably because they’re not built to scale with the popularity that comes with a bunch of people raving about you. I’d rather have hosting with a less popular, more stable company than be on that roller-coaster ride.</p> <p>I know you’re looking for dedicated, but *shrug* shared hosting is more than adequate for a blog, I think. Even one with lots of traffic. It has its advantages, too.</p> <p>I’m sorry this happened to you… I moved between several companies and had similar experiences before setting on pair. Bad service is a terrible thing. I usually associate it with a reseller, but it doesn’t sound like your company is reselling!</p> </div> <p class="CommentDate"> <strong>#10 Comment</strong> By <u>Emin</u> On September 1, 2007 @ 8:35 am </p> <div class="CommentContent"> <p>SoftLayer is by far the best. I have moved all my servers to them a few months ago and have never regretted. Excellent support, great service, solid infrastructure.</p> <p>I feel for you reading this post - I know exactly what it is that you are describing. I am only glad that due to this post 1and1 has lost a number of good potential customers as this blog is read by a lot of people. That is the only way we have to respond to poor service.</p> </div> <p class="CommentDate"> <strong>#11 Comment</strong> By <u>Eli</u> On September 1, 2007 @ 8:49 am </p> <div class="CommentContent"> <p>RackSpace dedicated servers are expensive, but the hardware, bandwidth and (especially) the service is absolutely top notch.</p> <p>They also have automatic monitoring — if the server goes down, someone is automatically notified and starts working on bringing it back up. We had some buggy applications that brought down the server a few times, but by the time I even noticed there was downtime, Rackspace support had already rebooted the server and SSH’d in to make sure everything was OK.</p> <p>The one time that I had a ticket that went more than a day unsolved (a problem with the managed backup), I called support and the woman on the other end apologized and tracked down an answer from the backup team within a few hours.</p> <p>Though, as I say, this kind of service isn’t cheap.</p> <p>Another option is a Joyent Accelerator, which is a bit like a very slick VPS.</p> </div> <p class="CommentDate"> <strong>#12 Comment</strong> By <u>peter</u> On September 1, 2007 @ 1:12 pm </p> <div class="CommentContent"> <p>Baron,</p> <p>The blog is just the part which was moved. We have a lot of other stuff on the server as well such as mail, internal company applications, version control system etc. This is just the first thing which was moved so I happily did not need to move anything back. </p> <p>I also like to have control of the system which shared hosting does not give.</p> </div> <p class="CommentDate"> <strong>#13 Comment</strong> By <u>Matt Simonsen</u> On September 2, 2007 @ 4:42 am </p> <div class="CommentContent"> <p>Hello Peter,</p> <p>If you are looking for a small to medium sized company, with top support and passionate employees, I hope you will consider SRI Hosting (http://srihosting.com).</p> <p>We have Celeron based HP DL320s which I think are a perfect value server, and could hosted the servers in multiple co-lo facilies for excellent redundancy. For higher availability applications, I would recommend using our DL360 or DL380s. Or for the lowest cost, our Xen options are perfect.</p> <p>Of the main perks at SRI, we do often focus on service. But beyond that, we have a premium product, great uptime over the last 4 years, and go so far as to offer tours of our colo facilies (1 in LA, 2 in Sacramento). Please let us know if there is anything we can do to earn your business.</p> <p>PS– We are based in California, I really am responding to emails at 4:30am <img src="http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_smile.gif" alt=":-)" class="wp-smiley" /></p> </div> <p class="CommentDate"> <strong>#14 Comment</strong> By <u>Joel Strellner</u> On September 2, 2007 @ 6:25 am </p> <div class="CommentContent"> <p>Hey Peter,</p> <p>If you decide to buy the equipment, you can colo in our rack in exchange for some of your infamous MySQL consulting each month.</p> <p>I have a full rack at XO in Sacramento, CA (US) and currently I am only using around 30U of it. You can put your 2-3 servers in there too as long as they are 1U or 2U systems.</p> <p>Let me know if this sounds interesting.</p> </div> <p class="CommentDate"> <strong>#15 Comment</strong> By <u>Marc Simony</u> On September 2, 2007 @ 8:19 am </p> <div class="CommentContent"> <p>Hi Peter,<br />We may be able to work out a barter agreement. It may not be totally free (like the MySQL Camp server for Jay), but we should chat.</p> <p>–Marc</p> </div> <p class="CommentDate"> <strong>#16 Comment</strong> By <u>peter</u> On September 2, 2007 @ 9:39 am </p> <div class="CommentContent"> <p>Thanks everyone for your help and offers,<br />I will be contacting various parties in the next few days. </p> <p>Regarding update of this issue - about 48 hours since server got down 1and1 still have not fixed it or got back to be with explanation (besides telling me the case is closed and asking to rate them). </p> <p>So stay away from 1and1 dedicated servers. Their domains seems to be OK (probably because I never had to use Support yet)</p> </div> <p class="CommentDate"> <strong>#17 Comment</strong> By <u>Markus</u> On September 2, 2007 @ 11:14 pm </p> <div class="CommentContent"> <p>I’m running my own linux distribution on a Hetzner-Server (German Hosting) without any problems so far.</p> <p>for ~50 Euro you can get:</p> <p>CPU: AMD Athlon 64 3700+<br />RAM: 1 GB DDR400 RAM<br />HDD: 2 x 160 GB SATA HDD<br />Unlimited Traffic (100Mbit/sec until 1TB, then 10MBit or you pay for each 100MBit/sec-GB)</p> <p>I’m using a mdadm-software-raid (because the onboard-raids are software too) but if you need a real hardware-raid you can get one from hetzner.</p> <p>[9] <a href="http://www.hetzner.de/rootserver_en.html" rel="external">http://www.hetzner.de/rootserver_en.html</a></p> </div> <p class="CommentDate"> <strong>#18 Comment</strong> By <u>besthosting blog</u> On September 3, 2007 @ 4:58 am </p> <div class="CommentContent"> <p>For dedicated server, my best experience is [10] <a href="http://www.lpdadicated.com/" rel="external">http://www.lpdadicated.com</a> , they were shared hosting company but growing rapidlly and it is now offering quality dedicated server. Tell them your requirements and they would able to advice accordingly . </p> <p>Good luck</p> </div> <p class="CommentDate"> <strong>#19 Comment</strong> By <u>Markus</u> On September 3, 2007 @ 10:53 pm </p> <div class="CommentContent"> <p>@besthosting blog<br />They are extrem expensive compared with hetzner. You have to pay up to $300 for the same configuration (hetzner only €60!?)</p> </div> <p class="CommentDate"> <strong>#20 Comment</strong> By <u>Michael Moody</u> On September 3, 2007 @ 11:07 pm </p> <div class="CommentContent"> <p>Dear Peter,</p> <p>In regards to backups, I should point you in the direction of a product called CDP, by r1soft.com</p> <p>We’re evaluating it for mysql backups (no load, real awesome live backups), and it’s been very exciting. I myself used it to set up replication with no downtime at all.</p> <p>In regards to the colocation, my company has sites in 4 datacenters, and here in silicon valley, we can probably offer you the colo space and bandwidth you need (at no cost). I’ll check with a higher to make sure.</p> <p>Michael</p> </div> <p class="CommentDate"> <strong>#21 Comment</strong> By <u>mariuz</u> On September 4, 2007 @ 1:48 am </p> <div class="CommentContent"> <p>In the good hosting options corner , this is what we are using currently<br />[11] <a href="http://defenderhosting.com/" rel="external">http://defenderhosting.com/</a><br />[12] <a href="http://www.layeredtech.com/" rel="external">http://www.layeredtech.com/</a><br />[13] <a href="http://nac.net/" rel="external">http://nac.net</a><br />[14] <a href="http://www.cyberhost.ro/" rel="external">http://www.cyberhost.ro/</a> (in romania)</p> <p>in past we used peer1<br />seems that they want to force you all kind of products (cisco firewall,new servers …)</p> </div> <p class="CommentDate"> <strong>#22 Comment</strong> By <u>peter</u> On September 4, 2007 @ 2:01 pm </p> <div class="CommentContent"> <p>The guys from 1and1 server team finally got back to me. It is Tuesday evening here, 4 days after I’ve filled emergency case with them.</p> </div> <p class="CommentDate"> <strong>#23 Pingback</strong> By <u>MySQL Performance Blog » VOIP Advice requested</u> On September 10, 2007 @ 8:27 am </p> <div class="CommentContent"> <p>[…] one of my last posts about issues with hosting provider got great response and We got a lot of good advice and offers I decided to ask for advice another […]</p> </div> <p class="CommentDate"> <strong>#24 Pingback</strong> By <u>ThemePassion - Best stuff about design! » Site was down today, support and web hosting.</u> On September 19, 2007 @ 1:59 pm </p> <div class="CommentContent"> <p>[…] Wendy Boswell wrote an interesting post today!.Here’s a quick excerptRequest for Advice: As this move did not work I’m looking for other hosting location in US I would either rent 2-3 servers or best get some rack space and buy my own servers instead. PS Some may tell me I’m just paying for being cheap. … […]</p> </div> <p class="CommentDate"> <strong>#25 Comment</strong> By <u>Rob Smith</u> On September 19, 2007 @ 4:47 pm </p> <div class="CommentContent"> <p>Hey Peter,</p> <p>I highly recommend Bluegecko (http://www.bluegecko.net). They’re a bit spendy but ungodly awesome. I also recommend simpli (http://www.simpli.biz).</p> <p>G’luck with your search!</p> </div> <p class="CommentDate"> <strong>#26 Comment</strong> By <u>Angel</u> On September 26, 2007 @ 12:11 am </p> <div class="CommentContent"> <p>Hey Peter,<br />Don’t get confused buddy, make your own decision about this because many of them are offering so many offers. If you will be doing like this it is the waste of time, time is so precious. Keep this in mind and go on.</p> </div> <p class="CommentDate"> <strong>#27 Pingback</strong> By <u>Traffic Tricks by Hosting Providers. | MySQL Performance Blog</u> On October 17, 2007 @ 2:20 pm </p> <div class="CommentContent"> <p>[…] already wrote once about hosting troubles which we had with this site a while back. Today we had another trouble to one of the European hit […]</p> </div> <hr class="Divider" style="text-align: center;"> </div> <p style="text-align: left;">Article printed from MySQL Performance Blog: <strong>http://www.mysqlperformanceblog.com</strong></p> <p style="text-align: left;">URL to article: <strong>http://www.mysqlperformanceblog.com/2007/08/31/site-was-down-today-support-and-web-hosting/</strong></p> <p style="text-align: left;">URLs in this post:<br />[1] APLUS: <b>http://www.aplus.net</b><br />[2] 1and1: <b>http://www.1and1.com</b><br />[3] ClickAider: <b>http://clickaider.com</b><br />[4] our Consulting Work: <b>http://www.mysqlperformanceblog.com/mysql-consulting/</b><br />[5] Aplus: <b>http://www.aplus.net</b><br />[6] Hurricane Electric: <b>http://www.he.net</b><br />[7] Black Lotus: <b>http://www.blacklotus.net</b><br />[8] Hostik: <b>http://www.hostik.com/</b><br />[9] http://www.hetzner.de/rootserver_en.html: <b>http://www.hetzner.de/rootserver_en.html</b><br />[10] http://www.lpdadicated.com: <b>http://www.lpdadicated.com</b><br />[11] http://defenderhosting.com/: <b>http://defenderhosting.com/</b><br />[12] http://www.layeredtech.com/: <b>http://www.layeredtech.com/</b><br />[13] http://nac.net: <b>http://nac.net</b><br />[14] http://www.cyberhost.ro/: <b>http://www.cyberhost.ro/</b></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-75202331110603962552008-04-14T02:40:00.000-07:002008-04-14T02:41:14.872-07:00So How can you bypass these caches?<h3 class="post-title" id="post-265"><a ca_clicked="0" href="http://www.mysqlperformanceblog.com/2007/09/12/query-profiling-with-mysql-bypassing-caches/" rel="bookmark">Query Profiling with MySQL: Bypassing caches</a></h3> <p>Quite frequently I run into question like this “I’m using <strong>SQL_NO_CACHE</strong> but my query is still much faster second time I run it, why is that ?</p> <p>The answer to this question is simple - because SQL_NO_CACHE only bypasses query cache but it has no change on other caches, which are<br /><strong>MySQL Caches </strong> - Innodb Buffer Pool and Key Buffer are best example though Falcon, PBXT and other storage engines have similar buffers. There is also table_cache both MySQL side and Internal Innodb one which can affect query execution speed.<br /><strong>OS Caches</strong> Operation Systems typically cache file IO unless you explicitely bypass it by using O_DIRECT flag or mounting file system in direct IO mode.<br /><strong>Hardware Caches</strong> State of CPU cache may affect query execution speed but only lightly, the hardware IO cache may however cause dramatic difference. Hardware RAID cache is the one but more important SAN caches which can be pretty big.</p> <p><a href="http://www.mysqlperformanceblog.com/2007/09/12/query-profiling-with-mysql-bypassing-caches/#more-265">So How can you bypass these caches?</a><br /><span id="more-265"></span></p> <p>For MySQL Caches you can restart MySQL and this is the only way to clean all of the caches. You can do <strong>FLUSH TABLES</strong> to clean MySQL table cache (but not Innodb table meta data) or you can do “set global key_buffer_size=0; set global key_buffer_size=DEFAULT” to zero out key buffer but there is no way to clean Innodb Buffer Pool without restart.</p> <p>For OS Caches on Linux you can use <a ca_clicked="0" href="http://www.linuxinsight.com/proc_sys_vm_drop_caches.html" onclick="javascript:pageTracker._trackPageview ('/outbound/www.linuxinsight.com');">drop caches</a> control available in new Linux Kernels. You could also remount file system in question and the safest thing is of course to reboot.</p> For Hardware Caches it is more hardware specific. Typically doing some IO will flush cashes but you can’t be sure as you do not know what policies exactly do they employ. For RAID hardware caches reboot of the box is also enough however SAN caches may survive longer. Though few of us have SAN available for performance benchmarkingUnknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-64321223020765707652008-04-14T02:36:00.000-07:002008-04-14T02:37:29.940-07:00MySQL Performance - eliminating ORDER BY function<p id="BlogTitle"><a href="http://www.mysqlperformanceblog.com/2007/10/17/mysql-performance-eliminating-order-by-function/">MySQL Performance - eliminating ORDER BY function</a></p> <p id="BlogDate">Posted By <u>peter</u> On October 17, 2007 @ 5:24 am In <u>optimizer</u> | <u><a href="http://www.mysqlperformanceblog.com/2007/10/17/mysql-performance-eliminating-order-by-function/print/#comments_controls">5 Comments</a></u></p> <p>One of the first rules you would learn about MySQL Performance Optimization is to avoid using functions when comparing constants or order by. Ie use <strong>indexed_col=N</strong> is good. <strong>function(indexed_col)=N</strong> is bad because MySQL Typically will be unable to use index on the column even if function is very simple such as arithmetic operation. Same can apply to order by, if you would like that to use the index for sorting. There are however some interesting exception.<br /><a name='more'></a></p> <p>Compare those two queries for example. If you look only at ORDER BY clause you would see first query which sorts by function is able to avoid order by while second which uses direct column value needs to do the filesort:</p> <div class="igBar"><span id="lsql-1">[1] <a href="http://www.mysqlperformanceblog.com/2007/10/17/mysql-performance-eliminating-order-by-function/print/#" rel="external">PLAIN TEXT</a></span></div> <div class="syntax_hilite"><span class="langName">SQL:</span> <div id="sql-1"> <div class="sql"> <ol><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;">mysql> <span style="color: rgb(153, 51, 51); font-weight: bold;">EXPLAIN</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">SELECT</span> * <span style="color: rgb(153, 51, 51); font-weight: bold;">FROM</span> tst <span style="color: rgb(153, 51, 51); font-weight: bold;">WHERE</span> i=<span style="color: rgb(128, 0, 0);">5</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">AND</span> date<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span>d<span style="color: rgb(0, 102, 0); font-weight: bold;">)</span>=date<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span>now<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span><span style="color: rgb(0, 102, 0); font-weight: bold;">)</span><span style="color: rgb(0, 102, 0); font-weight: bold;">)</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">ORDER</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">BY</span> date<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span>d<span style="color: rgb(0, 102, 0); font-weight: bold;">)</span> \G</div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;">*************************** <span style="color: rgb(128, 0, 0);">1</span>. row ***************************</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> id: <span style="color: rgb(128, 0, 0);">1</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> select_type: SIMPLE</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> <span style="color: rgb(153, 51, 51); font-weight: bold;">TABLE</span>: tst</div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> type: ref</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;">possible_keys: i</div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> <span style="color: rgb(153, 51, 51); font-weight: bold;">KEY</span>: i</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> key_len: <span style="color: rgb(128, 0, 0);">5</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> ref: const</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> rows: <span style="color: rgb(128, 0, 0);">10</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> Extra: <span style="color: rgb(153, 51, 51); font-weight: bold;">USING</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">WHERE</span></div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"><span style="color: rgb(128, 0, 0);">1</span> row <span style="color: rgb(153, 51, 51); font-weight: bold;">IN</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">SET</span> <span style="color: rgb(0, 102, 0); font-weight: bold;">(</span><span style="color: rgb(128, 0, 0);">0</span>.<span style="color: rgb(128, 0, 0);">00</span> sec<span style="color: rgb(0, 102, 0); font-weight: bold;">)</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> </div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;">mysql> <span style="color: rgb(153, 51, 51); font-weight: bold;">EXPLAIN</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">SELECT</span> * <span style="color: rgb(153, 51, 51); font-weight: bold;">FROM</span> tst <span style="color: rgb(153, 51, 51); font-weight: bold;">WHERE</span> i=<span style="color: rgb(128, 0, 0);">5</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">AND</span> date<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span>d<span style="color: rgb(0, 102, 0); font-weight: bold;">)</span>=date<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span>now<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span><span style="color: rgb(0, 102, 0); font-weight: bold;">)</span><span style="color: rgb(0, 102, 0); font-weight: bold;">)</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">ORDER</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">BY</span> d \G</div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;">*************************** <span style="color: rgb(128, 0, 0);">1</span>. row ***************************</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> id: <span style="color: rgb(128, 0, 0);">1</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> select_type: SIMPLE</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> <span style="color: rgb(153, 51, 51); font-weight: bold;">TABLE</span>: tst</div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> type: ref</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;">possible_keys: i</div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> <span style="color: rgb(153, 51, 51); font-weight: bold;">KEY</span>: i</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> key_len: <span style="color: rgb(128, 0, 0);">5</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> ref: const</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> rows: <span style="color: rgb(128, 0, 0);">10</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> Extra: <span style="color: rgb(153, 51, 51); font-weight: bold;">USING</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">WHERE</span>; <span style="color: rgb(153, 51, 51); font-weight: bold;">USING</span> filesort</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"><span style="color: rgb(128, 0, 0);">1</span> row <span style="color: rgb(153, 51, 51); font-weight: bold;">IN</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">SET</span> <span style="color: rgb(0, 102, 0); font-weight: bold;">(</span><span style="color: rgb(128, 0, 0);">0</span>.<span style="color: rgb(128, 0, 0);">00</span> sec<span style="color: rgb(0, 102, 0); font-weight: bold;">)</span> </div> </li></ol> </div> </div> </div> <p>If you take a closer look to WHERE clause you will find the reason - date(d) equals to date(now()) which is constant which means we're sorting by constant and so sort phase can be skipped all together. </p> <p>Note in this case MySQL Optimizer is rather smart and is able to do this even if we have function in ORDER BY and exactly the same function is equals to constant by WHERE clause. If course it works for direct constants as well. </p> <p>However if functions are different MySQL is not able to do this optimization even in cases when this would be possible:</p> <div class="igBar"><span id="lsql-2">[2] <a href="http://www.mysqlperformanceblog.com/2007/10/17/mysql-performance-eliminating-order-by-function/print/#" rel="external">PLAIN TEXT</a></span></div> <div class="syntax_hilite"><span class="langName">SQL:</span> <div id="sql-2"> <div class="sql"> <ol><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;">mysql> <span style="color: rgb(153, 51, 51); font-weight: bold;">EXPLAIN</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">SELECT</span> * <span style="color: rgb(153, 51, 51); font-weight: bold;">FROM</span> tst <span style="color: rgb(153, 51, 51); font-weight: bold;">WHERE</span> i=<span style="color: rgb(128, 0, 0);">5</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">AND</span> date<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span>d<span style="color: rgb(0, 102, 0); font-weight: bold;">)</span>=date<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span>now<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span><span style="color: rgb(0, 102, 0); font-weight: bold;">)</span><span style="color: rgb(0, 102, 0); font-weight: bold;">)</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">ORDER</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">BY</span> unix_timestamp<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span>date<span style="color: rgb(0, 102, 0); font-weight: bold;">(</span>d<span style="color: rgb(0, 102, 0); font-weight: bold;">)</span><span style="color: rgb(0, 102, 0); font-weight: bold;">)</span> \G</div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;">*************************** <span style="color: rgb(128, 0, 0);">1</span>. row ***************************</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> id: <span style="color: rgb(128, 0, 0);">1</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> select_type: SIMPLE</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> <span style="color: rgb(153, 51, 51); font-weight: bold;">TABLE</span>: tst</div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> type: ref</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;">possible_keys: i</div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> <span style="color: rgb(153, 51, 51); font-weight: bold;">KEY</span>: i</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> key_len: <span style="color: rgb(128, 0, 0);">5</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> ref: const</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> rows: <span style="color: rgb(128, 0, 0);">10</span></div> </li><li style="font-weight: bold; color: rgb(38, 83, 106);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"> Extra: <span style="color: rgb(153, 51, 51); font-weight: bold;">USING</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">WHERE</span>; <span style="color: rgb(153, 51, 51); font-weight: bold;">USING</span> filesort</div> </li><li style="font-family: 'Courier New',Courier,monospace; font-weight: normal; font-style: normal; color: rgb(58, 106, 139);"> <div style="font-family: 'Courier New',Courier,monospace; font-weight: normal;"><span style="color: rgb(128, 0, 0);">1</span> row <span style="color: rgb(153, 51, 51); font-weight: bold;">IN</span> <span style="color: rgb(153, 51, 51); font-weight: bold;">SET</span> <span style="color: rgb(0, 102, 0); font-weight: bold;">(</span><span style="color: rgb(128, 0, 0);">0</span>.<span style="color: rgb(128, 0, 0);">00</span> sec<span style="color: rgb(0, 102, 0); font-weight: bold;">)</span> </div> </li></ol> </div> </div> </div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-54114324549087337112008-04-14T02:15:00.000-07:002008-04-14T02:16:10.976-07:00RAID System performance surprises<h3 class="post-title" id="post-339"><a ca_clicked="0" href="http://www.mysqlperformanceblog.com/2008/03/05/raid-system-performance-surprises/#more-339" rel="bookmark">RAID System performance surprises</a></h3> <div class="post-content"> <p>Implementing MySQL database in 24/7 environments we typically hope for uniform component performance, or at least would like to be able to control it. Typically this is indeed the case, for example CPU will perform with same performance day and night (unless system management software decides to lower CPU frequency due to overheating). </p> <p>This is also almost the case with Hard Drives - there are possible performance differences based on where data is stored on the disk, amount of remapped sectors etc. There is also database and file system fragmentation however these also tend to accumulate in predictable fashion. </p> <p>If you have RAID controller this well may not be the case - to protect your data RAID controller may implement bunch of algorithms which can affect performance dramatically.<br /><span id="more-339"></span></p> <p>For example speaking about PERC5 (LSI MegaRaid) - Pretty typical controller from Dell installed on PowerEdge 1950, 2950 etc you should be aware of couple of things </p> <p><strong>Battery Learning and Charging </strong> Build in Battery has to pass through learning cycle every 3 months or so and this cycle takes about 7 hours according to the <a ca_clicked="0" href="http://support.ap.dell.com/support/edocs/storage/RAID/PERC5/en/UG/HTML/chapterb.htm" onclick="javascript:pageTracker._trackPageview ('/outbound/support.ap.dell.com');">docs</a>. During this time battery backed up cache will be disabled and system will operate with write through cache possibly slowing down write performance several times. </p> <p><strong>Patrol Read </strong> This is feature which should discover bad sectors before it is too late and it is doing so by doing disk read checks periodically. When it wakes up it will some IO resources (30% by default) which will affect your performance some way.</p> <p><strong>Consistency Checks</strong> This is another thing which I’ve seen initiated by controller (though I’m not sure on this one) - which pretty much checks the disks are in sync - this also can slow down performance dramatically. </p> <p><strong>So what you can do about these ?</strong></p> <p>First you should not have any of this to come as surprise for you when you discovered your server just stopped performance during the time you planned investor show case or other important event. Learn what cron jobs does your RAID card have and see how they can be controlled - may be schedule them during least busy intervals or something similar.</p> <p>Also you should be ready for degraded and rebuild RAID mode - when one of the disk fails and you replace it with another one which needs to be rebuilt. This means you already should leave some slack of the system. It often would be enough for consistency check and patrol read but not for battery backed up cache being temporary disabled. </p> <p>Another thing you can do is of course switch to another server and take this down for maintenance if this learning process can’t be scheduled when it is non intrusive. To do this properly however you need to know when it is about to happen.</p> </div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-8801290311650789782008-04-09T20:01:00.001-07:002008-04-09T20:01:46.127-07:00Configuring SSH on Windows 2003<h1 class="topictitle1"><a href="http://publib.boulder.ibm.com/infocenter/tivihelp/v14r1/index.jsp?topic=/com.ibm.tivoli.tpm.sec.doc/security/tsec_ssh2003.html">Configuring SSH on Windows 2003</a></h1> <div><div>When you configure OpenSSH on the <span>Tivoli<sup>®</sup> <span class="keyword">Intelligent Orchestrator</span></span> server on a Windows 2003 environment, <span>SSH is installed, when Cygwin is installed.</span></div><div class="section">To configure SSH on Windows 2003:</div> <ol><li class="stepexpand"><span>Log on as tioadmin.</span></li><li class="stepexpand"><span>Invoke a Cygwin window.</span></li><li class="stepexpand"><span>Type the following command in the Cygwin window: <samp class="codeph">/usr/bin/ssh-host-config -y</samp>.</span> After typing this command, the following functions occur:<ul><li>Host keys are generated. This command generates three different keys-<samp class="codeph">DSA, RSA, RSA1</samp>, each corresponding to a different encryption algorithm. These keys allow a system to establish SSH sessions with systems requiring any one of these encryption algorithms.</li><li>Privilege separation is enabled.</li><li>sshd is installed as a service.</li><li>Passwordless logon is enabled.</li><li>New user account is created.</li></ul> </li><li class="stepexpand"><span>You will then be prompted for a password for the new user account that has been created. Enter a password for the new user and ensure that this password matches the password rules given on your system.</span> Output should be similar to this: <pre>$ ssh-host-config -y<br />Generating /etc/ssh_host_key<br />Generating /etc/ssh_host_rsa_key<br />Generating /etc/ssh_host_dsa_key<br />Overwrite existing /etc/ssh_config file? (yes/no) yes<br />Generating /etc/ssh_config file<br />Overwrite existing /etc/sshd_config file? (yes/no) yes<br />Privilege separation is set to yes by default since OpenSSH 3.3.<br />However, this requires a non-privileged account called 'sshd'.<br />For more info on privilege separation read<br />/usr/share/doc/openssh/README.privsep.<br />Should privilege separation be used? (yes/no) yes<br />Generating /etc/sshd_config file Added ssh to /etc/inetd.conf Warning:<br />The following functions require administrator privileges!<br />Do you want to install sshd as service?<br />(Say "no" if it's already installed as service) (yes/no) yes<br /><br />You appear to be running Windows 2003 Server or later. <br />On 2003 and later systems, it's not possible to use the<br />LocalSystem account if sshd should allow passwordless logon<br />(e. g. public key authentication). If you want to enable that functionality,<br />it's required to create a new account sshd_server' with special privileges,<br />which is then used to run the sshd service under.<br /><br />Should this script create a new local account 'sshd_server' which has<br />the required privileges? (yes/no) yes<br />Please enter a password for new user 'sshd_server'. Please be sure that this<br />password matches the password rules given on your system. Entering no password<br />will exit the configuration. <br /><br />PASSWORD=<br /><br />User 'sshd_server' has been created with password 'xxxxx'.<br /><br />If you change the password, please keep in mind to change the password<br />for the sshd service, too. Also keep in mind that the user sshd_server<br />needs read permissions on all users'.ssh/authorized_keys file to allow<br />public key authentication for these users!. <br /><br />(Re-)running ssh-user-config for each user will set the required<br />permissions correctly.</pre> </li><li class="stepexpand"><span>Accept the default value for the environment variable <strong>CYGWIN</strong>, when sshd is started and press <strong>Enter</strong>.</span> Output should be similar to this: <pre>Which value should the environment variable CYGWIN have when sshd starts?<br />It's recommended to set at least "ntsec" to be able to change user context<br />without password.<br />Default is "ntsec". CYGWIN=<br /><br />The service has been installed under sshd_server account.<br />To start the service, call `net start sshd' or `cygrunsrv -S sshd'.<br /><br />Host configuration finished. Have fun!</pre> </li><li class="stepexpand"><span>If the user account created by Cygwin was not automatically added to the Administrators group, then you will get a warning similar to this: </span> <pre>WARNING: Adding user sshd_server to local group root Administrators failed!<br />Please add sshd_server to local group root Administrators before starting<br />the sshd service!</pre> </li><li class="stepexpand"><span>To manually add the user account created by Cygwin to the <strong>Administrators</strong> group, go to <span class="menucascade"><span class="uicontrol">My Computers</span> > <span class="uicontrol">Manage</span></span> and then add the user account to the <strong>Administrators</strong> group. </span></li><li class="stepexpand"><span>Ensure that the newly created user account has been added to the Administrators group by typing the following command in DOS window:</span> <pre>net localgroup administrators</pre> </li><li class="stepexpand"><span>Start the sshd service by running the following command (it will automatically start when rebooted) : </span> <pre>cygrunsrv -S sshd</pre> </li><li class="stepexpand"><span>Verify that <span class="cmdname">tioadmin</span> has a password in <span class="filepath">/etc/passwd</span>. If no password is found, then you must generate one by running the command:</span> <pre>cat /etc/passwd | grep tioadmin || mkpasswd -l > /etc/passwd</pre> </li><li class="stepexpand"><span>Configure SSH for tioadmin, by typing the following command: </span> <pre>ssh-user-config</pre> </li><li class="stepexpand"><span>When prompted to created the identity files, choose your option for the ssh version that will be used. This is a configuration example for using SSH2 (which is recommended). Press <strong>Enter</strong> when prompted for a passphrase. </span> Output should be similar to this: <pre>Shall I create an SSH1 RSA identity file for you? (yes/no) <strong>no</strong><br />Shall I create an SSH2 RSA identity file for you? (yes/no) (yes/no) <strong>yes</strong><br />Enter passphrase (empty for no passphrase):<br />Enter same passphrase again:<br />Do you want to use this identity to login to this machine? (yes/no) <strong>yes</strong><br />Shall I create an SSH2 DSA identity file for you? (yes/no) (yes/no) <strong>no</strong><br /><br />Configuration finished. Have fun!</pre> </li><li class="stepexpand"><span>Switch to the <samp class="codeph">.ssh</samp> directory by typing: <samp class="codeph">cd .ssh</samp>. Press <strong>Enter</strong>.</span></li><li class="stepexpand"><span>The user key must be put into the <samp class="codeph">authorized_keys</samp> file of the user account on the server. To perform this task, run the command:</span> <pre>cat id_rsa.pub >>authorized_keys</pre> </li><li class="stepexpand"><span>To configure SSH to accept connections from new hosts without prompting for confirmation, create a file in <samp class="codeph">/home/tioadmin/.ssh</samp> called <samp class="codeph">config</samp>.</span> <div class="note"><span class="notetitle">Note:</span> This directory depends on how you have set up your system. To confirm where <samp class="codeph">/home</samp> directory is, look at <samp class="codeph">/etc/passwd</samp>.</div> Run the command:<pre>echo "StrictHostKeyChecking no" > config</pre> The file should contain the value of <pre>StrictHostKeyChecking no</pre> </li><li class="stepexpand"><span>To verify that SSH is configured properly, do the following: </span><ol type="a"><li class="substepexpand"><span>Ensure the Cygwin service is started.</span></li><li class="substepexpand"><span>To log on to the local host through SSH, type <samp class="codeph">ssh tioadmin@<span class="synph"><span class="var"><localhost></span></span></samp>, where <span class="synph"><span class="var"><localhost></span></span> is your host name. If SSH is properly configured you will see the following message: </span> <pre>Fanfare!!!<br />You are successfully logged in to this server!!!</pre> </li><li class="substepexpand"><span>Exit the sshd session by typing <samp class="codeph">exit</samp>.</span> <div class="note"><span class="notetitle">Note:</span> Ensure that the sshd service is set to automatic startup. To verify that the service is set to automatic:<ol type="i"><li>Open the <span class="wintitle">Services</span> window.</li><li>Select <span class="uicontrol">sshd</span>. </li><li>Right-click <span class="uicontrol">Properties</span> then <span class="uicontrol">Start Mode</span>. <span class="uicontrol">Automatic</span> should be selected.</li></ol> </div> </li></ol> </li><li class="stepexpand"><span>Copy the <samp class="codeph">id_rsa.pub</samp> file, which contains the public keys, into the authorized keys file of the administrative account of any server in the data center that the <span>Tivoli <span class="keyword">Intelligent Orchestrator</span></span> server must communicate with or manage. Include any servers in the data center that <span>Tivoli <span class="keyword">Intelligent Orchestrator</span></span> is managing.</span><ol type="a"><li class="substepexpand"><span>Ensure that the managed server has an administrative account for which the SSH RSA keys (<samp class="codeph">id_rsa</samp>,<samp class="codeph"> id_rsa.pub</samp>, and <samp class="codeph">authorized_keys</samp>) have already been generated and should be contained into the .ssh directory of the respective administrative account home directory.</span></li><li class="substepexpand"><span>Append the content (a single line of text) of the <samp class="codeph">id_rsa.pub</samp> file which contains the public key from the server that will initiate the SSH session to the <samp class="codeph">authorized_keys</samp> file of the administrative account of any target server in the data center that the <span>Tivoli <span class="keyword">Intelligent Orchestrator</span></span> server must communicate with or manage. Include any servers in the data center that <span>Tivoli <span class="keyword">Intelligent Orchestrator</span></span> will be managing.</span></li><li class="substepexpand"><span>To verify, on the <span>Tivoli <span class="keyword">Intelligent Orchestrator</span></span> server, type:</span> <pre>ssh <tioadmin/other_administrative_account_on_the_target_server>@<target_server_ip_or_hostname></pre> There should be no password prompt, followed by the prompt on the remote machine. After a successful logon, an entry for the communication partner will be created into a <samp class="codeph">known_hosts</samp> file. As a troubleshooting step, sometimes this file may contain old or invalid entries associated with the managed server IP address or name. Deleting that entry should fix the connection problem.</li></ol> </li></ol> <div class="section">SSH is configured for Windows 2003.</div> </div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-30972036256574399412008-04-09T19:51:00.000-07:002008-04-09T19:53:09.512-07:00A Comparison of Free SSH and SCP Programs for Windows 9x, NT, ME, 2000 and XP<span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Introduction"><span style="color:#000000;"></span></a><span style="color:#000000;"><a href="http://www.jfitz.com/tips/ssh_for_windows.html">Source: http://www.jfitz.com/tips/ssh_for_windows.html</a><br /><br />Introduction</span></b><br /><br />If you want to access remote servers securely from Windows 9x, NT, ME, 2000 or XP and don't want to pay for programs that are freely available for UNIX-like platforms, you may find this document useful. It describes free SSH implementations for Windows.<br /><br />Although this document concentrates on Windows-specific implementations, much of the content, particularly relating to Port Forwarding and Authorized Keys, applies to any SSH implementation, regardless of the platform. VPN technologies, (including IPSEC), are discussed in passing, but these are not the main focus.<br /><br />Email me, <a href="mailto:fitz@jfitz.com">fitz@jfitz.com</a>, if you find better implementations, or if you think there's anything I can add here. Please note that I will only include references to products that are available free of charge, and I have a strong preference for open source implementations.<br /><br />For more links to SSH tools for Windows, (and Mac), I'd recommend trying the OpenSSH Windows/Mac pages: <a href="http://www.openssh.com/windows.html">http://www.openssh.com/windows.html</a>, or <a href="http://www.openssh.com/macos.html">http://www.openssh.com/macos.html</a>.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="What_is_SSH"><span style="color:#000000;">What is SSH?</span></a></b><br /><br />SSH is a protocol that uses public key cryptography to transfer data securely over insecure networks. At the core of most SSH implementations are the "ssh" and "scp" commands.<br /><br />"ssh" and "scp" are "secure" versions of the UNIX "rsh" and "rcp" commands, (which in turn are a bit like the more familiar "telnet" and "ftp" programs). The "ssh" command provides command-line, (non-GUI), access to a remote server. "scp" allows files to be copied to/from a remote server. Both programs use the SSH protocol to establish a secure connection and to encrypt all data passing between the client and the server.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="What_is_Port_Forwarding_or_Tunneling"><span style="color:#000000;">What is "Port Forwarding" or "Tunneling"?</span></a></b><br /><br />In addition to the direct access provided by the "ssh" and "scp" commands, the SSH protocol includes a feature called "Port Forwarding", or "Tunneling". This can be used to provide secure access to other services that do not normally encrypt data during transmission.<br /><br />For example, to receive email, many email programs use the (unencrypted) POP3 protocol to connect to a mailserver on port 110. If we can SSH to the mailserver before downloading our mail, we can configure the SSH session to "forward" port 110 through the encrypted SSH link. Now, if we configure our email client to connect to port 110 on our <i><b>local</b></i> machine it will in fact be connecting to the <i><b>remote</b></i> mailserver, but all the data, (i.e. our incoming email), will be transferred over the encrypted SSH link.<br /><br />If we also forward port 25, (which normally handles SMTP, the outgoing email protocol), then both incoming and outgoing email will be encrypted.<br /><br />If you are mainly interested in securing your Windows email client, you could try jumping straight to my <a href="http://www.jfitz.com/tips/putty_config.html">step-by-step guide</a> to configuring a <a href="http://www.chiark.greenend.org.uk/%7Esgtatham/putty/">PuTTY SSH client</a> for port forwarding email traffic.<br /><br />Any number of ports can be forwarded in a single SSH session. For example, we could also forward port 80, (HTTP), to provide secure access to a corporate webserver. In effect, it is usually possible to create a pretty good approximation to a VPN, (Virtual Private Network), just by forwarding a handful of common ports, and using scp, (or WinSCP), to transfer files. In fact, a VPN is roughly equivalent to an SSH session where <i><b>everything</b></i> is being forwarded over a single secure channel.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Should_I_use_SSH_or_a_VPN"><span style="color:#000000;">Should I use SSH or a VPN?</span></a></b><br /><br />If you are trying to decide between SSH and a VPN, I find that SSH is generally appropriate in situations where occasional remote access is needed to specific services, (for example, checking email, downloading files). If persistent access to a wide range of services is required, (for example, working exclusively on a corporate network while telecommuting), then a VPN is probably a better choice.<br /><br />One of the problems with a VPN is that both the client and server need to be customized for the VPN to work, whereas most modern UNIX-like servers will provide SSH access by default when a user account is established, (so only the client PC needs configuring). A second problem is that traditional VPNs tended to be difficult to configure correctly, particularly on the server-side. However, solutions such as OpenVPN, (<a href="http://openvpn.net/">http://openvpn.net</a>), are much easier to install and configure than many of the more traditional, IPSEC-based VPN solutions, so if you have administrative (root) privileges on the server it may be worth exploring OpenVPN. A VPN may require more work initially to get running on the server, but it will ultimately provide more flexibility on the client.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="PuTTY"><span style="color:#000000;">PuTTY</span></a></b><br /><br />PuTTY, (and its file transfer utility, pscp), are excellent Windows ssh/scp implementations, and are a piece of cake to get up and running.<br /><br />Download putty.exe and pscp.exe here: <a href="http://www.chiark.greenend.org.uk/%7Esgtatham/putty/">http://www.chiark.greenend.org.uk/~sgtatham/putty/</a>, and you're ready to go. These programs are not zipped and do not require any installation.<br /><br />When you run PuTTY, a configuration screen is presented first. It's a good idea to play with the configuration until you get colors, fonts and other settings that suit you. You can then give the session a name and save it, so it's easy to restore these settings next time you run PuTTY, (just double-click the stored session name). This process can be a little awkward because you need to exit and restart PuTTY to update a stored session, but once you have a good set of settings saved, you can use that session as a template to create other stored sessions for different host machines.<br /><br />To help you get up and running with PuTTY, I've created a step-by-step guide to configuring a "typical" PuTTY session, (including port forwarding for incoming/outgoing email). You can check it out here: <a href="http://www.jfitz.com/tips/putty_config.html">http://www.jfitz.com/tips/putty_config.html</a><br /><br />When you first connect to a new machine using PuTTY or pscp, they will give you a message to indicate that they don't recognize the machine. Choose "OK" to add the machine to the list of known hosts and to continue connecting.<br /><br />PuTTY works a bit like an xterm. If you highlight text with the mouse it is automatically copied to the clipboard. A right-click of the mouse will paste the copied text at the command-line cursor position. Right-click anywhere on PuTTY's Title Bar, (or left-click on the PuTTY icon in the top left corner), to access a menu where the current session settings can be changed.<br /><br />To remove PuTTY automatically, run "<span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">putty -cleanup</span>", (without the quotes). To do the job manually, delete the files you downloaded and use regedit to remove all the keys under<br /><span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">HKEY_CURRENT_USER->Software->SimonTatham->PuTTY</span>.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="WinSCP"><span style="color:#000000;">WinSCP</span></a></b><br /><br />WinSCP provides an easy-to-use graphical interface to "scp" functionality. It creates an SSH link to the remote server, then displays local and remote files in "Explorer-like" windows. Copying files securely to/from the remote machine is as simple as dragging and dropping, (or cutting and pasting), files in the WinSCP windows.<br /><br />As of version 3, WinSCP also supports SFTP, ("secure ftp"). Unless you specifically need to use SFTP, it's probably as easy to stick with SCP -- the basic WinSCP functionality handles many of the issues that SFTP was designed to address.<br /><br />You can download WinSCP here: <a href="http://winscp.net/eng/">http://winscp.net/eng/</a><br /><br />WinSCP starts with a configuration screen and can save session settings in a manner that is very similar to PuTTY.<br /><br />As of version 3, WinSCP includes a "Cleanup" option that allows you to remove any possibly sensitive settings that may have been created on your computer. With version 2, to remove WinSCP delete WinSCP2.exe and, to clean up the registry, use regedit to remove all the keys under<br /><span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">HKEY_CURRENT_USER->Software->Martin Prikryl->WinSCP 2</span>.<br /><br />WinSCP is released under the GPL. The source code is available for download with the application. WinSCP is partly based on PuTTY code, (it is usually pretty up-to-date -- check the WinSCP site for version details).<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="pscp"><span style="color:#000000;">pscp</span></a></b><br /><br />pscp is a companion program of PuTTY. It is a pretty straightforward Windows implementation of scp. It's a command-line program only, so you need to run it in an MS-DOS window.<br /><br />The basic format of a pscp command, (or any scp command for that matter), is:<br /><br /><span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">pscp myusername@remotehost:remotefilespec localfilespec</span><br /><br />...to download from remotehost to your local machine, or:<br /><br /><span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">pscp localfilespec myusername@remotehost:remotefilespec</span><br /><br />...to upload to remotehost from your local machine.<br /><br />Type pscp with no arguments for a list of other parameters that can be supplied on the command line.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Tera_Term_ProTTSSH"><span style="color:#000000;">Tera Term Pro/TTSSH</span></a></b><br /><br />Another good terminal emulation package that supports SSH is Tera Term Pro. You can download it here: <a href="http://hp.vector.co.jp/authors/VA002416/teraterm.html">http://hp.vector.co.jp/authors/VA002416/teraterm.html</a><br /><br />Tera Term Pro comes with a regular Windows setup program. As with PuTTY, I'd recommend changing the terminal settings to suit your preferences. Save the setup as teraterm.ini to replace the defaults.<br /><br />By default, Tera Term Pro does not include SSH. For this, you need to download the SSH extension, TTSSH. The original download site was here: <a href="http://www.zip.com.au/%7Eroca/ttssh.html">http://www.zip.com.au/~roca/ttssh.html</a>, but this doesn't seem to be accessible any more. You can try the following mirror instead: <a href="http://public.planetmirror.com/pub/ttssh/">http://public.planetmirror.com/pub/ttssh/</a>, (the required file is ttssh154.zip).<br /><br />Unzip this file in the same directory that Tera Term Pro is installed in. It adds a program called ttssh.exe. Run this program in place of Tera Term Pro itself, (ttermpro.exe), and you should now have SSH available as an option during connect. Additionally, you will find that some SSH options are added to the Setup menu. You may wish to change the defaults so that SSH, (on port 22), is the default connection method.<br /><br />Tera Term Pro with TTSSH used to be the best free SSH implementation for Windows, but, in my opinion at least, recent versions of PuTTY have surpassed them for ease of use and improved functionality. You may wish to try both and draw your own conclusions.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Command_lineX_Windows_CygwinOpenSSH"><span style="color:#000000;">Command line/X Windows - Cygwin/OpenSSH</span></a></b><br /><br />If you're happiest working in a UNIX-like command-line environment, and are already familiar with the "ssh" and "scp" commands, you may want to try Cygwin, (<a href="http://www.cygwin.com/">http://www.cygwin.com/</a>). Cygwin is a pretty complete GNU based, (i.e. UNIX-like), environment for Windows.<br /><br />If you don't want/need the full Cygwin UNIX-like environment, OpenSSH for Windows, (<a href="http://sshwindows.sourceforge.net/">http://sshwindows.sourceforge.net/</a>), is basically a "cut-down" version of Cygwin which includes command line ssh/scp/sftp functionality. It also includes ssh/sftp servers, and it has a fairly straightforward Windows installation program. (Note that OpenSSH for Windows replaces the older, and no longer maintained, Network Simplicity OpenSSH server solution, <a href="http://www.networksimplicity.com/openssh/">http://www.networksimplicity.com/openssh/</a>.)<br /><br />If you do choose Cygwin, and you are comfortable working in the X-Windows environment, in addition to the rich selection of GNU command-line programs, (including sshd, ssh, scp and sftp), there are also Cygwin ports of XFree86, (<a href="http://cygwin.com/xfree/">http://cygwin.com/xfree/</a>), KDE, (<a href="http://kde-cygwin.sourceforge.net/">http://kde-cygwin.sourceforge.net/</a>), and Gnome, (<a href="http://cygnome.sourceforge.net/">http://cygnome.sourceforge.net/</a>). While this is no substitute for dual-booting with a good Linux distro or BSD release, it is handy for doing quick X-Windows work on a server without needing to reboot your Windows machine. I haven't used Cygwin's X-Windows tools extensively, but I have noticed that the quality and stability seems to be improving with each new release.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Known_Hosts"><span style="color:#000000;">Known Hosts</span></a></b><br /><br />The "known hosts" list contains the public keys of host machines that you "trust" when using ssh or scp. All implementations of ssh/scp, (including PuTTY, WinSCP and TTSSH), will give warnings if you are connecting to a new machine that is not in the list of known hosts. In addition, they will refuse to connect if the host's public key does not match the stored value.<br /><br />PuTTY stores the list of known hosts in the registry, under the key:<br /><br /><span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">HKEY_CURRENT_USER->Software->SimonTatham->PuTTY->SshHostKeys</span>.<br /><br />WinSCP also stores the known hosts in the registry, under the key:<br /><br /><span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">HKEY_CURRENT_USER->Software->Martin Prikryl->WinSCP 2->SshHostKeys</span>.<br /><br />Tera Term Pro stores its list of known hosts in the following file:<br /><br /><span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">C:\"Program Files"\ttermpro\ssh_known_hosts</span><br /><br />(assuming the program was installed in the default folder).<br /><br />If a host machine's public key is legitimately changed, (as part of an upgrade for example), you will need to remove the old key from the file, (or the registry in the case of PuTTY or WinSCP), to successfully connect to that host. You should, of course, be <b>100% sure</b> that the key was genuinely changed, and that this is not a hacker's machine masquerading as the host -- this is why these keys are stored and checked in the first place.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Automatic_login"><span style="color:#000000;">Automatic login</span></a></b><br /><br />With any of the installations covered so far we can only establish an SSH connection provided we supply a password each time. It is possible to automate SSH connections by generating "passphrase-less" secure keys and modifying our connection settings to use the new keys. In general, I would only recommend this procedure if you have a specific requirement for automating file transfers, and you clearly understand the security implications.<br /><br />Keys can be generated on the Windows client-side, using PuTTYGen, or on the server-side, using ssh-keygen. Either program will generate a public key file and a private key file. Different key file combinations can be generated for different SSH protocol versions. If you specify a blank "passphrase", then only the key files will be required to authenticate the connection, thereby allowing unattended connections. (If you include a passphrase you will benefit from a "doubly-secure" authentication, based on both the key and the passphrase.)<br /><br />To use the keys, save the key files in a <b>secure</b> location on the <b>client</b> machine. Then copy the contents of the <b>public</b> key file into the file <b><span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">$HOME/.ssh/authorized_keys</span></b>, (SSH protocol version 1), or <b><span style="font-family:Courier New, Courier, Arial, Helvetica, sans-serif;font-size:85%;">$HOME/.ssh/authorized_keys2</span></b>, (SSH protocol version 2), on the <b>server</b> machine. Finally, modify the PuTTY, WinSCP or TTSSH session details so that it uses the saved <b>private</b> key file, (making sure it also uses the correct protocol for the specified key). You will find the option to specify the Private Key File under the SSH settings.<br /><br />Assuming you've got the right files in the right places, and the correct session settings, when you attempt to connect you should find that you can connect without supplying a password or passphrase.<br /><br />If you are using Cygwin and the command-line ssh/scp versions, you can check the man pages for ssh to determine where you need to save your key files so that the ssh and scp commands will connect without requiring a password/passphrase.<br /><br />This "passphrase-less" approach is reasonably secure, provided access to the client machine is restricted. However, if someone manages to steal your private key file, (which might not be that difficult on most Windows machines), your server account will be fully compromised.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="SSH_Servers"><span style="color:#000000;">SSH Servers</span></a></b><br /><br />The following is a brief introduction to SSH server software for Windows.<br /><br />Cygwin, (<a href="http://www.cygwin.com/">http://www.cygwin.com</a>), includes a port of the OpenSSH server software. Cygwin is basically a GNU, (UNIX-like), subsystem that runs on 32-bit Windows. To get SSH support for Cygwin, install the "openssh" package in addition to the basic Cygwin packages.<br /><br />If you have any problems installing and configuring Cygwin "by hand", you could also try <a href="http://sshwindows.sourceforge.net/">http://sshwindows.sourceforge.net/</a>. This implementation basically wraps Cygwin's OpenSSH port in a package that includes a standard Windows installation program.<br /><br />If you have concerns about running open source secure services, such as OpenSSH or OpenVPN, on Windows, the main alternative is the IPSEC functionality that comes built-in to Windows 2000 servers. IPSEC is fully supported by Microsoft, and Windows-compatible implementations are also available for most *nix. This seems intuitively more secure, since it involves fewer components that are not integral parts of the Windows O/S.<br /><br />You could also use Windows Terminal Services. This has built-in encryption and has the advantage that it gives you GUI access. You might want to use this in conjunction with IPSEC if you have any concerns about the Terminal Services encryption algorithms.<br /><br />If you prefer open source solutions, it is still possible to get secure remote GUI access to a Windows server using freely available software. VNC, (<a href="http://www.realvnc.com/">http://www.realvnc.com/</a>), provides remote GUI access to Windows machines, and it uses predictable port assignment, so a VNC client session can be port-forwarded over an SSH link provided the Windows server is running an SSH server, (in addition to the VNC server).<br /><br />I believe it is also possible to port-forward a Terminal Services client session over SSH -- the server appears to use port 3389 by default according to the following Microsoft support article: <a href="http://support.microsoft.com/default.aspx?scid=kb;en-us;150543">http://support.microsoft.com/default.aspx?scid=kb;en-us;150543</a>.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Old_Versions_of_this_Document"><span style="color:#000000;">Old Versions of this Document</span></a></b><br /><br />The previous version of this document, (which you may find useful if you are using older versions of Windows SSH utilities), is available here: <a href="http://www.jfitz.com/tips/ssh_for_windows_doc_version2.html">http://www.jfitz.com/tips/ssh_for_windows_doc_version2.html</a>.<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Summary_of_Links_in_this_Document"><span style="color:#000000;">Summary of Links in this Document</span></a></b><br /></span><ul><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">OpenSSH Windows page: <a href="http://www.openssh.com/windows.html">http://www.openssh.com/windows.html</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">OpenSSH Mac page: <a href="http://www.openssh.com/macos.html">http://www.openssh.com/macos.html</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">OpenVPN, an open source, SSL VPN solution: <a href="http://openvpn.net/">http://openvpn.net</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">PuTTY, pscp and related tools: <a href="http://www.chiark.greenend.org.uk/%7Esgtatham/putty/">http://www.chiark.greenend.org.uk/~sgtatham/putty/</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Configuring PuTTY: <a href="http://www.jfitz.com/tips/putty_config.html">http://www.jfitz.com/tips/putty_config.html</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">WinSCP: <a href="http://winscp.net/eng/">http://winscp.net/eng/</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Tera Term Pro: <a href="http://hp.vector.co.jp/authors/VA002416/teraterm.html">http://hp.vector.co.jp/authors/VA002416/teraterm.html</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">TTSSH, (Tera Term Pro SSH extension): <a href="http://www.zip.com.au/%7Eroca/ttssh.html">http://www.zip.com.au/~roca/ttssh.html</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">TTSSH, (mirror site): <a href="http://public.planetmirror.com/pub/ttssh/">http://public.planetmirror.com/pub/ttssh/</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Cygwin: <a href="http://www.cygwin.com/">http://www.cygwin.com/</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">XFree86 for Cygwin: <a href="http://cygwin.com/xfree/">http://cygwin.com/xfree/</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">KDE for Cygwin: <a href="http://kde-cygwin.sourceforge.net/">http://kde-cygwin.sourceforge.net/</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Gnome for Cygwin: <a href="http://cygnome.sourceforge.net/">http://cygnome.sourceforge.net/</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">OpenSSH for Windows: <a href="http://sshwindows.sourceforge.net/">http://sshwindows.sourceforge.net/</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Microsoft's Terminal Services Advanced Client FAQs: <a href="http://www.microsoft.com/windows2000/techinfo/administration/terminal/tsacfaq.asp">http://www.microsoft.com/windows2000/techinfo/administration/terminal/tsacfaq.asp</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Microsoft Terminal Services TCP port support article: <a href="http://support.microsoft.com/default.aspx?scid=kb;en-us;150543">http://support.microsoft.com/default.aspx?scid=kb;en-us;150543</a></span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">VNC - Remote GUI access for Windows: <a href="http://www.realvnc.com/">http://www.realvnc.com/</a></span></li></ul><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> </span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Revision_History"><span style="color:#000000;">Revision History</span></a></b><br /><br />Version 1.0 22 June 1999 - Initial document<br />Version 2.0 22 January 2001 - Revised format, rewrote much of the content<br />Version 2.1 7 February 2001 - Added section on ssh servers<br />Version 2.2 16 November 2001 - Added notes on PuTTY port forwarding and Cygwin<br />Version 2.3 3 December 2001 - Removed link to commercial software<br />Version 2.4 6 December 2001 - Added section on WinSCP<br />Version 2.5 28 January 2002 - Added link to PuTTY configuration guide<br />Version 3.0 30 January 2002 - Rewrote much of the content<br />Version 3.1 6 March 2002 - Started Questions and Answers section, (ftp solution)<br />Version 3.2 27 March 2002 - Added psftp question/answer<br />Version 3.3 8 April 2002 - Added tunneling HTTP question/answer<br />Version 4.0 23 April 2002 - Revised format<br />Version 4.1 11 May 2002 - Expanded SSH server section, reintroduced Revision History<br />Version 4.1.1 15 May 2002 - Added links to Cygwin packages: XFree86, KDE and Gnome<br />Version 4.1.2 17 July 2003 - Updated WinSCP links<br />Version 4.2 25 May 2004 - Edited and updated links<br />Version 4.2.1 8 July 2004 - Mentions PuTTY's "-cleanup" option<br />Version 4.2.2 21 March 2005 - Updated WinSCP links<br />Version 4.2.3 29 March 2006 - Added link to OpenVPN<br /><br /></span><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr><tr><td bgcolor="#a0a0a0" height="1" width="100%"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="1" /></td></tr><tr><td height="2"><img src="http://www.jfitz.com/images/clear.gif" alt="" border="0" height="2" /></td></tr></tbody></table><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b><a name="Questions_and_Answers"><span style="color:#000000;">Questions and Answers</span></a></b><br /><br /><b>Question:</b><br /><i>Subject: FTP solution<br />I have a school PC running Windows NT terminal server on Citrix on Red Hat and Windows NT server 4.0 at home. I have next to no access at school, and can't install anything, but I can access HTTP (and SSH via a computer on the edge of the firewall). What I want to do is copy files from home, through the firewall, to and from school. Is this possible using a GUI? If not is it possible another way? How?</i><br /><br /><b>Answer:</b><br />You can use WinSCP to do exactly what you describe. It uses the SSH protocol to transfer files, and provides an "Explorer-like" GUI for dragging/dropping files.<br /><br />If you can SSH directly to the internal box through the firewall, then WinSCP will work "as-is".<br /><br />If not, then you need a little bit of port-forwarding to get where you want to go: </span><ul><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Log onto the firewall, (or the machine that is directly accessible via the firewall), using SSH, forwarding an arbitrary local port to port 22 on the internal machine.</span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">When you run WinSCP connect to "localhost" on the arbitrary port you selected.</span></li></ul><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> For example: </span><ul><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Say "schoolfirewall.myschool.edu" is your school's firewall, (or the machine that is visible outside the firewall).</span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">From "schoolfirewall.myschool.edu", you are trying to access a machine called mypc.myschool.edu", (which is not visible outside the firewall).</span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">You (randomly) pick port 2200 as the local port to forward.</span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Create, (and save), a PuTTY session that connects to "schoolfirewall.myschool.edu" using SSH, with Local Forwarding of port "2200" to "mypc.myschool.edu:22", (specified in the "SSH Tunnels" section).</span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Before running WinSCP, run PuTTY using the saved session, (this will connect to the firewall with port 2200 forwarded - you must leave PuTTY connected for the duration of the WinSCP session).</span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Run WinSCP. Connect to "localhost" on port 2200. This will in fact be connecting to the SSH server on port 22 of "mypc.myschool.edu".</span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">Provide the appropriate login details for "mypc.myschool.edu", and you should now be able to drag/drop files to/from the internal machine.</span></li></ul><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> Notes: </span><ul><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">"mypc.myschool.edu", (the internal server), needs to be running an SSH server. Red Hat should have that installed by default. You can also get a version for Windows if you need it, (see the section on <a href="http://www.jfitz.com/tips/ssh_for_windows.html#SSH_Servers">SSH Servers</a>).</span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">If Local Forwarding to "mypc.myschool.edu:22" doesn't work, try using the internal IP address of the machine instead - it's possible that your client or the firewall can't resolve the internal address. For example, if "mypc.myschool.edu" is at IP address "10.0.2.100", try forwarding to "10.0.2.100:22".</span></li></ul><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> <b>Question:</b><br /><i>Subject: psftp<br />Why not use psftp, (PuTTY's implementation of sftp)?</i><br /><br /><b>Answer:</b><br />Note: sftp is a "secure ftp" program. It works like a regular ftp client, but uses SSH to secure the traffic passing between the client and the server. As of version 3, WinSCP also supports sftp functionality.<br /><br />I have a couple of niggling concerns with sftp: </span><ul><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">sftp requires sftp-server on the server side. Some sites don't enable sftp-server, so it can be annoying to become overly dependant on sftp. All the functionality of sftp can be achieved with ssh/scp, without the need for new server-side programs.</span></li><li><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;">sftp adds a new layer of protocol, which introduces new possibilities for security vulnerabilities.</span></li></ul><span style="font-family:Verdana, Arial, Helvetica, sans-serif;font-size:85%;"> My preference is to stick with tools that are implemented using the "basic" SSH protocols, but I accept the point that sftp would also get the job done, and may be easier to work with in many situations.<br /><br /><b>Question:</b><br /><i>Subject: tunneling HTTP<br />Can I tunnel all my HTTP traffic through an SSH connection to a friend's Linux host?</i><br /><br /><b>Answer:</b><br />Tunneling only works from one host to another (single) host.<br /><br />To tunnel all HTTP requests, you would need to establish tunnels between your machine and every other machine on the internet. Tunneling HTTP requests to a handful of webservers behind a firewall is feasible, but directly securing all HTTP traffic is not.<br /><br />However, it is possible to tunnel HTTP requests indirectly:<br /><br />You could install a HTTP proxy on the Linux machine, (such as squid, <a href="http://www.squid-cache.org/">http://www.squid-cache.org/</a>), then tunnel communications with the proxy, (port 3128 by default for squid). On the client machine, the HTTP proxy would appear to be running on localhost:3128. It would in fact be talking to squid on port 3128 on the Linux box.<br /><br />This will secure traffic between your machine and the Linux box. However, the actual HTTP requests from squid to the rest of the Internet are not, (and cannot), be secured, (unless you somehow had SSH or SSL access to every other machine on the Internet). Given this fact, there's little to be gained from tunneling HTTP access through a proxy, unless you specifically want to encrypt the data between your machine and the proxy.</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-85879469361684119932008-04-09T19:18:00.000-07:002008-04-09T19:22:49.119-07:00"typical" PuTTY SSH client<span style=";font-family:Verdana,Arial,Helvetica,sans-serif;font-size:85%;" ><a href="http://www.jfitz.com/tips/putty_config.html">Source: http://www.jfitz.com/tips/putty_config.html</a><br /><br /><br /><img src="http://www.jfitz.com/tips/putty1.png" alt="Session Settings" /><br /><br /><img src="http://www.jfitz.com/tips/putty2.png" alt="Window Settings" /><br /><br /><img src="http://www.jfitz.com/tips/putty3.png" alt="Appearance Settings" /><br /><br /><img src="http://www.jfitz.com/tips/putty4.png" alt="Connection Settings" /><br /><br /><img src="http://www.jfitz.com/tips/putty5.png" alt="SSH Settings" /><br /><br /><img src="http://www.jfitz.com/tips/putty6.png" alt="Tunnel Settings" /><br /><br /><img src="http://www.jfitz.com/tips/putty7.png" alt="Save Session Settings" /><br /><br /><img src="http://www.jfitz.com/tips/putty8.png" alt="Create Shortcut" /><br /><br /><img src="http://www.jfitz.com/tips/putty9.png" alt="Modify Shortcut Target" /><br /><br /><img src="http://www.jfitz.com/tips/putty10.png" alt="Save Shortcut Changes" /><br /><br /><img src="http://www.jfitz.com/tips/putty11.png" alt="Configure Email Client" /><br /><br /><img src="http://www.jfitz.com/tips/putty12.png" alt="Select Account Properties" /><br /><br /><img src="http://www.jfitz.com/tips/putty13.png" alt="Set Incoming/Outgoing mail to localhost" /></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6930918428485106265.post-35969656229799637592008-04-09T19:05:00.000-07:002008-04-09T19:18:24.412-07:00How to create self-issued SSL digital certificat<p><a href="http://pigtail.net/nicholas/openssl.html"><span style="font-size:130%;">How to create your own "self-signed or self-issued" SSL digital certificate<br />(free, not a 90-day-free gimmick)</span></a></p> <p>If you run a secure web server (e.g. apache-ssl or apache 2 with SSL support ), you need to have a SSL certificate.<br />You need a SSL certificate if you want to run TLS/SSL enabled imap and pop3 servers such as University of Washington's imapd and ipopd</p> <p>This article shows you how to create a SSL certificate, using Open Source software. <b><br />It really is free</b>.<br />However, if you work for large corporations or governments, don't do something useful like this,<br />go buy a SSL certificate (and keep paying every year). Your job is to spend lots of money to make you look important.</p> <p>From a cryptographic perspective, there is no difference in security whether you use a self-signed<br />SSL certificate or buy an expensive SSL certificate. The only difference is perhaps cost and a convoluted sense of security:<br />after you spend money buying expensive SSL certificates from some unknown and self-declared "trust-worthy and reputable"<br />company, it must make you feel good. But back in your logical mind, you may be bothered by these haunting questions:<br />Have you ever met them ? Do you know where they are ? Do you know who they really are ? How long have you known them ?<br />Do you trust them more than you trust yourself ? Based on what evidence and logical thinking that make you think they are trust-worthy ?</p> <p> <img style="border: 0px solid ; width: 40px; height: 44px;" alt="" src="http://pigtail.net/nicholas/topiclinux.gif" align="middle" />Linux system:<br />( I assume your Linux box has openssl installed, most Linux systems have openssl installed )<br />( Note: 3650 days gives you approximately 10 years validity period; default 365 only gives you 1 year validity period )</p> <table id="table1" bg border="0" width="100%" style="color:#c0c0c0;"> <tbody> <tr> <td><b>openssl req -new -x509 -days 3650 -nodes -out apache.pem -keyout apache.pem</b></td> </tr> </tbody> </table> <p>You will be asked to answer a few questions:<br />Country Code is a <a href="http://www.iana.org/cctld/cctld-whois.htm"> 2-letter code</a> of the country your server is.<br />State or Province is a free-text field.<br />Locality is a free-text field, typically it is the city where the server is.<br />Organization Name is your company name, or use your own name.<br />Organization Unit is your department/division name, or enter your phone number, or the name of the machine.<br /><b>Common Name</b> is very important, it should be your web server's full external name such as <b>www.mydomain.com</b><br />Email address is optional, it is expected that you enter your real email address.</p> <p>The SSL certificate is now created as a file called <b>apache.pem</b>, copy this file to the directory where your apache web server needs it.<br />Restart the Apache server using /etc/init.d/apache force-reload</p> <p>For Apache on Linux, you also need a symlink to the hash value of <span style="font-weight: bold;">apache.pem</span> Find out the hash value of apache.pem<br /></p> <table id="table1" bgcolor="#c0c0c0" border="0" width="100%"> <tbody> <tr> <td><b>openssl x509 -hash -noout <></b></td> </tr> </tbody> </table><br />Create a symlink<br /><table id="table1" bgcolor="#c0c0c0" border="0" width="100%"> <tbody> <tr> <td><span style="font-weight: bold;">ln -sf apache.pem <hashvalue>.0</hashvalue></span></td> </tr> </tbody> </table><br />For Apache in Linux, change the file attribute<br /><table id="table1" bgcolor="#c0c0c0" border="0" width="100%"> <tbody> <tr> <td><b>chmod 600 apache.pem</b></td> </tr> </tbody> </table><br /><hr /> <p> <img style="border: 0px solid ; width: 28px; height: 27px;" alt="" src="http://pigtail.net/nicholas/windows.png" align="middle" /> Windows system: </p> <p>Download openssl-xxx-bin.exe from <a href="http://sourceforge.net/projects/gnuwin32/"> http://sourceforge.net/projects/gnuwin32/</a> </p> <p>Double click openssl-xxx-bin.exe and install files to <b> c:\temp</b></p> <p>Move or copy the .exe and .DLL files from c:\temp\GnuWin32\bin to <b>c:\temp</b></p> <p>Right click <a href="http://pigtail.net/nicholas/openssl.config">on this link</a> and choose Save; save the configuration file to <b>c:\temp</b> </p> <p>Pop a CMD screen (click Start, Run, <span style="color: rgb(128, 0, 0);"><b>cmd</b> </span>), a black screen comes up, type <span style="color: rgb(128, 0, 0);"><b>cd c:\temp</b></span></p> <table id="table2" bgcolor="#c0c0c0" border="0" width="100%"> <tbody> <tr> <td><b>openssl req -config openssl.config -new -x509 -nodes -out apahce.pem -keyout apache.pem -days 3650</b></td> </tr> </tbody> </table> <p>The certificate is now generated as a file called <b>apache.pem</b>, move this file to the place where your secure server needs it.</p> <p>If you need a PKCS#12 SSL certificate, you can convert the apache.pem SSL certificate to PKCS#12 format.</p> <table id="table3" bgcolor="#c0c0c0" border="0" width="100%"> <tbody> <tr> <td><b>openssl pkcs12 -export -in apache.pem -out mycert.pfx -name "My Digital Certificate"</b></td> </tr> </tbody> </table> <p>Your PKCS#12 SSL certificate is now converted to a file called <b>mycert.pfx</b></p> <p> </p> <p>Reference: <a href="http://www.openssl.org/">www.openssl.org</a> </p>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6930918428485106265.post-16358390083853464052008-04-09T18:32:00.000-07:002008-04-09T18:34:02.641-07:00how to use public key authentication<a href="http://pigtail.net/LRP/printsrv/no-passwd.html">Source: http://pigtail.net/LRP/printsrv/no-passwd.html</a><br />Sometimes it is desirable to establish a ssh connection (e.g. machine to machine) without (human typing) a password at the console. <p> <img style="width: 497px; height: 204px;" alt="g" src="http://pigtail.net/LRP/printsrv/no-passwd.png" /></p> <p>For example, to automatically back up files from "Office" (using scp or rsync encrypted with ssh) to a Linux server at a remote data center.<br />(many business are required to have "off site" back-up of important data files in case of theft or fire damages). </p> <p>At the office computer, pop a cygwin <img style="width: 38px; height: 32px;" alt="g" src="http://pigtail.net/LRP/printsrv/cygwin.png" align="middle" /> windows, generate a <span style="font-weight: bold; color: rgb(200, 29, 204);">Private key</span> and a corresponding <span style="font-weight: bold; color: rgb(102, 0, 0);">Public key</span>, (geeks call this a key pair)<br />when asked for location to store keys, just hit <span style="font-weight: bold; color: rgb(102, 0, 0);">Enter</span>, when asked for passphrase, just hit <span style="font-weight: bold; color: rgb(102, 0, 0);">Enter</span>.<br />The keys are now stored in a so called "hidden" directory at "<span style="font-weight: bold; color: rgb(102, 0, 0);">~/.ssh</span>"<br />The actual directory, if you install cygwin package in c:\cygwin, is c:\cygwin\home\currently_login-user\.ssh </p> <table style="text-align: left; width: 100%;" border="0" cellpadding="2" cellspacing="2"> <tbody> <tr> <td style="width: 100%; background-color: rgb(255, 255, 102);"><span style="font-weight: bold;">ssh-keygen -t rsa<br />cd ~/.ssh<br />dir </span></td> </tr> </tbody> </table> You will see some files,<span style="font-weight: bold;"> id_rsa</span> is your <span style="color: rgb(204, 51, 204); font-weight: bold;">private key</span>, <span style="font-weight: bold;">id_rsa.pub</span> is your <span style="color: rgb(102, 0, 0); font-weight: bold;">public key</span> <p>At the remote data center Linux serve, (assuming you have an account called "john"), create a .ssh directory.<br />If the .ssh directory already exists, it will give an error message "cannot create directory", that is OK. </p> <table style="text-align: left; width: 100%;" border="0" cellpadding="2" cellspacing="2"> <tbody> <tr> <td style="width: 100%; background-color: rgb(192, 192, 192);"><span style="font-weight: bold;">login john<br />mkdir .ssh<br />exit</span></td> </tr> </tbody> </table> <p>At the office computer, pop a cygwin <img style="width: 38px; height: 32px;" alt="g" src="http://pigtail.net/LRP/printsrv/cygwin.png" align="middle" /> windows, copy your <span style="font-weight: bold; color: rgb(102, 0, 0);">Public Key</span> to the remote Linux server</p> <table style="text-align: left; width: 100%;" border="0" cellpadding="2" cellspacing="2"> <tbody> <tr> <td style="width: 100%; background-color: rgb(255, 255, 102);"><span style="font-weight: bold;">cd ~/.ssh<br />scp id_rsa.pub john@remote_linux_server_ip_address:~/.ssh/newkey</span></td> </tr> </tbody> </table> <p>At the remote data centre Linux server, add (technically speaking, append) the office's <span style="font-weight: bold; color: rgb(102, 0, 0);">Public Key</span> to a special key file called "authorized_keys"</p> <table style="text-align: left; width: 100%;" border="0" cellpadding="2" cellspacing="2"> <tbody> <tr> <td style="width: 100%; background-color: rgb(192, 192, 192);"><span style="font-weight: bold;">login john<br />cd ~/.ssh<br />cat newkey<br />cat newkey >> authorized_keys<br />rm newkey<br />chmod 600 authorized_keys<br />exit</span></td> </tr> </tbody> </table> <p>At the office computer, pop a Cygwin <img style="width: 38px; height: 32px;" alt="g" src="http://pigtail.net/LRP/printsrv/cygwin.png" align="middle" /> windows,<br />ssh to the remote Data Center's Linux server, it should not ask for a password anymore.</p> <table style="text-align: left; width: 100%;" border="0" cellpadding="2" cellspacing="2"> <tbody> <tr> <td style="width: 100%; background-color: rgb(255, 255, 102);"><span style="font-weight: bold;">ssh john@remote_linux_server_ip_address</span></td> </tr> </tbody> </table> <br />As usual with any public key authentication scheme, you have to be careful not to leak out the Private key (i.e., ~/.ssh/id_rsa )<br />Geeks call this method of allowing access "<span style="font-weight: bold;">authentication using public keys</span>".<br /><br /><a href="http://pigtail.net/LRP/printsrv/cygwin-sshd.html">See this page</a> on how to install openssh-cygwin on Windows 2000 and Windows <img style="width: 22px; height: 17px;" alt="" src="http://pigtail.net/LRP/printsrv/windows.png" /> XP. <p>It is also possible to use this method using Putty (ssh client) instead of Cygwin ssh client.<br />You need to generate/import/export the keys between Putty and sshd. Use PUTTYGEN to do that.</p> <p>To disable password login completely:<br />edit /etc/ssh/sshd_config<br />ChallengeResponseAuthentication no<br />PasswordAuthentication no<br />UsePAM no</p>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6930918428485106265.post-53092179832354369842008-04-09T18:22:00.000-07:002008-04-09T18:23:28.936-07:00How to print to a remote printer securely using a ssh tunnel<p><a href="http://pigtail.net/LRP/printsrv/tunnel-how.html"><b><span style="color:#660000;">How to print to a remote printer securely using a ssh tunnel</span></b></a></p> <p><img src="http://pigtail.net/LRP/printsrv/tunnel-how.png" alt="g" border="0" height="267" width="479" /> </p> <p>On the "head office" side, you have 3 choices to set up a SSH server:<br />(1) [Legacy] set up a LRP firewall with hard disk option and install sshd or<br />(2) Setup a Linux<img src="http://pigtail.net/LRP/printsrv/tux.png" alt="g" align="middle" border="0" height="23" width="20" /> box on the internal network with an openSSH server (called sshd), or<br />(3) Setup a Windows <img src="http://pigtail.net/LRP/printsrv/windows.png" alt="g" align="middle" border="0" height="17" width="22" /> XP box on the internal network, install cygwin-openssh and install <a href="http://pigtail.net/LRP/printsrv/cygwin-sshd.html">sshd as a Windows service</a>.</p> <p>If you choose (2) or (3), set your router or firewall to port forward tcp port 22 (ssh traffic) to the<br />internal ssh server's IP address.</p> <p><b><a name="cygwin"></a></b>On the "road warrior" laptop, install <a href="http://pigtail.net/LRP/printsrv/cygwin-ssh.html">cygwin and ssh</a> or <a href="http://www.chiark.greenend.org.uk/%7esgtatham/putty/" target="_blank">putty</a></p> <p>Next, add a "special" printer that prints to IP=127.0.0.1 , port=9100</p> <table border="2" width="606"> <tbody> <tr> <td width="606"> <p>Please have the <span style="color:#660000;"><b>appropriate "printer driver" ready</b></span>, either on a floppy, or on a hard disk, or on a network shared drive. If your printer is not included in Windows' default list, use the "Have Disk" method.</p> <p> <img src="http://pigtail.net/LRP/windows.png" alt="g" align="middle" border="0" height="17" width="22" /> <b>Windows XP</b>:<br />Click Start.. (Control Panel).. Printers and Faxes...Add Printer...Local Printer...(check off Auto Detect PnP)...Next....Create new port...Standard TCP/IP port...<br />IP address =<b> 127.0.0.1</b>, port name=PrintSrv, Custom, Settings.. Raw.. Port 9100</p> <p> <img src="http://pigtail.net/LRP/windows.png" alt="g" align="middle" border="0" height="17" width="22" /> <b>Windows 2000</b> (method A)<br />First, install <a href="http://pigtail.net/LRP/printsrv/install-axis.html">AXIS Print</a> Monitor.<br />Click Start...Settings, Printers, Add Printers, Local Printer, Create New Port,<br />Choose <b>AXIS</b> port, choose <b>RAW</b> TCP/IP port,<br />IP address =<b> 127.0.0.1</b>, port number =<b> 9100</b></p> <p> <img src="http://pigtail.net/LRP/windows.png" alt="g" align="middle" border="0" height="17" width="22" /> <b>Windows 2000</b> (method B)<br />Click Start...Settings, Printers, Add Printers, Local Printer, Create New Port,<br />Choose Standard TCP/IP port,<br />IP address =<b> 127.0.0.1</b>, port name=PrintSrv, Custom, Settings.. Raw.. Port 9100</p> <p> <img src="http://pigtail.net/LRP/windows.png" alt="g" align="middle" border="0" height="17" width="22" /> <b>Windows 2000 quirks</b><br />Sometimes the printer driver becomes corrupted, if that happens, delete the printer driver by re-installing the same printer driver on the LPT1 (local) port. Then repeat method A or method B above. Thanks to Hillie Sample of Ottawa, Canada for this tips.</p> <p> <img src="http://pigtail.net/LRP/win98-tiny.png" alt="g" align="middle" border="0" height="14" width="20" /> <b>Windows NT</b>:<br />First, install <a href="http://pigtail.net/LRP/printsrv/install-axis.html">AXIS Print</a> Monitor.<br />Start...Settings, Printers, Add Printers, "My Computer", Add Port,<br />Choose <b>AXIS</b> port, choose <b>RAW</b> TCP/IP port, click OK,<br />IP address =<b> 127.0.0.1</b>, port number =<b> 9100</b></p> <p><a name="win95"></a> <img src="http://pigtail.net/LRP/win98-tiny.png" alt="g" align="middle" border="0" height="14" width="20" /> <b>Windows 98/ME</b>:<br />First, install <a href="http://pigtail.net/LRP/printsrv/install-axis.html">AXIS Print</a> Monitor.<br />Click Start...Settings, Printers, Add Printers, Local Printer, choose AXIS port<br />(Windows 98 will ask for the Windows 98 CD, unless c:\windows\options\cabs is intact).<br />Right click on the printer icon, choose Properties, Details, Add Port, click Other,<br />choose <b>AXIS port</b>, click OK, choose <b>RAW</b> (TCP/IP), click OK,<br />IP address =<b> 127.0.0.1</b>, port number =<b> 9100</b>, click Apply, click OK</p> <p> <img src="http://pigtail.net/LRP/win98-tiny.png" alt="g" align="middle" border="0" height="14" width="20" /> <b>Windows 95</b><br />First, install <a href="http://pigtail.net/LRP/printsrv/install-axis.html">AXIS Print</a> Monitor.<br />Click Start...Settings...Printers Add Printers ... Local Printer .. install the printer driver on LPT1.<br />(Windows 95 will ask for the Windows 95 CD, unless c:\windows\options\cabs is intact)<br />Right click the Printer Icon, choose Properties, Details...Add Port, click Other,<br />choose <b>AXIS port</b>, click OK, choose <b>RAW</b> (TCP/IP), click OK,<br />IP address =<b> 127.0.0.1</b>, port number =<b> 9100<br /> </b>Click OK until you are back to the correct Printer Icon, right click, Properties, Details,<br />At the "Print to the Following Port" drop-down box, choose<br />"127.0.0.1_9100 AXIS Port", click OK.<br /> </p> </td> </tr> </tbody> </table> <p><b><span style="color: rgb(204, 0, 0);"><span style="font-weight: bold;"></span></span></b><span style="font-weight: bold; color: rgb(204, 0, 0);">If you have Windows XP SP2, you need to tweak the "windows firewall" to open</span><br /> <span style="font-weight: bold; color: rgb(204, 0, 0);">TCP Port 22.</span><br />Click Start...Control Panel...(in Category View mode)...Security Centre...Windows Firewall<br />...Exceptions Tab...Add port....port name ssh, port 22, TCP<br /></p> <p>Thanks to Stefano of Sardegna, Italy for the XP-SP2 reminder. <img alt="" src="http://pigtail.net/LRP/flags/it.png" style="width: 14px; height: 14px;" /></p> <p><b>When the "road warrior" or "branch office" is ready to print to the remote printer:<br /></b>Double click the cygwin <img src="http://pigtail.net/LRP/printsrv/cygwin.png" alt="g" align="middle" border="0" height="32" width="38" /> icon, a black screen pops up, type<br /><span style="color:#660000;"><b>ssh root@myoffice.com -L 9100:192.168.1.252:9100</b> </span> (log on to LRP box as root)<span style="color:#800000;"><b><br /></b></span>or<span style="color:#800000;"><b><br /></b></span><span style="color:#660000;"><b> ssh user@myoffice.com -L 9100:192.168.1.252:9100</b></span> (log on to Linux box or Windows box as user)</p> <p>Above command invokes ssh to logon to the head office's ssh server and creates a "tunnel".<br />(assuming you setup a DNS to point myoffice.com to the IP address of the LRP)<br />The -L option says "tunnel port 9100 traffic on the road warrior's computer to the remote side,<br />send the road warrior's TCP traffic to port 9100 of a computer with IP address=192.168.1.252 on the remote side".</p> <p>Printer traffic on port 9100 is now encrypted (tunneled) by the ssh protocol. <img src="http://pigtail.net/LRP/printsrv/smile.png" alt="g" align="middle" border="0" height="15" width="15" /><br /><b><span style="color:#cc0033;">Now the road warrior can print confidential documents</span></b> to the remote printer safely and securely. <img src="http://pigtail.net/LRP/printsrv/smile.png" alt="g" align="middle" border="0" height="15" width="15" /> </p> <p>If the "road warrior" hates <img src="http://pigtail.net/LRP/printsrv/no-smile.png" alt="g" border="0" height="15" width="15" /> typing long commands, create a <img src="http://pigtail.net/LRP/printsrv/batch-icon.png" alt="g" align="middle" border="0" height="30" width="30" /> <b>Windows batch file</b><br />that has the following content. Double clicking on the batch file icon will do all the typing <img src="http://pigtail.net/LRP/printsrv/smile.png" alt="g" align="middle" border="0" height="15" width="15" /></p> <table border="1" cellpadding="0" cellspacing="0" width="550"> <tbody> <tr> <td><b><span style="font-family:Arial;font-size:85%;color:#660000;">@echo off<br />c:<br />cd c:\cygwin\bin<br />bash --login -c "ssh -l root myoffice.com -L 9100:192.168.1.252:9100"</span></b></td> </tr> </tbody> </table> <p>(you can also use an ip address instead of <i>myoffice.com</i>)</p> <p>If the "road warrior" also hates typing password, set up authentication using <a target="_blank" href="http://www.rsasecurity.com/">RSA</a> public key and<br />private key, so that the "road warrior" does not have to type password to login to the sshd server.<br /><img src="http://pigtail.net/LRP/printsrv/caution-small.png" alt="g" align="middle" border="0" height="14" width="64" /> Obviously this is a <a href="javascript:alert('If the road warrior laptop is stolen, the private key is in it!');">security risk</a>, you have to balance between safety and convenience.<br />Here is <a href="http://pigtail.net/LRP/printsrv/keygen.html">how</a> to set up RSA public key and private key authentication so that password is not required.</p> <p><img src="http://pigtail.net/LRP/printsrv/fancy-bar.png" alt="g" border="0" height="5" width="400" /></p> <p>You can use the same technique to securely access other resources behind a firewall.</p> <p><a name="secure-email"></a>For example, <b>if the road warrior or branch office wants to do secure email</b>,<br />setup a ssh session to the LRP box or Linux box or Windows box in the internal network,<br />tunnel port <b>110</b> to the other side, for POP3 mail (for most email users)<br />tunnel port <b>143</b> to the other side, for IMAP mail (only needed by advanced email users)<br />tunnel port <b>25</b> to the other side, for SMTP outbound traffic (to send email via the Linux box)<br />In this case, configure the road warriors email client software to use<br />POP3 server = 127.0.0.1<br />IMAP server = 127.0.0.1<br />SMTP server = 127.0.0.1<br />Create a <a href="http://pigtail.net/LRP/printsrv/sample-batch.html">batch file like this example</a> and a desktop icon.<br />Double click the batch file icon to launch the ssh session and setup tunnels.</p> <p><img src="http://pigtail.net/LRP/printsrv/inhouse-pop.png" alt="g" border="0" height="267" width="479" /></p> <p>If you use LRP as the firewall, you may want to adjust LRP Seawall's timeout to lengthen<br />the ssh session timeout. <a href="http://pigtail.net/LRP/printsrv/ipmasq-timeout.html">See this page</a> on how to adjust Seawall's ssh session timeout.</p> <p><b>Design footnote:</b><br />In the above system design, the corporate firewall only opens <a href="javascript:alert('port 25 is used by in-house email server to send and receive emails to and from other email servers');"> port 25</a> and <a href="javascript:alert('port 22 is used for ssh, secure shell');">port 22</a>.<br />This is a better design than exposing port 110 or port 143 of the in-house email server to the outside world.<br />Personnel behind the firewall can use port 110 or port 143 to read their emails.<br />Personnel working from home or on the road must come in via SSH and set up tunnels<br />to read and send corporate emails under the protection of SSH's strong encryption.</p> <p>The world is moving away from plain text protocols by hardening them with TLS or SSL:<br />newer versions of POP3 has TLS support at port 110, and SSL support at port 995<br />newer versions of IMAP has TLS support at port 143, and SSL support at port 993<br />newer versions of SMTP has TLS support at port 25<br /><span style="color: rgb(153, 153, 153);">a version of "</span><span style="font-weight: bold; color: rgb(153, 153, 153);">smtps</span><span style="color: rgb(153, 153, 153);">" uses port 465 with SLL support, now becomes legacy</span><br />newer versions of telnet has SSL support at port 992</p> <p>SSH uses pure tcp protocol, it does not have the nasty firewall traversal problems like IPSec or PPTP.<br />SSH with tunneling has become the foundation technology of next generation VPN designs.<br />This next generation VPN is called SSL VPN.<br />One excellent SSL VPN implementation is <a href="http://www.openvpn.net/">OpenVPN</a>.</p>Unknownnoreply@blogger.com1