Jakarta Commons Online Bookshelf


JakartaCommons ONLINE BOOKSHELF Vikram Goyal MANNING © Copyright 2005 Manning Publications Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu i preface Welcome to the wonderfully eclectic world of Jakarta Commons, where everything is always changing and the only constant is change itself. If you’ve never heard of Jakarta Commons, you’re in for a treat; and if you’ve heard of it, then you’re in for some surprises. Jakarta Commons, or Commons for short, bypasses technologies and architectures, provides solutions to common problems, and helps make the work of technical architects and software developers easier. It’s a unique collection of components that are dissimilar in what they do but are united by the common goal of providing solutions for everyday tasks. This preface provides an insight into the Commons world. It describes the history of this suite of components and discusses its organization, rationale, and charter. This preface also gives you an idea of what to expect in this book’s modules. Let’s start at the beginning. In the beginning In the beginning, there was chaos. This chaos resulted from disparate systems creating similar utility modules to solve common problems. The result was multiple copies of multiple utilities thriving in several systems, performing basically the same functions. The problem was compounded when these diverse systems began modifying shared code, resulting in out-of-sync utilities that resembled new utilities rather than a common shared code base. On March 19, 2001, the first attempt was made to create order out of this chaos. That day, the Jakarta Project Management Committee (PMC) officially adapted a charter1 that mandated the creation of a subproject under the Apache/Jakarta umbrella called Commons. The PMC wanted to kill two birds with one stone by creating this subproject: ƒ Promote package-driven development for individual server-side utility components. ƒ Provide a playing ground for all Apache Jakarta committers (developers who have permissions to modify Jakarta code base) to test new ideas, experiment with frameworks, and gauge the feasibility of moving existing utility components to Commons. Toward this end, the charter divided the Commons project into two logical and physical entities: the Sandbox and the Commons Proper. The Sandbox is a test pit that serves the charter’s second purpose, and the Commons Proper is the repository of reusable package-driven components. The Sandbox is usually full of components that are in a high state of flux, but it doesn’t contain any actual releases. These components stay in the Sandbox until they’re ready to be released in the Commons Proper, usually via a voting mechanism among the Commons committers. But we’re getting ahead of ourselves. The initial code base for the Commons subproject came from existing projects in Jakarta and some donated packages. These included Database Pools (now in Pool and DBCP; see module 9), Collections (module 7), Cactus (no longer a Commons project, but a separate Jakarta subproject), HttpClient (module 1), Digester (module 4), Logging (module 14), and BeanUtils (module 8). 1 This Charter was revised on May 19, 2004. Licensed to Tricia Fu ii JAKARTA COMMONS ONLINE BOOKSHELF The PMC, which was only trying to bring together a few common utility components with this charter, was in for a big surprise: Commons became hugely popular with the open-source Java community and soon developed into the central point for companies to donate package-based Java components and for committers to share ideas. A lot has changed since that initial charter. Jakarta Commons is now host to 33 (as of February 2005) released components in the Commons Proper. The Sandbox is host to more than 20 components in various stages of development. Several components, like Cactus and HttpClient2 have grown in stature and are now recognized as separate subprojects within the Jakarta framework. A good measure of the popularity of a technology is the number of books written about it. At last count, there were six books, including this one, in various stages of development on Jakarta Commons. Clearly, Jakarta Commons has arrived and is important enough that all software professionals should understand it. What’s in it for me? Are you a technical architect? a software developer/programmer/analyst? a software project manager? If you answered “Yes” to any of these questions, Jakarta Commons means a lot to you. I’m a software programmer/developer/analyst Well, good for you! In addition to being in a role that puts you in direct contact with everything the software world has to offer, you’re also directly responsible for making or breaking the future of your company. (No pressure!) If only you could lay your hands on something that would take the grunt work out of everyday tasks. Suppose you’re writing a routine in Java to parse a configuration file to create an application’s domain objects. Based on the configuration file architecture, you decide to compose the routine so that it handles additions to or deletions from the array of objects you expect (based on those pesky marketing folks’ suggestions). You’re happy with yourself for having written a brilliant routine that is future-proof, and you look forward to your next task. And then you’re fired. Too bad! Being the brilliant programmer you are, you have no trouble finding another job. But what’s this? A new day, a new job, but you find yourself doing the same thing as before! Your first task is to write a configuration parser for the new company’s applications. If only you could have sneaked away with the copyrighted code from your previous company. Ethical behavior prevented you from doing so, right? The point of this scenario is to illustrate the fact that as software developers, we encounter tasks in our programming life cycles that are repetitive and nonunique. Around the world, software developers encounter the same problems and do exactly what each of us does. We repeat solutions instead of converging on one shared and proven technology. Sharing saves time and alleviates common mistakes. Of course, some of us (actually, a lot of us, considering the last count of the Commons mailing list) have started to see the light—the light at the end of the Jakarta Commons tunnel. The tunnel is filled with solutions to everyday, repetitive, proven tasks, with a worldwide community of developers as a support base. Can you afford not to look at the light? I’m a technical architect That means you already sort of know about Jakarta Commons. At least, you already know about the open- source solutions provided for common architectural problems from the Apache stable. Jakarta Commons must already be in your periphery. 2 HttpClient was only recently granted separate status; for all practical purposes, it’s still a Commons component. Licensed to Tricia Fu PREFACE iii Even if it isn’t, it’s time that you looked at it seriously. You’re responsible for creating the blueprints of software applications and creating robust, maintainable architectures; this requires knowledge of the best possible solutions to common architectural problems. Jakarta Commons offers a smorgasbord of technologies that are relevant and indispensable once you realize their benefits. For example, consider that as part of your architecting duties, you’re supposed to provide solutions to consistently and effectively write detailed logs from an application. As an architect, you want a generalized approach to such a problem rather than narrowing the focus by suggesting exact logging techniques and software components (and earning the wrath of software developers for encroaching on their territory). But how do you do that? Do you suggest the creation of an in-house Logging API and expand previous resources? Do you take a leap of faith and suggest exact external solutions—for example, by espousing the use of one product over another? Maybe. But there is a better way. You could, instead, suggest the use of the Jakarta Commons Logging component (see module 14). This component doesn’t tie your application to a single logging architecture; instead, it lets you use multiple implementations via a single interface. It’s a wrapper around other logging components, using a simple, intuitive, well-defined access mechanism. Not only does this solution endear you to your clients (perhaps because you used an open-source solution, thus saving money; or because you saved time in delivering a solution), it also causes you to be well liked by the software developers, because they’re free to choose the right logging algorithm. Commons is full of such components. Happy exploring in this book’s table of contents! I’m a project manager I’ve been meaning to talk to you about the raise I was promised! On a more serious note, you’re responsible for delivering cost-effective and timely solutions to your clients (whoever they may be, internal or external to your organization) and managing the day-to-day activities of application development. One of today’s hot topics, open-source solutions, is your gateway to cutting costs and still delivering effective solutions. Although this isn’t a book about the virtues of open-source, this is a book about open-source technologies. Specifically, this book is about the Jakarta Commons suite of open-source components. If you’ve already started using open-source in your organization, you already know the benefits it provides. If you haven’t, then this book can serve as an introduction to several of these components at once. Each of the Jakarta Commons components can be a gentle introduction to the world of open-source for you, and this book is the best medium to learning about them. Licensed to Tricia Fu iv JAKARTA COMMONS ONLINE BOOKSHELF acknowledgments Writing a book isn’t an easy task. When I first proposed the schedule for this book to Ben Sullins, my acquisitions editor, he wisely pointed out that my estimates for completing the manuscripts were grossly underestimated. I replied with false bravado that I was going to be very focused on writing this book, and I wanted to stick to my schedule. Yeah, right! One and a half years later (more than a year over my initial estimate), I’m writing these acknowledgements. During this period, I have changed houses three times, changed my day job once, survived two computer crashes, gone on an overseas trip, and remembered innumerable birthdays and anniversaries. Throughout, my wife—my friend—stood by me. She encouraged me to continue when I felt I couldn’t finish the book, and she threatened me with dire consequences when I said I would quit writing. She complained less about the time we couldn’t spend together, and she tried to make most of the time that we did spend together. She stood by me all this time and didn’t carry out the threat to divorce me. I proudly dedicate this book to her. Several other people helped me with the book-writing process. First, Marjan Bace had the vision to agree to publish a book on a not-so-hot topic from a not-so-known author. Ben Sullins, my acquisitions editor, gets a special nod for helping me settle into the writing process. Bruce Byfield was my initial development editor and provided some valuable insight into my first review chapter. Jackie Carter took the initial work done by Bruce and guided me through the rest of the chapters with considerable ease. She provided lots of input, and I hope I was able to apply most of them in my writing. David Roberson started the review process for the book, and Karen Tegtmeyer took it to completion; both of them did a fine job of organizing the best possible group of people to review the book. Karen did an exceptional job in the last few weeks of the book’s production by organizing and collating all the information from the reviewers. She further helped by providing essential copy for the book’s web site. Many thanks to all the reviewers: Jack Herrington, Oliver Zeigermann, Yoav Shapira, Glen Smith, Doug Warren, John Keyes, Dirk Verbeeck, Robert Burrell Donkin, Henri Yandell, Cos DiFazio, and Wahid Sadik. This book has benefited many times over because of your input. Doug Warren also agreed to be the technical proofreader for the book and was exceptionally thorough. In the production stage, Mary Piergies, my project editor, made sure every little detail was on track and that all the necessary people were on board. Leslie Haimes provided the cover design and spoilt me for choice. At the moment, I’m working with Helen Trimes in promoting this book, and I’m confident in her ability to provide the maximum possible coverage. Finally, I would like to thank Tiffany Taylor, who, as a copy editor, made an infinite number of corrections to my manuscript. I didn’t realize that in spite of my best efforts to say things the correct way, there was always a better way, and I’m thankful to Tiffany for making me realize this. I will endeavor to make fewer mistakes in my next book. Licensed to Tricia Fu ABOUT THIS BOOK v about this book Jakarta Commons Online Bookshelf provides detailed technical information about 18 components from Jakarta Commons Proper and 1 component from the Commons Sandbox (Chain, module 13). Modules, which can be purchased individually from the Manning web site at www.manning.com/goyal, are organized roughly according to related concepts, as follows: ƒ Web related—HttpClient (module 1); FileUpload (module 2); Net (module 3) ƒ XML related—Digester (module 4); JXPath and Betwixt (module 5) ƒ Packages—Validator (module 6); Collections (module 7); BeanUtils and Lang (module 8) ƒ Utilities—Pool and DBCP (module 9); Codec (module 10); Modeler (module 11); CLI (module 12) ƒ Frameworks—Chain (module 13); Logging and Discovery (module 14) To cater to this book’s various audiences, each module is organized in a similar format; each module is divided into sections with their own area of relevance. You can skip sections, depending on whether you’re reading from a software developer’s point of view or that of a technical architect. However, the boundaries between the sections aren’t hard and fast, and there is nothing stopping you from reading a module from start to finish. The structure of each module is as follows: 1. Introduce a component. 2. Develop the background for the component’s technology. 3. Explain the technology with examples, if necessary. 4. Explain the component’s structure. 5. Show beginning to intermediate (to advanced, in some cases) examples of using the component. If you’re a software developer, and you aren’t interested in the background technology of a component, you can skip the beginning of each module and jump to later sections. On the other hand, if you want detailed information about the technology behind each component—for example, if you want to know about a component’s background protocols and rationale—you may want to read the first part. To make the best use of this book, I suggest that you do read the information about the background technologies before you try the examples. This is especially important for components that involve varied and obscure technologies, such as Modeler (module 11). Knowing how Dynamic MBeans work makes the task of understanding the Modeler component much easier. Licensed to Tricia Fu vi JAKARTA COMMONS ONLINE BOOKSHELF What’s covered? As I said before, this book provides detailed examples and thorough background information about 18 Commons components. This section gives you a bird’s-eye view of what is covered in each module: ƒ Module 1, HttpClient—A component that helps you deal with the client side of HTTP. ƒ Module 2, FileUpload—The server end of the processing required for uploading files. ƒ Module 3, Net—Provides implementations for a diverse range of Internet protocols like FTP, SMTP, NNTP, and so on. ƒ Module 4, Digester—An XML-to-Java object mapping technology to help parse XML configuration files. ƒ Module 5, JXPath and Betwixt—JXPath traverses complex objects using the XPath syntax, whereas Betwixt is a Java-XML mapping tool. ƒ Module 6, Validator—Provides a guideline and API for validating user data from any source. ƒ Module 7, Collections—Leverages the Java Collection API by providing several new collection classes and enhancements for existing ones. ƒ Module 8, BeanUtils and Lang—BeanUtils works with the Java Reflection and Introspection API to provide access to JavaBeans properties, whereas Lang enhances the classes of the java.lang API. ƒ Module 9, Pool and DBCP—Pool is an object pooling architecture, whereas DBCP is a precise implementation that uses it to manage database connections. ƒ Module 10, Codec—Provides several common (and some not-so-common) encoding and decoding routines. ƒ Module 11, Modeler—Provides an easy way to manage Model MBeans, which are used in Java Management Extensions. ƒ Module 12, CLI—Provides an easy-to-use interface to deal with application startup options. ƒ Module 13, Chain—An implementation of the Chain of Responsibility pattern. ƒ Module 14, Logging and Discovery—Logging is used as a wrapper around existing logging implementations. Discovery is an attempt to provide common resource-finding services. Each module is independent of the others, so you can jump straight to the component of your choice. There are no dependencies on prior learning; for example, knowing about Logging isn’t a prerequisite to learning about any other component. However, it’s expected that you know about Java, know how to download each component, know how to set Java CLASSPATH, and know how to code and compile. However, if you’re reading this book without doing any coding, you don’t need to know how to download or compile the components. The background reference material doesn’t assume any coding experience. You can access each component’s web page by going to http://jakarta.apache.org/commons and following the individual links. Licensed to Tricia Fu ABOUT THIS BOOK vii How to holler for help Sometimes you’ll get stuck. In spite of the best intentions and hours of trying to solve problems, there will be times when you want to ask someone else for help. A fresh insight into a problem can bring rich dividends. I’ll be available for help in the Author’s Online forum at Manning’s web site as soon as the book is released. You can log onto the forum by going to http://www.manning.com and then navigating to the Author Online link. The book has its own devoted section, and I can help on any Commons-related topic. You can also access the Author Online forum from the Jakarta Commons Online Bookshelf web page (as well as purchase individual modules) at www.manning.com/goyal. In addition to the Author Online forum, the Commons mailing lists are excellent resources to ask for help. The Commons committers who develop and maintain the Commons packages monitor these lists and help out with most problems. To subscribe to the Commons User mailing list, send an email to commons- user-subscribe@jakarta.apache.org. You can also browse the archives at http://www.mail- archive.com/commons-user@jakarta.apache.org. The Commons HttpClient has its own mailing list; to subscribe, send an email to httpclient-user-subscribe@jakarta.apache.org. You can browse the archives at http://nagoya.apache.org/eyebrowse/SummarizeList?listId=280. Source code downloads Source code for the examples in each module can be downloaded from the publisher’s web site at www.manning.com/goyal. The purchase of an individual module entitles the reader to a free download of the source code for that module. About the author Vikram Goyal is a Java Developer by day and a technical writer by night. Vikram has worked with Java since its inception and remembers the time when Tomcat meant an animal and not a servlet engine. Vikram regularly writes how-to articles on open source projects and his series of articles on Jakarta Commons was the first such effort to make some sense out of the chaotic world that Jakarta Commons is. This article series is still reflected in the official main entry page of Jakarta Commons as the only online series covering these components. Vikram enjoys working with Open Source technologies, Team Leading and Civilization 3. Vikram lives in Brisbane, Australia, with his wife. Licensed to Tricia Fu viii JAKARTA COMMONS ONLINE BOOKSHELF about the cover illustration The figure on the cover of the Jakarta Commons Online Bookshelf is a “Sultana, or Kaddin,” which means “wife” in Turkish. The illustration is taken from a collection of costumes of the Ottoman Empire published on January 1, 1802, by William Miller of Old Bond Street, London. The title page is missing from the collection and we have been unable to track it down to date. The book's table of contents identifies the figures in both English and French, and each illustration bears the names of two artists who worked on it, both of whom would no doubt be surprised to find their art gracing the front cover of a computer programming book...two hundred years later. The collection was purchased by a Manning editor at an antiquarian flea market in the "Garage" on West 26th Street in Manhattan. The seller was an American based in Ankara, Turkey, and the transaction took place just as he was packing up his stand for the day. The Manning editor did not have on his person the substantial amount of cash that was required for the purchase and a credit card and check were both politely turned down. With the seller flying back to Ankara that evening the situation was getting hopeless. What was the solution? It turned out to be nothing more than an old-fashioned verbal agreement sealed with a handshake. The seller simply proposed that the money be transferred to him by wire and the editor walked out with the bank information on a piece of paper and the portfolio of images under his arm. Needless to say, we transferred the funds the next day, and we remain grateful and impressed by this unknown person’s trust in one of us. It recalls something that might have happened a long time ago. The pictures from the Ottoman collection, like the other illustrations that appear on our covers, bring to life the richness and variety of dress customs of two centuries ago. They recall the sense of isolation and distance of that period—and of every other historic period except our own hyperkinetic present. Dress codes have changed since then and the diversity by region, so rich at the time, has faded away. It is now often hard to tell the inhabitant of one continent from another. Perhaps, trying to view it optimistically, we have traded a cultural and visual diversity for a more varied personal life. Or a more varied and interesting intellectual and technical life. We at Manning celebrate the inventiveness, the initiative, and, yes, the fun of the computer business with book covers based on the rich diversity of regional life of two centuries ago‚ brought back to life by the pictures from this collection. Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Browsing with HttpClient Vikram Goyal MODULE MANNING 1 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 1 Browsing with HttpClient 1.1 A taste of HttpClient: downloading a web page....................................................................................................... 1 1.2 Reviewing the HttpClient RFCs ............................................................................................................................... 3 1.3 The HttpClient API................................................................................................................................................. 14 1.4 HttpClient in action................................................................................................................................................. 19 1.5 Summary................................................................................................................................................................. 40 Index ............................................................................................................................................................................. 41 Ah! HTTP. I can hear the sighs of relief and the shakes of head in an affirmative action, as you contemplate the title of this module. Yes, it’s about HTTP—something, as developers, most of us are intimately aware of: the Hypertext Transfer Protocol that drives the thing we lovingly call the Internet. Specifically, it’s about the Commons component HttpClient. When you think about it, this is the best possible name for the component; it exactly demonstrates what it does. HTTP + Client = HttpClient—a component to help deal with the client side of HTTP. But wait a minute. I can hear you ask, “Isn’t my browser an HTTP client?” Of course, it is. In fact, with the help of HttpClient, you can build your own browser, or embed it into your distributed application, or write a web services client. With the help of HttpClient, you can do anything that you want to do over HTTP (on the client side, anyway). To start this module, we’ll look at the simplest case of using HttpClient, by making it act as a browser and letting it download a web page. We’ll then take a step back and take a brief look at the basics of HTTP: specifically, the Requests for Comments (RFCs) that HttpClient implements.1 This will help you understand the fundamentals of HTTP and better prepare you for learning about HttpClient. We’ll then take a whirlwind tour of the HttpClient API, to see its various packages and how they relate to each other. This will get you ready for the examples of using HttpClient that will follow. 1.1 A taste of HttpClient: downloading a web page In this section, we’ll look at how to use HttpClient in the simplest possible case of downloading a static web page. This is just a teaser section. More detailed examples will follow after we’ve discussed the HttpClient API. Listing 1.1 shows the code required to download a web page using HttpClient. In this listing, HttpClient acts as a browser by requesting Google’s home page. Listing 1.1 A taste of HttpClient: Downloading Google’s home page package com.manning.commons.chapter01; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; 1 See module 2, section 2.1.1, for a definition of an RFC. Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF public class HttpClientTeaser { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); GetMethod method = new GetMethod("http://www.google.com"); int returnCode = client.executeMethod(method); System.err.println(method.getResponseBodyAsString()); method.releaseConnection(); } } HttpClient relies on three other Commons components for it to work. Before you run listing 1.1, make sure that you have the following libraries in your CLASSPATH, in addition to commons-httpclient.jar: commons- lang.jar (discussed in module 7, “Enhancing Java core libraries with collections”), commons-logging.jar, and commons-codec.jar (discussed in module 10, “Codec: encoders and decoders”). If you run this Java application on a command line, the HTML for Google’s default page will be printed (provided you’re connected to the Internet). A browser would interpret the tags in this HTML and format its display accordingly, but here we were only interested in getting the HTML. We could just as easily have asked for a text document, image, or results of scripts—anything available on the Internet via HTTP—and HttpClient would have fetched it for us. Using HttpClient starts with creating an instance of the client, as in the listing. In this case, the client that is created uses default settings. These settings include the connection manager used to create underlying connections, the attributes of these connections, the timeouts for giving up delayed connections, and so on. Next, the GetMethod class is used to specify the document to retrieve using the instance of HttpClient that was just created. The GetMethod class implements the HTTP GET method for retrieving documents of the Internet. (Not surprisingly, you would use the PostMethod class for the HTTP POST method.) The document to retrieve is specified as the sole argument to the GetMethod constructor. Notice that in this case, the argument specifies an absolute web address, as opposed to a relative web address, to Google’s default page (the argument must be interpreted as http://www.google.com/index.html or http://www.google.com/index.htm or whatever Google uses as its default page). The HttpClient instance is used next to execute the method request to the default page of Google. HttpClient’s executeMethod expects an HttpMethod interface implementation as its argument and returns an integer, which represents an HTTP response code for the execution of this request. The GetMethod class extends the abstract class HttpMethodBase, which implements the HttpMethod interface. Finally, the response from Google is printed on the command line by using the method getResponseBodyAsString on the GetMethod instance created earlier. This method evaluates the response received from the execution of a request and returns it by converting the bytes received into their String representation based on the character set specified in the response’s Content-Type header. You could choose to receive this response as a stream by using the getResponseBodyAsStream method, or as raw bytes by using the method getResponseBody. As you can see, you need only four lines of code to retrieve a web page from the Internet using HttpClient. However, there is much more to HttpClient than simply retrieving web pages. Before we get into these extras, let’s take a brief look at the RFCs that HttpClient implements. These RFCs define HTTP, and you need to understand them before you can begin to understand HttpClient itself. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 3 1.2 Reviewing the HttpClient RFCs It’s a fair bet that most of you would like to skip this section, fearing an overdose of theoretical interpretations and lengthy, boring text about HTTP RFCs. Fear not! In this section, we’ll review the RFCs in a relaxed manner, presenting only text that is relevant and highlighting key points. As you may be aware, HTTP was first formally introduced in rfc1945 in early 1996. This version of HTTP was called HTTP 1.0. Even while this version was being documented, it was clear that it wouldn’t serve the purposes of future browser-server communications. Therefore, HTTP 1.1 went into specification mode almost immediately and was first documented in rfc2068; the final version was documented in rfc2616. HttpClient supports both versions of this protocol. In addition to these two RFCs, HttpClient supports several related specifications. For example, HttpClient implements cookies, which in HTTP are used to maintain user information and state; they’re defined in rfc2109. Table 1.1 lists all the RFCs supported by HttpClient. Table 1.1 RFCs supported in HttpClient RFC number Protocol implemented 1945 HTTP 1.0 2616 HTTP 1.1 2109/2965 Cookies 2617 HTTP authentication (Basic and Digest) 2396 Uniform Resource Identifier (URI) definition 1867 Form-based file upload in HTML In addition to implementing rfc2617 for HTTP authentication, HttpClient also supports authentication using proprietary Microsoft technology. This technology is called Windows NT LAN Manager (NTLM) and we’ll review it in this section as well. Let’s begin our review of these RFCs with HTTP 1.1, defined in rfc2616. We won’t review rfc1867 in this module, because it’s discussed in detail in module 2, “Uploading files with FileUpload.” Note: To view any of these RFCs in full, visit www.faqs.org/rfcs/. 1.2.1 Understanding rfc2616 for HTTP 1.1 The RFC that finalized HTTP 1.1, rfc2616, ensured that clients and servers could talk to each other correctly even if they supported different versions of the HTTP protocol. Because HTTP 1.1 didn’t change the basic operation of this protocol from HTTP 1.0, it’s safe to assume that when we’re talking about HTTP 1.1, we also mean HTTP 1.0. There are several key differences, mainly enhancements, which we’ll list at the end of this section. This RFC defines a protocol for transferring hypermedia information using request-response architecture. Hypermedia is nothing but hypertext (text that is a link) expanded to include multimedia content as well. Therefore, when you click a picture in a browser to go to another web site, you’re clicking on hypermedia; the link to the second page is in form of an image, a multimedia component. In a nutshell, rfc2616 defines a way for clients and servers to transfer information between each other using hypermedia-enabled documents. In the next few sections, we’ll explore the key features of HTTP 1.1 as defined by this RFC. Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF Request-response architecture A request initiates from the client and is in form of a message. This message contains the request method (GET, POST, and so on), the URI (the location on the server of the requested information), the protocol (HTTP 1.1 or 1.0), and other request-specific information encapsulated in a Multipurpose Internet Mail Extension (MIME) format. This information may contain details about the client (is it Internet Explorer, Netscape, or HttpClient? does it accept images or multimedia files? and so on), extra information about the request in the form of request modifiers, and, finally, any data that the client may be sending to the server to complete the request. A typical request message is shown in figure 1.1. Figure 1.1 A typical request message and its parts The server responds in kind to this request. It sends a response message, which is similar in structure to the request message. This message starts with a response code on a status line, which indicates the success or failure of the request, followed by response headers. The message finishes with the requested document (if found) encapsulated as a MIME message. A typical response message is shown in figure 1.2. Figure 1.2 A typical response message and its parts Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 5 Since HTTP is driven by this request-response architecture, a response dies after it has completed a given request. This means that following requests have no recollection of previous requests and their responses. Thus HTTP is called a stateless protocol. Stateless protocol By not providing any means to maintain information between different requests, HTTP leaves open the provision of managing state to external factors. One of these factors is cookie management, which is defined in rfc2109. (We’ll cover this RFC a little later, in section 1.2.2.) But why did the writers of rfc2616 want this protocol to be stateless? Well, one of the binding reasons to create this protocol was to provide a fast responsive system for accessing distributed data. When you think about it, clicking a link on a web page accesses data across distributed systems. By requiring these systems to maintain state and defining a protocol that would mandate that, the protocol designers would have introduced several layers of complexity that would take more time than was necessary to accomplish the task of getting data. The HTTP protocol is straightforward enough that state management factors can be imposed on it, but these factors are outside the purview of this protocol. Hence, simple systems that don’t require state management can stay simple, whereas complex systems can use cookies, among other factors, to introduce state. Methods Methods tell servers serving requests what operation is to be performed with the rest of the information coming in from the client. In the case of the GET method, this means retrieving the resource requested in the header. In the case of HEAD, it means retrieving meta-information about a resource, not the resource itself, thus preserving bandwidth when the resource isn’t required by the client. Table 1.2 lists all the methods supported by HTTP 1.1. These method names must always be uppercase. Table 1.2 Methods supported in rfc2616 for HTTP Method name Purpose GET Retrieves a resource/document following this method HEAD Retrieves a resource’s meta-information: everything returned by the GET request except the resource itself POST Posts data to the server in the form of a message body preceded by request headers PUT Allows the client to create or modify an existing resource identified by the request URI DELETE Requests that the resource following this method be deleted TRACE Used for diagnostic purposes: Returns the request message in its entirety to the client as part of the response body CONNECT Used for HTTP tunneling via a proxy OPTIONS Used to interrogate the abilities of a server (by using * for the request URI) or a particular resource identified by the request URI You’ll see examples of all these methods in the code listings for HttpClient later in this module. Proxies and tunnelling A request on its way to the destination server could conceivably pass through a series of proxy servers, through which the response would also be expected. A proxy server is an intermediate machine that channels request and response messages; it acts like a client to the destination server and like a server to the originating client. Proxy servers reduce the time for oft-requested messages because they cache the response to identical requests, but they introduce the complexity of stale messages. (Caching is covered in the next section.) Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF Proxies may modify request and response messages to add extra header information that may help in either request processing or response caching by the client. A proxy that doesn’t modify any part of the request or response is called a transparent proxy. As opposed to a proxy, a request or response may pass through a series of servers, which may not understand the semantics of the message. A common example of such a server is a firewall, commonly employed in organizations to thwart unwarranted connections from either side of their access. In such cases, a tunnel for an HTTP connection is established that acts as a blind relay point through the server that it’s passing through. This tunnel, via the server on which it’s acting, can’t change the message, because it doesn’t understand the message. However, nothing stops the server from keeping a copy of the message, which brings us to the case of caching in HTTP. Caching HTTP allows systems through which request and response messages pass to cache them, to provide faster response times for the identical request. In HTTP 1.0, the only way to avoid getting cached messages was to set the Pragma general header as no-cache. A request message that contained this header would be passed through to the designated server (proxies included) without retrieving a cached copy, even if a copy was available on any servers through which the request had to pass. The Pragma header has given way to the Cache-Control header. HTTP 1.1 defines several parameters via the Cache-Control header to override the caching of request and response messages. A value of no-cache in a request message means that the request must be passed as is to the requested server without returning a cached copy for the response from any servers that it may pass through. A value of no-cache in a response message, however, means that any proxy or gateway along the way must not cache this response for identical requests. You can restrict this so it’s applicable for only certain fields by specifying them explicitly. For example: Cache-Control: no-cache = "field1" Table 1.3 lists all the possible values for the request Cache-Control header. Table 1.3 Possible values for the request Cache-Control header Value Comment no-cache Response to this request must come from the targeted server and not from any cache. no-store Request and its response must not be stored on backup systems. max-age = "" Indicates to the server (or any proxy or gateway in between) that the client is willing to accept a cached response to this request that isn’t older than the age specified in seconds. max-stale [= ""] Indicates to the server (or any proxy or gateway in between) that the client is willing to accept a cached response to this request that has gone stale. If max-stale has a value after the = sign, it indicates that the response shouldn’t be stale by more than the given number of seconds. min-fresh = "" Client will accept a cached response that will be fresh for at least the specified number of seconds. no- transform A request must not be transformed by a proxy or gateway on its way to the server. only-if- cached Only a cached response will be returned. If no cached response is found for this request, the proxy or gateway must return a 504 error, indicating Gateway timeout. Similarly, table 1.4 lists the values for the response Cache-Control header. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 7 Table 1.4 Possible values for the response Cache-Control header Value Comment no-cache = "field1" Any proxy or gateway must not cache this response. If a field is specified after the = sign, only this field must not be cached; others may be. public Response may be cached by any proxy or gateway on its way to the client. private = "field1" Response is cacheable only for one user. Other users’ requests must not get the same request. This may be applied to either the whole response or only parts of it, by specifying these parts after the = sign. no-store Response and its associated request must not be stored on backup systems. no- transform Response must not be transformed by any proxy or gateway on its way to the client. must- revalidate A proxy or gateway must request the server for fresh copy of the response, once it becomes stale. proxy- revalidate Same as must-revalidate, but applicable only on a per-user basis. max-age = "" Response is cacheable and is valid only for the given number of seconds. s-maxage = "" Same as max-age, except this entry isn’t valid in cases where a cache (a proxy or a gateway) is stored on a per-user basis. This covers the most basic features of HTTP. Let’s finish this section by highlighting the key differences between HTTP 1.0 and HTTP 1.1. What is the difference between HTTP 1.1 and HTTP 1.0? There are many differences between these two versions of HTTP; but as we said earlier, the basic process of HTTP remains the same. In this section, we’ll cover some of the most important differences so that you can better utilize HttpClient to deal with servers that may still conform only to HTTP 1.0. Following are the most important HTTP 1.1–specific features, with a brief explanation of each change: ƒ Explicit domain information in each request—This information can be provided using either the HOST header or using the absolute URI. HTTP 1.1 mandated the use of the HOST header to allow a single IP address to be used for multiple domains. The HOST header determines the target domain for the request. This allowed efficient use of the existing IP addresses in the World by supporting virtual hosting. If the HOST header is missing, then each request must contain an absolute URI to the requested resource. Here are two examples: GET /index.html HTTP/1.1 HOST: www.manning.com GET www.manning.com/index.html HTTP/1.1— ƒ Persistent connections—In HTTP 1.0, each resource had to be requested on its own connection. This was inefficient and led to the concept of persistent connections in HTTP 1.1. By default, all connections in HTTP 1.1 are persistent and must be explicitly closed by sending the Connection: close header as part of the request. Since all connections are persistent, multiple resources can be requested on a single connection by a browser or User-Agent such as HttpClient. This is typical for a web page, which may contain several images, scripts, and other media files in-lined within the page. ƒ Chunked transfers—To allow servers to start sending responses to requests before the total length of the response has been determined, HTTP 1.1 introduced chunked transfer encoding for message bodies. Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF This encoding is useful in cases where the response is driven by a server-side script that produces a lot of data, which takes time to be produced. The server, instead of storing this data in memory, can begin sending it to the client; the server lets the client know that more data is expected by setting the content-transfer-encoding header to chunked. The full response then consists of separate chunks of information; each chunk is sent when it becomes available. ƒ The Date header—Caching by proxy servers and gateways can’t be implemented efficiently if the original response to a request didn’t contain the date on which it was generated. This is the case because caches must know how to evaluate a response as stale or fresh, as the case might be, when serving an identical request. HTTP 1.1 mandates that each response must contain a Date header set to the date on which the response was generated, to enable caches to determine the appropriate action. The new concepts in HTTP 1.1 allow for faster response times and flexibility. But all these differences are backward compatible, so communication between disparate systems is allowed. This section completes the discussion of the main HTTP RFC, rfc2616. However, HTTP has several support RFCs that enhance its basic functionality. These RFCs deal with several factors, including state management and authentication. In the next few sections, we’ll examine these support RFCs. 1.2.2 Maintaining state with cookies: rfc2965 State management with cookies was first introduced in rfc2109. By including only two headers, one for request and one for response, this RFC added the concept of state management to HTTP—which is essentially a stateless protocol. These headers are Cookie for request and Set-Cookie for response. However, rfc2109 was made obsolete by the introduction of rfc2965. It defines three headers, Cookie, Cookie2, and Set-Cookie2; the first two are for request messages and the third for response messages. The Cookie2 header is used by a client and indicates that the client understands rfc2965, so a server can use Set-Cookie2 instead of the Set-Cookie header. HttpClient can handle headers from both RFCs. The process of state management via cookies is fairly simple. State is managed via a logical concept known as a session. A session is a series of connections between a client and a server, which continues until either one of them closes it explicitly or a timeout happens. Only the server can initiate a session, but a client can reinitiate an old session if the server so permits. To start a session, rfc2965 mandates that the server must send a Set-Cookie2 header as part of a response; after that, the client and the server are said to be participating in a session. The syntax of this header is shown here as an example: Set-Cookie2: cookie1="cookie1Value"; max-age="0"; Domain=".manning.com"; Version="1", cookie2="cookie2Value"; max-age="3600"; path="/forum"; Comment="Cookie for forum"; Version="1" Cookies are required to have a name. In the example, two cookies are defined: cookie1 and cookie2, separated by a comma. Both cookies have a value after the equal (=) sign and some attributes with values separated by semicolons (;). Except for the Version attribute, all other attributes are optional. These attributes, which are case-insensitive, are listed in table 1.5. For backward compatibility, using the Set-Cookie header is identical to using Set-Cookie2, except that it doesn’t support the CommentURL, Discard, and Port attributes. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 9 Table 1.5 Optional attributes of a cookie in a Set-Cookie2 header Attribute Description Default action (if this attribute is absent in each cookie header) Comment Short text describing the purpose of the cookie. None CommentURL Location on the Internet where more information about this cookie can be found. None Discard Indicates that when the client shuts down the User- Agent used to make the connection to the server, this cookie must be discarded and not used again. This attribute doesn’t require a value. Depends on the Max-Age attribute value, if present Domain Domain with which this cookie is associated. If no domain, the value should begin with a period (.); but this isn’t necessary because the User-Agent must supply it. The host to which the request was made Max-Age Number of seconds after which the cookie is discarded. A value of 0 indicates that the cookie shouldn’t be stored. Discard the cookie when the User-Agent exists; same as Discard Path URL, relative to the domain to which this cookie applies. The request URL Secure Indicates that the client must make an effort to transfer this cookie information to the server via a secure channel. The details of this secure channel aren’t defined. This attribute also doesn’t require a value. Send over any channel even if insecure Port Port(s) on which the cookie value may be returned to the server. Any request port The session continues when a client, having received a cookie before, sends it back as part of the request headers—provided the cookie hasn’t been discarded and hasn’t expired, as mandated in the original Set- Cookie2 header. To send the value of the cookie back, the User-Agent uses the Cookie header (not the Cookie2 header, which is used to indicate the acceptance of rfc2965). Here’s an example: Cookie: $Version="1"; cookie2="cookie2Value"; $Path="/forum" Note that the Version and Path attributes have a $ in front of them when they’re sent back to the server. Only the Version, Path, and Domain attributes need to be sent back, along with the value of the cookie; the other attributes are for cookie management on the client side. The server receives the value of the cookie, uses it to identify the current session of the client, and acts accordingly. It can send a fresh value for the cookie as part of the response or tell the client to discard the cookie, as it sees fit. If a fresh value for the cookie is sent, it overwrites the existing value on the client; if a Max-Age of 0 is sent back, the older cookie and the newer cookie are both discarded. Although cookies can manage state, another RFC is required to manage control over resources on the server and authenticate users. Rfc2617 details authentication and is discussed next. 1.2.3 Authenticating users: rfc2617 Rfc2617 defines two ways of authenticating users over HTTP: Basic and Digest. The two methods are almost identical in the process that they follow, but one doesn’t send the authentication details in clear text. Of course, Basic sends authentication details in clear text, and Digest uses a hashing algorithm to send hashes of authentication details (rather than the details themselves). The Basic authentication scheme was introduced in the original RFC for HTTP; that is, it has been part of HTTP from its inception. The process is simple and is applicable to the Digest authentication scheme as well. Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF When a client asks the server for a resource that has been marked as protected, and it doesn’t send any authentication details as part of the request, the server sends a header called WWW-Authenticate as part of the response. The response carries a status code of 401, which means the server is telling the client (or the User-Agent) to provide its credentials for accessing the given resource. An example response is shown here: HTTP/1.1 401 Unauthorized WWW-Authenticate: Basic realm="protectedSpace" The WWW-Authenticate header should first list the scheme being used, which in this case is Basic. This is followed by a logical name for the protected resources on the server to which the user’s credentials are applicable. This logical name is called a realm; in the previous example, its value is protectedSpace. This is displayed to the user by the User-Agent on the client. As far as Basic authentication scheme goes, this is all the server sends. For the Digest authentication scheme, the server sends several other parameters: ƒ domain—A space-separated list of URLs applicable to this logical realm. ƒ nonce—A unique string that is generated by the server for insertion into each response. ƒ opaque—A string of data that should be returned by the client unchanged in each request for this realm and domain. ƒ stale—A value of true or false. A true value indicates that the previous request wasn’t acceptable because it had expired. A false value indicates that the request was rejected because of an invalid or absent username/password. The default is false. ƒ algorithm—An optional parameter. The algorithm(s) the client should use to create hash values. The default is MD5. ƒ qop—An optional parameter. The value indicates the quality of protection offered by the server and affects the way the client sends its request. The value can be either auth or auth-int. An example response using Digest authentication is shown here: HTTP/1.1 401 Unauthorized WWW-Authenticate: Digest realm="protectedSpace", nonce="457dhedfshjir39f00df03f0sdf0f0df3450dfieo" opaque="dontchangemeiaminclear" qop="auth" Once the client receives a 401-status response from the server, it gathers the user’s credentials and puts them in a request body with the Authorization header. For Basic authorization, this amounts to Base64- encoding the user’s username and password, separated by the colon (:) character.2 For example, if the username is Vikram and the password is My Password, the User-Agent on the client puts it together in a string as Vikram:My Password, Base64-encodes it to get the string VmlrcmFtOk15IFBhc3N3b3Jk, and puts it in the header: Authorization: Basic VmlrcmFtOk15IFBhc3N3b3Jk 2 See module 9, “Pool and DBCP: Creating and using object pools,” for an explanation of Base64. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 11 Finally this is sent to the server, along with the rest of the request and any subsequent requests. For Digest authorization, the request body contains several other parameters; this is to be expected, considering that the response message contained several parameters. These parameters are listed here: ƒ username—The clear-text username of the user making the request. ƒ uri—The URL to the resource that is being requested. ƒ qop—A value that is the same as the qop value specified in the response header. If the response didn’t contain a value for this parameter, it shouldn’t be present in the request, either. ƒ cnonce—A required parameter, if a qop parameter has been set. However, if the server didn’t send a qop parameter, this parameter must not be set in the response. The value for this parameter is used in creating the hash of credentials that is sent to the server. It’s sent to the server in a request message so that the server can use it to match the credentials. ƒ nc—A count of the number of times a request has been sent, which includes the nonce value received in a response. Recall that the response message contains the value of a nonce parameter, and the corresponding requests are supposed to send it back. As with cnonce, this value must not be set if there was no qop parameter in the response message and must be set if there was a qop parameter. ƒ response—The parameter whose value contains the hash for user’s credential information. The process to calculate this value is shown in figure 1.3. In addition to these parameters, the Authorization header of a request message must also return the nonce, realm, and opaque parameter values received from the server. An example request Authorization header using Digest authentication is shown here: Authorization: Digest realm="protectedSpace", nonce="457dhedfshjir39f00df03f0sdf0f0df3450dfieo", uri="/index.html", qop="auth", nc=00000001, cnonce="somecleartext", response="rrt56879393348fjs9785dfd307c4ef1" opaque="dontchangemeiaminclear" The process to calculate the value of the response parameter is shown in figure 1.3. You shouldn’t need to use this process to calculate the response value yourself. HttpClient supports almost all branches in this process shown, except when the value of the qop parameter is set to auth-int. This setting for qop indicates that the server wishes to check the integrity of the data transfer as opposed to simply hashing the user credentials. When the server receives a request message with the Authorization header as shown in figure 1.3, in either Digest or Basic mode, it responds accordingly with a response header. The cycle continues until one party breaks it off. Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF Figure 1.3 Process to calculate the value of the response parameter based on the user’s credentials and other parameters Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 13 Basic and Digest authentication aren’t the only way of authenticating requests between client and server. Microsoft-specific NTLM authentication is another way of establishing trust. Let’s discuss this proprietary technology, which works only with Internet Explorer as the client and Internet Information Server (IIS) as the server. 1.2.4 Authenticating users: using NTLM NTLM differs slightly from the Basic and Digest authentication processes. The key difference relates to the fact that NTLM authenticates a connection, not a request; therefore, each time a connection is made, the authentication process has to be reinitiated. This works well with HTTP 1.1, because it automatically creates persistent connections. An NTLM authentication session starts with the client making a request to the server for a protected resource. The server responds, as in rfc2617, with a WWW-Authenticate header in a 401 response. The value of the header in this case is equal to NTLM. The server also closes the connection after sending this response. An example is shown here: HTTP/1.1 401 Unauthorized WWW-Authenticate: NTLM Connection: Close Clients that understand and support the NTLM protocol (mostly IE, but increasingly other browsers as well) respond to this response with what is known as the Type 1 request message in an Authorization header. This is called the negotiation stage of the authentication process. The Type 1 message contains the features that are supported by the client and are requested of the server. For example, the client may include in a Type 1 message a flag to indicate that it supports Unicode strings for data transfer. In addition to flags, the Type 1 message may also contain the computer name and domain of the client, which helps the server determine if there is any point in continuing (because the domain or the computer name may be restricted from accessing the requested document). The server, on receipt of a Type 1 message in a request Authorization header, responds with a Type 2 message in a WWW-Authenticate header. The Type 2 message serves two purposes. First, it agrees (or disagrees) with the flags requested by the client in its Type 1 message. Second, it contains a challenge to the client to authenticate itself based on the challenge. The challenge is nothing but a random string of data, 8 bytes in length, which the client uses to respond with a Type 3 message. The Type 3 message finalizes the client’s side of the authentication process. The client constructs this Type 3 message on receipt of the Type 2 message from the server by creating the password hash for the requested resource along with the challenge received in the Type 2 message, using a complex series of protocols. (These protocols are discussed in detail at: http://davenport.sourceforge.net/ntlm.html.) The Type 3 message is Base64 encoded, and the original request is resubmitted to the server, along with this Type 3 message in an Authenticate header. The server checks the validity of the header and allows access to the original resource. Further requests on the same connection are allowed without any checks, because the process is connection-oriented; therefore, the connection is considered sanitized for all future requests. NTLM authentication is widely popular in the Microsoft world, and HttpClient does well by supporting it fully. Next, we’ll discuss the RFC for locating resources on the Internet using the URI syntax. 1.2.5 Identifying resources by URI: rfc2396 Rfc2396 defines a way to consistently identify resources on the Internet. Regardless of whether resources exist physically, this RFC makes the process of referencing them a matter of making sure the general grammar of Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF accessing them remains the same across systems and processes. This allows URIs developed in one system to consistently reference the same resource, even if they’re ported to another system or process. Uniform Resource Identifiers (URIs) are strings of characters that provide location information for a resource. A resource, in this case, can be a file, an image, a server process, or anything else that can be located. URIs aren’t tied to a specific protocol. Thus URIs can be used to express the location of resources in various protocols. However, each URI must conform to this specific format: scheme:scheme-specific-part The scheme part identifies the protocol that is being used to locate the resource in question. Examples include, http, ftp, mailto, and file. The scheme-specific-part further qualifies the resource and differs to a large degree between schemes. However, a generic format for the scheme-specific-part, which is followed by most schemes, is identified as follows: ://? Note that this format isn’t mandated by rfc2396—it’s only a suggestion, but it’s the format followed by most schemes. Most of us are familiar with this format, because HTTP follows it for the location of documents on the Internet. For example, in the URI http://www.google.com/search?q=fishing, http stands for the scheme, www.google.com stands for the authority, search stands for the path of the document being referenced (in this case, a CGI program) and q=fishing is the query. This short refresher on rfc2396 concludes the list of RFCs that we’ll go through before looking at the HttpClient API. Note that we haven’t discussed rfc1867, which covers form-based file upload, because it’s covered in detail in module 2. 1.3 The HttpClient API In this section, we’ll discuss the key classes and libraries of the HttpClient API. It’s important to do this before we begin looking at code samples, because this discussion will help you decide which classes are relevant in which situations. Version note: This discussion and the code samples that follow are based on HttpClient 3.0 alpha2. 1.3.1 The org.apache.commons.httpclient package The main classes in HttpClient are located in the parent org.apache.commons.httpclient package. This package contains the HttpClient class, which is the starting point for every transaction. This class depends on two other classes, the HttpConnectionManager interface and the HttpClientParams class (which is in the org.apache.commons.httpclient.params package and is discussed in section 1.3.5 later). HttpConnectionManager is an interface in this package, and it has two implementations: SimpleHttpConnectionManager and MultiThreadedHttpConnectionManager. As the name suggests, this interface is responsible for managing the underlying network connections between client and servers, a bit like a connection pool. SimpleHttpConnectionManager is the default connection manager, and it provides access to a single shared connection. HttpClient instances are expected to borrow this single Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 15 connection from this manager for executing a request and return it when they’re finished. It’s re-borrowed when the next execution needs to be performed. The expected sequence of executions is sequential. If you need to execute requests in a multithreaded environment, where you may simultaneously execute several requests, you need to use the MultiThreadedHttpConnectionManager class. This class uses a HashMap to internally store a mapping between connections and hosts to which connections are made. The default number of connections per host is 2, and the default value for the maximum connections overall is 20. Once a request is executed using a connection from any one of the connection managers, the state of the request/connection can be gauged from the HttpState class. Recall, however, that HTTP is a stateless protocol. So, what state are we talking about? The state here refers to the values of cookies and authentication details, which may persist between requests/connections. The HttpState class provides methods to gauge the state of these attributes and to modify them if required for future requests. The HttpState class doesn’t keep the cookie and authentication details (the username, password, and domain, if applicable) on its own. It uses the Cookie class and the Credentials interface, respectively, to hold these values. The Cookie class is used with the classes from the org.apache.commons.httpclient.cookie package to provide the complete cookie-handling routine. The Credentials interface is implemented by the UsernamePasswordCredentials class, which in turn is extended by the NTCredentials class. The first class holds the username and password for a requesting user in a color-separated String object. The second class holds extra information, in addition to a colon-separated username password String, about the host and domain for which the username/password are a valid combination; these extra values are required in the NTLM authentication scheme. Note that the classes in the package org.apache.commons.httpclient.auth are required to provide a complete authentication process. Requests in HttpClient are executed using one of the instances of the HttpMethod interface. The HttpMethod interface represents a complete request and response cycle. This interface has an abstract base implementation in the HttpMethodBase class, but the actual implementations are in the org.apache.commons.httpclient.methods package. We’ll cover these implementations in the next section. In addition to the classes mentioned, the org.apache.commons.httpclient package contains several utility classes. For example, the Header class and its associated HeaderElement and HeaderGroup classes are used to represent an HTTP request or response header. Similarly, the StatusLine class is used to represent the first line received from the server for a request as part of the response. It breaks this line into its constituent parts (HTTP version, response code, and response reason) and provides methods to retrieve these parts. There are several exception classes, as well, most notable among them being ProtocolException; it indicates a protocol error, be it in a status line, a cookie, or an authorization. Finally, this package contains the implementation class for rfc2396 in the URI class. This generic class implements this RFC and is extended by the more specific schemes of HttpURL and HttpsURL. 1.3.2 The org.apache.commons.httpclient.methods package The previous package contained the interface definition and a base implementation for the various HTTP methods in the interface HttpMethod and the abstract class HttpMethodBase, respectively. However, the package org.apache.commons.httpclient.methods contains concrete implementations for the various HTTP methods. Table 1.6 lists the methods and the corresponding implementation classes. Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF Table 1.6 HTTP methods and their implementation classes HTTP method Corresponding class GET GetMethod PUT PutMethod HEAD HeadMethod POST PostMethod OPTIONS OptionsMethod TRACE TraceMethod DELETE DeleteMethod Missing a method in this table? If you compare it with table 1.2, you’ll notice that there doesn’t seem to be an implementation class for the HTTP CONNECT method. For some unknown reason, the CONNECT method implementation class is in the main org.apache.commons.httpclient package and is called ConnectMethod. All these classes (including ConnectMethod, but excluding PostMethod and PutMethod) extend the HttpMethodBase class. However, the POST and PUT method implementations have been conceptually grouped by extending them from the ExpectContinueMethod class, which in turn extends the HttpMethodBase class. The ExpectContinueMethod class is an abstract class; it’s used for handshakes during the POST and PUT requests because these requests require the server to send an intermediate 100 Continue status before sending the request. Another HTTP method class is implemented in this package, by extending the ExpectContinueMethod abstract class: the MultipartPostMethod class. This class is used to send multipart requests to the server, which is another form of the POST request, allowing request data to be sent in multiple parts. Module 2 describes this in more detail. Figure 1.4 shows the relationship between the various classes in this package. Note that this figure shows only the primary classes of this package and some classes from the main org.apache. commons.httpclient package. 1.3.3 The org.apache.commons.httpclient.auth package The package org.apache.commons.httpclient.auth contains the implementation of various authentication schemes supported by HttpClient for HTTP authentication. The three authentication schemes we’ve discussed (Basic, Digest, and NTLM) are all implemented in this package, along with support classes. The base definition for all schemes is specified in the AuthScheme interface. This interface defines methods to create authenticating requests and process responses from the server. The implementation classes implement this interface and provide their own mechanisms for creating these requests and processing server responses. These implementation classes are listed in Table 1.7. Table 1.7 Supported authenticating schemes in HttpClient Scheme Corresponding implementation class Basic BasicScheme Digest DigestScheme NTLM NTLMScheme BasicScheme and DigestScheme both extend from the RFC2617Scheme class (which implements the AuthScheme interface) because they’re similar in terms of the process they follow for authentication. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 17 Figure 1.4 UML diagram showing relationships in the org.apache.commons.httpclient.methods package This package contains several utility classes, which help in the authentication process. These classes are as follows: ƒ AuthState—Provides information about the state of the authentication process and allows you to change the authentication scheme being used. It can also be used to invalidate the whole process. ƒ AuthPolicy—Provides a one-stop shop for getting the proper scheme for authentication. Authentication schemes are registered with this class at startup and can be looked up as required. This allows custom authentication schemes to be plugged in. ƒ CredentialsProvider—Used to request credentials from the user, if the authentication process can’t find any credentials for the current process. ƒ AuthChallengeParser, AuthChallengeProcessor—Together, help to first parse the response headers in an authentication process and then process the challenge presented in the headers. This package contains several exception classes as well, which are used to indicate problems at various stages of the authentication process. For example, AuthChallengeException and Malformed- ChallengeException are used to indicate that a challenge from the server couldn’t be processed properly. Similarly, InvalidCredentialsException and CredentialsNotAvailableException are used to Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF indicate invalid and unavailable credentials from the user making the request. Note that all these exception classes, except MalformedChallengeException, derive from the general AuthenticationException class, which is a catchall exception for this package. 1.3.4 The org.apache.commons.httpclient.cookie package The org.apache.commons.httpclient.cookie package provides classes that handle cookies in HttpClient. However, as specified in section 1.3.1, this package relies on the Cookie class in the org.apache.commons.httpclient main package for the actual creation of cookies. The classes in this package are responsible for parsing response headers for Set-Cookie header values and creating Cookie instances out of it, for creating Cookie instances to be sent out with the request header Cookie, and for validating cookies according to the specifications. Toward this end, the interface CookieSpec in this package defines the methods that allow these functions to be performed. Two classes, CookieSpecBase and IgnoreCookiesSpec, implement this interface. As the name suggests, IgnoreCookiesSpec does nothing with cookies and headers that it gets; it returns empty values and performs no actions. CookieSpecBase is the base class for handling cookie functions. Two classes, RFC2109Spec and NetscapeDraftSpec, extend it and provide specific functionality based on rfc2109 (discussed earlier) and Netscape-developed specifications for cookie handling, respectively. The only exception class in this package is called MalformedCookieException (derived from ProtocolException). It indicates that a cookie is invalid according to the specification for which it’s being handled. 1.3.5 The org.apache.commons.httpclient.params package Managing HttpClient preferences in a separate package is a new concept in HttpClient version 3.0. The idea is to move the configuration of various instances from the instances themselves to a separate package. This allows centralized management of the configuration values and lets new configuration architectures be plugged in without causing too much disruption. The package contains classes that represent all the possible parameters that can be configured for instances of HttpClient, for instances of HTTP connection managers used in HttpClient, for instances of HTTP connections, and for instances of HTTP methods. These classes are listed in Table 1.8 along with the instances they represent. Table 1.8 Parameter classes in the org.apache.commons.httpclient.params package Instance Relevant class HttpClient HttpClientParams HostConfiguration HostParams HttpConnectionManager HttpConnectionManagerParams HttpConnection HttpConnectionParams HttpMethod HttpMethodParams For example, while creating an instance of an HttpClient, you would first create the HttpClientParams class and set the connection manager parameter on this class to a value you want. This HttpClientParams instance is then used to create the actual HttpClient instance. The DefaultHttpParams class in this package represents the default values of the parameters. Along with the DefaultHttpParamsFactory class, it lets you fall back to default values in case a parameter isn’t set. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 19 In addition to the packages we’ve discussed so far, the HttpClient API contains two other packages, org.apache.commons.httpclient.protocol and org.apache.commons.httpclient.util. The first package is used to create sockets using the protocol that is currently requested (SSL/non-SSL), and the second package contains utility functions. We won’t discuss these packages separately, but you’ll encounter their usage in the examples that follow. 1.4 HttpClient in action Section 1.1 gave you a taste of how HttpClient works. However, the example in that section was trivial, and you can do much more with HttpClient. In this section, you’ll see examples of HttpClient in various scenarios. These scenarios are divided into their own sections to give credible and comprehensive examples for each one. We’ll start with examples showing how to change the configuration parameters in HttpClient using the new org.apache.commons.httpclient.params package. 1.4.1 Managing HttpClient configuration using the Preferences Architecture There is a reason why the new API for managing configurations in HttpClient 3.0 is called the Preferences Architecture. The reason stems from the fact that configuration parameters follow a hierarchy where the value of the parameter at the lowest level is given preference over its predecessor’s value. This is best explained with an example. One of the headers sent as part of a request message in HTTP is called User-Agent. Its value identifies the process on the client machine that is making the request. For example, when you use IE to fetch pages, it identifies itself using this header as follows: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0). HttpClient, by default, uses the value Jakarta Commons-HttpClient/3.0-alpha2 or something similar, based on the version you’re using. What happens if you want to change this value? It’s very easy to change this default value in HttpClient. To do so, all you need to do is change the value of the global parameter httpclient.useragent to the value you want. The following code snippet shows how to change this value: client.getParams().setParameter("http.useragent", "My Own Browser at Client level"); Part of the snippet, client.getParams(), returns the collection of parameters applicable to the instance of HttpClient that is being worked with. This collection is encapsulated in the class HttpClientParams, which is part of the new org.apache.commons.httpclient.params package; that is, it’s part of the new Preferences Architecture. Similar to HttpClientParams, this architecture also includes HttpConnectionManagerParams for managing connection manager parameters, HttpConnectionParams for managing connection parameters, HttpMethodParams for managing method parameters, and HostParams for managing remote host parameters. To change the value of a parameter, you must access the parameters for that instance by calling the getParams() method. The new parameter value can then be changed by calling the setParameter(String name, String value) method, as in the previous snippet, and supplying the new value. Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF Although in the previous snippet we changed the value of the http.useragent parameter at the client level, it can also be changed at the host or method level. The following code snippet does exactly the same thing as the earlier one, except that the value is changed at the method level : method.getParams().setParameter("http.useragent", "My Own Browser at Method level"); What happens if the value is changed at both the client level and the method level? Which change is implemented? It depends which level you’re working on. Listing 1.2 shows this in action. Listing 1.2 Changing a parameter value at the client and method level package com.manning.commons.chapter01; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.methods.GetMethod; public class HttpClientPreferencesV1 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); System.err.println("The User Agent before changing it is: " + client.getParams().getParameter("http.useragent")); client.getParams().setParameter( "http.useragent", "My Own Browser at Client level"); System.err.println("Client's User Agent is: " + client.getParams().getParameter("http.useragent")); GetMethod method = new GetMethod("http://www.google.com"); method.getParams().setParameter( "http.useragent", "My Own Browser at Method level"); try{ client.executeMethod(method); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } System.err.println("Method's User Agent is: " + method.getParams().getParameter("http.useragent")); } } Although http.useragent is a global parameter, its value depends on the level you’re working on. Therefore, at the client level, in listing 1.2, this value is set to "My Own Browser at Client level", but at the individual method level, this value is changed to "My Own Browser at Method level". Note that the value at the method level won’t be changed until the method is executed as in the listing. If you User-Agent changed at client level Prints client’s User-Agent User-Agent changed at method level Must execute method before method-level change is accepted Prints method’s User-Agent Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 21 were to now create a new method, it would inherit its parameters from the client level; therefore, its http.useragent value would be set to "My Own Browser at Client level". The idea is to make sure that components (HttpClient, methods, hosts and so on) at a lower level prefer the value of parameters at their own level, even though the same parameter may have been set to a different value at a higher level. Only if a parameter hasn’t been set at their level do the components go up a level until they find a component where the value has been defined. Of course, if the parameter doesn’t have a value at any level, the value is reverted to the highest level—called the global level—which may have a default value for that parameter. Figure 1.5 shows this hierarchy of components, along with the classes of parameters at each level in the org.apache.commons.httpclient.params package that implement the hierarchy. Although we have talked about the hierarchy, we haven’t discussed any of the parameters that can be changed. Listing all of them in this book would be too cumbersome, because there are too many. Please refer to the HttpClient documentation, especially the Preferences Architecture, for a complete list. Table 1.9 shows only those parameters that have a value at the global level. Table 1.9 A list of parameters that have a value at the global level Parameter Description Default value http.useragent Value used to identify the client using the User-Agent request header Jakarta Commons-HttpClient/3.0-alpha2 or similar, based on your version http.protocol.version Default protocol to use HTTP/1.1 http.protocol.element- charset Character set to use for encoding and decoding the HTTP status lines and headers US-ASCII http.protocol.content- charset Character set to be used for encoding the body of messages ISO-8859-1 http.protocol.cookie- policy Cookie policy to use RFC 2109 http.method.retry- handler Retry handler for retrying failed connections Default HttpClient retry handler called DefaultHttpMethodRetryHandler http.dateparser.patterns Date patterns to look for when parsing dates Several; refer to the documentation http.connection- manager.class Default connection manager to use for managing connections SimpleHttpConnectionManager class Now that you know how to get and set the various parameters of HttpClient, it’s time to look at how to use HttpClient with its various methods. The next section covers this with a variety of examples. 1.4.2 Using HttpClient methods HTTP method implementations are defined in the org.apache.commons.httpclient.methods package, except the CONNECT method, which is defined in the main org.apache.commons.httpclient package. In this section, you’ll learn how to use each of these methods. The basic setup is similar for each method; therefore, we’ll set the stage with GetMethod and then concentrate on the differences in method executions, rather than repeating ourselves. Downloading resources using GetMethod The GET method is the simplest possible HTTP method. It’s used to download resources from the Internet, and these resources can be the output of a server-side script, or a static document. The GetMethod class in the org.apache.commons.httpclient.methods package is used to execute these requests. Listing 1.3 shows a comprehensive example of how to use the GetMethod class. Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF Figure 1.5 Hierarchy of components for the Preferences Architecture Listing 1.3 Using GetMethod in a variety of ways package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.HttpException; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.protocol.Protocol; import java.io.File; import java.io.IOException; import java.io.FileOutputStream; public class GetMethodExampleV1 { public static void main(String args[]) { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "Test Client"); client.getParams().setParameter( "http.connection.timeout", Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 23 new Integer(5000)); GetMethod method = new GetMethod(); FileOutputStream fos = null; try { method.setURI(new URI("http://www.google.com", true)); int returnCode = client.executeMethod(method); if(returnCode != HttpStatus.SC_OK) { System.err.println( "Unable to fetch default page, status code: " + returnCode); } System.err.println(method.getResponseBodyAsString()); method.setURI( new URI("http://www.google.com/images/logo.gif", true)); returnCode = client.executeMethod(method); if(returnCode != HttpStatus.SC_OK) { System.err.println( "Unable to fetch image, status code: " + returnCode); } byte[] imageData = method.getResponseBody(); fos = new FileOutputStream(new File("google.gif")); fos.write(imageData); HostConfiguration hostConfig = new HostConfiguration(); hostConfig.setHost("www.yahoo.com", null, 80, Protocol.getProtocol("http")); method.setURI(new URI("/", true)); client.executeMethod(hostConfig, method); System.err.println(method.getResponseBodyAsString()); } catch (HttpException he) { System.err.println(he); } catch (IOException ie) { System.err.println(ie); } finally { method.releaseConnection(); if(fos != null) try { fos.close(); } catch (Exception fe) {} } } } Listing 1.3 shows a variety of ways in which the GetMethod can be executed. First, an instance of HttpClient is created, and two of its parameters are set. We have already covered the http.useragent Use GetMethod to get response body as String Get response body as byte array Execute GetMethod by specifying host HttpException indicates protocol error Always release connection Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF parameter. The second parameter, http.connection.timeout, is used to specify the time in milliseconds that HttpClient should wait for connections to be established. The first example is the simplest. The method is created and set with the document to fetch using the URI class. Note that this URI class isn’t java.net.URI but HttpClient’s own implementation, in the org.apache.commons.httpclient package. When a URI is set on a method using only the domain name or IP address and without an actual document, it defaults to fetching the index page of the domain, as in this case. When the client executes the method, it stores the response internally after separating it into its constituent parts; it returns only the response code, which is the status code returned by the server. To get the parts of the response, such as the body and headers, you need to call the relevant methods. For example, to get all the headers returned from the server, you would use method.getResponseHeaders(), which returns a collection of headers where each value is of type Header. The Header class in org.apache.commons.httpclient package is a simple name-value pair class. If you wanted the value of a specific header, you would instead use method.getResponseHeader(String headerName), passing in the specific header name. In this case, we wanted to get the body of the response, which is the index page of the domain requested. To get the body, we use getResponseBodyAsString(), which converts the received data to String format using the value of the Content-Type response header (or ISO-8859-1, if no such header value is available). If, instead of receiving the response body as String, we want to get the body in the form of a byte array, we can do so using the method getResponseBody(), as is done in the second example in listing 1.3. Here, we fetch an image of the Internet and store it on the local machine. Note that to fetch the image, we used the method from the previous fetch, but we changed the URI. To receive the response body as a Stream, you can use the getResponseBodyAsStream method, which returns an InputStream or null if no response body is available. It’s recommended that you use this method for large response bodies, where the size of the response body may not be known in advance. The third example in listing 1.3 shows how the GetMethod can be executed by specifying the host that the method should connect to in a separate configuration. The advantage of using the HostConfiguration class is that the remote host can be specified in a centralized place; therefore, changes to the host need to be made in only place. However, HttpClient only uses HostConfiguration set this way if the URI specified for the subsequent method(s) isn’t absolute. The methods derive HostConfiguration and override it, if an absolute URI is used, from the URI itself. The HostConfiguration is created by first specifying the host to connect to. Note that the URI of the document to fetch is specified on the method as a relative URI. Then the method is executed, passing in the host to connect to and the method to execute. Now that you know how to use HttpClient to execute methods using GetMethod, let’s see how to use the other methods in this API. Sending data using PostMethod The POST HTTP method lets you send data to the host machine by putting the data in the body of the request message. HttpClient makes sending this data very easy, by using PostMethod. To add data to the body of the response, the PostMethod class provides the addParameter(String name, String value) method. The PostMethod is created as the GetMethod in the previous section, and the data to be sent is combined using this method. You can also use the NameValuePair class to add these parameters. The NameValuePair class is a holder of names and values for parameters, and the PostMethod class provides a method each to add either individual parameter values or an array of these values. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 25 Listing 1.4 shows an example of running PostMethod. Listing 1.4 Using PostMethod to run a query on Yahoo package com.manning.commons.chapter01; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.methods.PostMethod; import java.io.BufferedReader; import java.io.InputStreamReader; public class PostMethodExampleV1 { public static void main(String args[]) { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "Test Client"); BufferedReader br = null; PostMethod method = new PostMethod("http://search.yahoo.com/search"); method.addParameter("p", "\"Jakarta Commons in Action\""); try{ int returnCode = client.executeMethod(method); if(returnCode == HttpStatus.SC_NOT_IMPLEMENTED) { System.err.println("The Post method is not implemented by this URI"); // still consume the response body method.getResponseBodyAsString(); } else { br = new BufferedReader( new InputStreamReader(method.getResponseBodyAsStream())); String readLine; while(((readLine = br.readLine()) != null)) { System.err.println(readLine); } } } catch (Exception e) { System.err.println(e); } finally { method.releaseConnection(); if(br != null) try { br.close(); } catch (Exception fe) {} } } } Notice how the search term is set as the character p. This is what Yahoo’s search service expects it search term to be called, so that’s what it’s set as. It value is set to “Jakarta Commons in Action”, including the quotes. This search term is executed by calling the executeMethod. Add search term What if Yahoo doesn’t implement POST for search? If error occurs, response is consumed Read response as InputStream Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF Next, we check to see whether Yahoo accepted the query to search or rejected it by sending a Not Implemented error. This is possible, because Yahoo may only accept a search based on the GET protocol (at the time of this writing, it does accept a POST for search). Note that even if Yahoo rejects the search request, its response must still be consumed; otherwise the connection used for this request won’t be released by HttpClient. Finally, the response is read via an InputStream, because we expect the response body to be quite big, and we don’t know its size in advance. Sending multipart data using MultipartPostMethod When files are uploaded via HTTP, the file data must be sent in parts. This is commonly known as sending data in the form of multipart/form-data. PostMethod can’t be used to send the contents of files, because file data is likely to contain binary data. Using MultipartPostMethod is similar to using PostMethod. You use MultipartPostMethod’s overloaded addParameter methods to add parameters and their values to the method. If the parameter that is being added is a normal form element—that is, it isn’t file data—you use addParameter(String paramName, String paramValue), which is same as the PostMethod addParameter method. If it’s file data, you use addParameter(String parameterName, File parameterFile). Of course, the file to be uploaded must exist and must be readable; otherwise a FileNotFoundException is thrown. What happens, however, if the data that you want to send using this method is already loaded into memory and must be sent as a file? For example, suppose that in the act of sending data, a separate process in your application creates a report using a PDF file generator. The report data is already loaded into memory, so it seems futile to first save it to disk and then attach it using the addParameter method for files. MultipartPostMethod solves this problem by using the classes in org.apache. commons.httpclient.methods.multipart (specifically, in this case, the class ByteArrayPartSource) and by providing a method called addPart. The ByteArrayPartSource class is used by wrapping the in-memory bytes in a constructor. For example: ByteArrayPartSource source = new ByteArrayPartSource("report.pdf", bytes) This source is then used to create a FilePart object, to be added to the method using the addPart method, as shown here: method.addPart(new FilePart("report.pdf", source)); This adds the contents of the in-memory bytes to the request body as an independent part, which is identified as a file part and has the name report.pdf. Querying a server using OptionsMethod The OPTIONS HTTP method allows you to query a server to determine the list of methods it supports. This is handy in cases where you’re unsure of the capabilities of a remote server and want to find out your options before you request a particular method. Listing 1.5 shows a simple example of how to use OptionsMethod from HttpClient. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 27 Listing 1.5 Using OptionsMethod to determine a server’s capabilities package com.manning.commons.chapter01; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.methods.OptionsMethod; import java.util.Enumeration; public class OptionsMethodExampleV1 { public static void main(String args[]) { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "Test Client"); OptionsMethod method = new OptionsMethod("http://www.verisign.com"); try{ int returnCode = client.executeMethod(method); Enumeration list = method.getAllowedMethods(); while(list.hasMoreElements()) { System.err.println(list.nextElement()); } } catch (Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } This listing queries the server at www.verisign.com for a list of methods it supports and prints this list on the command line. OptionsMethod’s getAllowedMethods method is used to find these supported methods. OptionsMethod also contains a handy method to determine whether an HTTP method is supported. This method is called isAllowed(String methodName), and it returns a boolean true or false based on the methods support. Of course, it can only be called once the method has been executed. Proxy tunneling with ConnectMethod HttpClient provides a convenient way for applications to use the HTTP CONNECT method. The CONNECT method is provided so that connections can be made via proxy server/firewalls to remote servers through a tunneled socket connection over HTTP, even though the applications may themselves be talking to each other via non-HTTP protocols. The support for tunneling in HttpClient is provided by three classes: ƒ ConnectMethod derives from the HttpMethodBase class. ƒ ProxyClient is a handy class not only for making the connections and opening a tunneled socket via a proxy, but also for handling authentication. The ProxyClient class, in essence, replaces the Return enumerated list of allowed methods Licensed to Tricia Fu 28 JAKARTA COMMONS ONLINE BOOKSHELF HttpClient for tunneled proxy connections. ƒ ConnectResponse, which is a static inner class in ProxyClient class, provides access to the socket created via the ProxyClient class. Connecting to remote servers via a tunneled proxy in HttpClient comes down to implementing three steps: 1. Make an instance of ProxyClient (not HttpClient), and set the details of the proxy to be used and the remote server to which a connection is to be established. 2. Make the connection via this ProxyClient and, if successful, retrieve the socket that is created. 3. Write data to this socket using whichever protocol is required, and close the connection. A complete example of using ProxyClient is shown in listing 1.6. Listing 1.6 Using ProxyClient to make a tunneled socket connection via a proxy to a remote server package com.manning.commons.chapter01; import org.apache.commons.httpclient.ProxyClient; import org.apache.commons.httpclient.ConnectMethod; import org.apache.commons.httpclient.ProxyClient.ConnectResponse; import java.net.Socket; public class ConnectMethodExampleV1 { public static void main(String args[]) { ProxyClient client = new ProxyClient(); client.getParams().setParameter( "http.useragent", "Proxy Test Client"); client.getHostConfiguration().setHost("www.verisign.com"); client.getHostConfiguration().setProxy( "localproxyaddress", 80); Socket socket = null; try{ ConnectResponse response = client.connect(); socket = response.getSocket(); if(socket != null) { // write your raw data over this socket } else { // connection via the socket could not be established ConnectMethod method = response.getConnectMethod(); Give ConnectMethod that ProxyClient uses Create ProxyClient and set local and remote servers Write data over Socket here Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 29 System.err.println("Socket not created: " + method.getStatusLine()); } } catch (Exception e) { System.err.println(e); } finally { if(socket != null) try { socket.close(); } catch (Exception fe) {} } } } As you can see, the ConnectMethod that the ProxyClient uses internally to issue the CONNECT HTTP command can be retrieved from the response by calling the response.getConnectMethod() method. Because this ConnectMethod derives from HttpMethodBase class, it can be used like other HttpClient methods. Here, it’s used to determine the reason for the failed connection, by using the getStatusLine() method. Next, we’ll cover the remaining HttpClient methods, most of which are trivial enough to be bundled into one section. The remaining methods This section covers the rest of the HTTP methods supported by HttpClient. They’re bundled together in this section because they’re used very similarly, as is true for all the methods of HttpClient. The DeleteMethod class is used to delete a resource at a server by specifying the resource (as is done with the other methods) and then executing the method. However, support for this HTTP method isn’t widespread, so the method is of limited use. The same is true of PutMethod. This method is used to upload files to a server. However, it also isn’t widely supported due to security concerns. The difference between POST (multipart) and PUT lies in the fact that the resource identified by the PUT method is the exact place where the uploaded data is to be stored. This differs from the POST method, which only identifies the resource that will be used to handle the uploaded file. To use PutMethod, first create the PutMethod instance by identifying the resource on the remote server that the uploaded file will be saved as: PutMethod method = new PutMethod("http://someserver.com/uploads/logo.gif"); Next, set the request body of the method by attaching the file to be uploaded: method.setRequestBody(new FileInputStream("logo.gif")); Then, as with the other methods, execute this method using an HttpClient instance. The final HTTP method is TraceMethod. It’s used for debugging but also isn’t widely implemented. The idea is that on execution of this method, the remote server will return the unmodified original request body as it receives it. This helps detect whether the request body is being modified along the way to the server by a proxy or gateway, and it also helps trace errors. So far, we have only considered using HttpClient in a single-threaded environment, where methods are executed sequentially and there is no danger of data corruption due to simultaneous connection requests. In Reason connect failed Licensed to Tricia Fu 30 JAKARTA COMMONS ONLINE BOOKSHELF the next section, we’ll change this scenario a bit and show you how to use HttpClient in a multithreaded environment. 1.4.3 Using HttpClient in a multithreaded environment Using HttpClient in a commercial environment, it’s conceivable that an instance of HttpClient would be used to execute multiple requests simultaneously to cater to several requests at once. For example, consider your browser. At one time, an instance of your browser may be downloading the latest news from CNN.com, doing a search on Google, and posting your resume on several job sites. If you only had the option to perform these tasks one after the other, it would take a long, long time before you would net any jobs! You need the ability to simultaneously effect several requests using HttpClient. Let’s first consider what happens when you try to do this with HttpClient without any changes to the way it manages connections. Listing 1.7 shows an example of downloading two different files from a local server at the same time using two different threads. Listing 1.7 Simultaneously executing requests without proper connection management package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; public class ThreadingExampleV1 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "Test Client"); HostConfiguration host = new HostConfiguration(); host.setHost(new URI("http://localhost:8080", true)); MethodThread bigDataThread = new MethodThread(client, host, "/big_movie.wmv"); bigDataThread.start(); MethodThread normalThread = new MethodThread(client, host, "/"); normalThread.start(); } } class MethodThread extends Thread { private HttpClient client; private HostConfiguration host; private GetMethod method; Start big movie download thread Start normal page download thread Thread class to manage method execution Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 31 public MethodThread( HttpClient client, HostConfiguration host, String resource) { this.client = client; this.host = host; this.method = new GetMethod(resource); } public void run() { System.err.println("Connecting to: " + host); try{ client.executeMethod(host, method); method.getResponseBodyAsStream(); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } If you run this listing, to be honest, I’m not sure what output you’re likely to get. The results are truly unpredictable. Why is that? On the surface, the listing should work as it’s intended. We start two threads: one to download a big file, the other to download the index page of the same local server. Each thread gets the same instance of HttpClient, the same HostConfiguration, and its own individual resource to get. Each thread then makes its own instance of GetMethod and executes it in its run method. Sounds simple enough, until you realize that each GetMethod tries to use the same connection instance to handle each request. This is because HttpClient, by default, uses a connection manager class that provides access to this same connection instance across all requests per HostConfiguration. This connection manager class is SimpleHttpConnectionManager. The SimpleHttpConnectionManager class ensures that if a new request is being made to the same host, then the last request to the host is forcibly closed before the connection for the next request is released. If a new request is made to a different host, it clears up the previous host details, sets the same connection with the new host details, and then releases the connection. As you can see, if two requests arrive at the same time for the same shared connection, unpredictability in the response is almost certain. One inelegant way to solve this problem is to have a separate client instance for each thread. Each client has its own connection instance, so there is no chance of unpredictable results. However, creating a new HttpClient for each request is fraught with resource-intensive operations. The best way to solve this problem is provided by HttpClient itself: Use the MultiThreadedHttpConnectionManager class. As the name suggests, this connection manager class is best suited for multithreaded applications, where simultaneous requests are handled by supplying a separate connection for each request. Connections are bundled together per host, which means that a finite number of connections are created per host, and requests to the same host are provided through this bundle via synchronized reuse. If a connection isn’t available at the time it’s requested, the connection manager waits a finite time before trying again; it times out if no connection is available. The default number of connections created per host is limited to two, but you can easily change that, as you’ll see in listing 1.8. Execute methods Read server responses Licensed to Tricia Fu 32 JAKARTA COMMONS ONLINE BOOKSHELF To make HttpClient use the MultiThreadedHttpConnectionManager, the HttpClient instance must be created with this manager or installed on it with the setHttpConnectionManager method before making any requests. Listing 1.8 shows a complete example of using this manager, borrowing from the situation created in listing 1.7. Listing 1.8 Simultaneously executing requests with proper connection management package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager; import org.apache.commons.httpclient.params.HttpConnectionManagerParams; public class ThreadingExampleV2 { public static void main(String args[]) throws Exception { MultiThreadedHttpConnectionManager manager = new MultiThreadedHttpConnectionManager(); HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams(); connectionManagerParams.setDefaultMaxConnectionsPerHost(5); manager.setParams(connectionManagerParams); HttpClient client = new HttpClient(manager); client.getParams().setParameter("http.useragent", "Test Client"); HostConfiguration host = new HostConfiguration(); host.setHost(new URI("http://localhost:8080", true)); // first Get a big file MethodThread bigDataThread = new MethodThread(client, host, "/big_movie.wmv"); bigDataThread.start(); // next try and get a small file MethodThread normalThread = new MethodThread(client, host, "/"); normalThread.start(); } } If you run this listing, you’ll get the expected results. The big_movie file will be downloaded in its own thread and the index page in its own. One final thing about this example to note before we move on: An HttpConnectionManagerParams class is used to set the default maximum number of connections per host. This way of setting the parameters of a component in HttpClient is different from the way we discussed in section 1.4.1, but it’s valid. You can use each parameter class in HttpClient directly to change the value of parameters, instead of using the generic Set maximum connections for any host Set parameter class with manager Set client with new manager Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 33 setParameter method. In addition, the maximum connections are set as the default for all hosts. What if you wanted to set this value higher or lower per host? Easy: You can do so as shown here: connectionManagerParams.setDefaultMaxConnectionsPerHost(host, 10); This allows you to set the connection limit higher for highly requested sites (or lower for rare sites) and keep a bare minimum for all other hosts by using setDefaultMaxConnectionsPerHost. 1.4.4 Authenticating requests Accessing protected and secure resources over the Internet requires authentication between the client and the server. As you know, HttpClient supports three types of authentication, in this order: NTLM, Digest, and Basic. The order of authentication types is important because if multiple authentication types are available for a request/connection, HttpClient picks NTLM over Digest over Basic. You can change this order if required. We’ll begin this section with the simplest authentication scheme, Basic. The mechanics of using the Basic and Digest schemes are similar, both to the end user and to the developer, so we won’t discuss the Digest scheme. To set the background for the Basic authentication example, let’s first create a page that must be protected and deploy it in a web server protected by a username and password. We’ll use Tomcat (4.1.27) for the web server, and we’ll use its prebuilt roles for authentication. The role we’ll use is called tomcat, and the username and password for this role are tomcat and tomcat, respectively (set in the file tomcat-users.xml under the /conf directory). Listing 1.9 shows the entries to be made in a web.xml file to make a page protected. The page itself is simple; it shows the message Welcome to the protected page and is called protected.jsp. Listing 1.9 Modifying web.xml to make parts of a web application protected Security Constraint Protected Area /chapter01/* tomcat BASIC Chapter 1 Protected Area Note that this listing is only part of web.xml; the file must include other entries before it can be used. Deploy this web.xml on the server along with the protected page (protected.jsp). When you start Tomcat after this change and request the page via your web browser, Tomcat requests the authentication details as shown in figure 1.6. Correct authentication details will let you see the protected page. Protect everything under chapter01 directory Give protected area a realm name Licensed to Tricia Fu 34 JAKARTA COMMONS ONLINE BOOKSHELF Figure 1.6 Accessing the protected page from Tomcat via browser However, this isn’t about accessing a protected page via a browser, but about accessing a protected page via HttpClient. Listing 1.10 shows our first attempt at fetching this protected page via HttpClient. Listing 1.10 First attempt at fetching a protected page: will fail package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; public class BasicAuthenticationV1 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "My Browser"); HostConfiguration host = client.getHostConfiguration(); Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 35 host.setHost(new URI("http://localhost:8080", true)); GetMethod method = new GetMethod("/commons/chapter01/protected.jsp"); try{ client.executeMethod(host, method); System.err.println(method.getStatusLine()); System.err.println(method.getResponseBodyAsString()); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } If you run this code, it won’t fetch the protected page. Why (other than the fact that no authentication details have been supplied)? Because authentication is a two-step process. First the client makes a request for a protected page, and the server responds with a message saying it requires authentication details before it can send that page. In the second step, the client repeats the request with the authentication details attached, and the server responds with the page. We need to either request the page in two steps or tell HttpClient to perform only the second step. With HttpClient, you can use either of these two options. Listing 1.11 shows how to use the first, and listing 1.12 shows how to use the second. Listing 1.11 Two-step authentication package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpState; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.Credentials; import org.apache.commons.httpclient.auth.AuthScope; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.UsernamePasswordCredentials; public class BasicAuthenticationV2 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "My Browser"); HostConfiguration host = client.getHostConfiguration(); host.setHost(new URI("http://localhost:8080", true)); GetMethod method = new GetMethod("/commons/chapter01/protected.jsp"); try{ int statusCode = client.executeMethod(host, method); if(statusCode == HttpStatus.SC_UNAUTHORIZED) { Check whether first attempt rejected due to lack of authorization Licensed to Tricia Fu 36 JAKARTA COMMONS ONLINE BOOKSHELF System.err.println("Authorization required by server"); Credentials credentials = new UsernamePasswordCredentials("tomcat", "tomcat"); AuthScope authScope = new AuthScope(host.getHost(), host.getPort()); HttpState state = client.getState(); state.setCredentials(authScope, credentials); client.executeMethod(host, method); } System.err.println(method.getStatusLine()); System.err.println(method.getResponseBodyAsString()); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } To authenticate an HttpClient request, we start with a Credentials object. The UserNamePasswordCredentials class stores the username and password independent of any host or realm. The AuthScope is created next; it narrows the scope (the host, the port, the realm on the host) to which the credentials will be applied (not just the credentials created earlier, but any credentials). These two, scope and credentials, are combined in the HttpState of the current client instance to provide authentication credentials for the current Host. When the client reexecutes the method, it uses these credentials to provide authentication details and download the protected page. The HttpState class is a storehouse for persistent information across HttpClient requests. Thus, the state can store not only the credentials, but cookie information as well (we’ll study this in the next section). Notice that we didn’t specify the authentication scheme anywhere in the previous code. This makes sense because the server requests the Basic authentication scheme (recall that it was done in the web.xml file in listing 1.9), and HttpClient automatically switches to using this scheme. Listing 1.12 shows a one-step authentication process in which HttpClient handles authentications on its own. Listing 1.12 One-step authentication package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpState; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.Credentials; import org.apache.commons.httpclient.auth.AuthScope; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.UsernamePasswordCredentials; Create authorization credentials Retrieve client’s current state Resubmit original request Update state with credentials and scope Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 37 public class BasicAuthenticationV3 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "My Browser"); HostConfiguration host = client.getHostConfiguration(); host.setHost(new URI("http://localhost:8080", true)); Credentials credentials = new UsernamePasswordCredentials("tomcat", "tomcat"); AuthScope authScope = new AuthScope(host.getHost(), host.getPort()); HttpState state = client.getState(); state.setCredentials(authScope, credentials); GetMethod method = new GetMethod("/commons/chapter01/protected.jsp"); try{ client.executeMethod(host, method); System.err.println(method.getStatusLine()); System.err.println(method.getResponseBodyAsString()); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } As you may notice, the state of the HttpClient is set before any method is executed. After this, any time a request is made to the specified host (and in this case, for any realm on that host, for a protected page), authentication is automatically handled when executeMethod is called. You can restrict the realm for which credentials are valid by specifying it as a parameter to the scope. For example, new AuthScope(host.getHost(), host.getPort(), "Chapter 1 Protected Realm") is a valid scope for the "Chapter 1 Protected Realm" realm only on Host localhost at port 8080. If you wanted the username/password combination of tomcat/tomcat to be valid for any realm on any host and any port, you would create the Scope as new AuthScope(null, -1, null). One niggling doubt remains about the way credentials are provided in code. Notice that both the username and the password in each of the listings in this section were provided as part of the source code. This is a highly undesirable way of specifying this information for several reasons, security of the authentication details being the most significant. It would be nice if an admin user running this application could provide these values at runtime or a separate process could get these details from secure storage. HttpClient provides the CredentialsProvider interface with a single method that has the signature Credentials getCredentials(AuthScheme scheme, String host, int port, boolean proxy) It can be implemented to provide the credentials from an external store at runtime, or from any other suitable medium. Thus, a class that implements this interface, called CredentialsProvider, returns the credentials based on the requested authorization scheme and host, and the implementation details are left for Credentials and scope are created before method is executed Method automatically handles authentication Licensed to Tricia Fu 38 JAKARTA COMMONS ONLINE BOOKSHELF the class to decide on. However, you must make HttpClient aware of this CredentialsProvider by calling client.getParams().setParameter(Credentials.PROVIDER, new CustomCredentialsProvider()) where CustomCredentialsProvider is the name of this provider. Before we finish this section, a note about the NTLM authentication scheme. As far as the developer is concerned, using NTLM authentication is exactly like using the Basic authentication as shown in the previous examples, except for one small detail. NTLM authentication requires the developer/user to provide the domain on the NT server for which authentication is being performed, in addition to the username and password. Therefore, instead of using UsernamePasswordCredentials in the previous examples, you must use the NTCredentials class, which extends UsernamePasswordCredentials. Supplying the domain and host, in addition to the username and password, as shown here, creates the NTCredentials class for use in authentication. The other details of the authenticating process remain the same: Credentials credentials = new NTCredentials("user", "password", "127.0.0.1", "mydomain.com") We mentioned earlier that the HttpState class contains persistent information between requests for an HttpClient instance. One such piece of persistent information related to authentication details, which we have covered in this section. The other relates to cookies. It’s time to take a deeper look at the cookie- handling mechanism in the next section. 1.4.5 Handling and using cookies HttpClient automatically handles cookies by accepting whatever cookies the server sends and then sending these cookies back if required. It also allows you to create your own cookies and send them back to the server; what the server will do with those cookies that it didn’t initially send is a different matter. In this section, we’ll demonstrate how to read the cookies that a server sends to HttpClient and how to modify a specific cookie before it’s re-sent to the server. Reading cookies sent by a server is fairly easy. Listing 1.13 shows an example of how to read cookies sent by the Tomcat server for accessing the index page. Listing 1.13 Printing a list of cookies sent by a server package com.manning.commons.chapter01; import org.apache.commons.httpclient.Cookie; import org.apache.commons.httpclient.HttpState; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; public class CookieExampleV1 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "My Browser"); GetMethod method = new GetMethod("http://localhost:8080/"); Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 39 try{ client.executeMethod(method); Cookie[] cookies = client.getState().getCookies(); for (int i = 0; i < cookies.length; i++) { Cookie cookie = cookies[i]; System.err.println( "Cookie: " + cookie.getName() + ", Value: " + cookie.getValue() + ", IsPersistent?: " + cookie.isPersistent() + ", Expiry Date: " + cookie.getExpiryDate() + ", Comment: " + cookie.getComment()); } } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } All the cookies associated with the current instance of HttpClient are retrieved, regardless of the host they came from. You can narrow down this array of cookies by using this method: client.getState().getCookies(String domain, int port, String path, boolean secure) method It gets all the cookies received from the specified domain by HttpClient. To modify a cookie before it’s automatically sent back to the server, start by retrieving the cookie, and then modify any of its configurable properties. However, note that servers have the final say in cookie management, and if the server thinks that the cookie it originally sent isn’t the same as the cookie it received back, it may reject that cookie. The following code snippet shows how to modify the value of a cookie once you’ve extracted this cookie from the current HttpState: cookie.setValue("My own value"); When the modified cookie is re-sent, its value is changed to “My own value”. Note that because HttpClient handles cookies automatically—that is, it sends cookies with requests only if the cookie hasn’t expired—the cookie may not be sent. However, you can change the expiration date on cookies and still send them back to servers. This isn’t guaranteed to work, and the result is mostly unpredictable. Because HttpClient doesn’t allow for permanent storage of cookies, there is no mechanism for storing permanent cookies—that is, cookies with no expiration date. As far as HttpClient is concerned, a session is the life of the HttpClient application. Once the application exits the Java Virtual Machine (JVM), these cookies are no longer valid, unless you take specific steps before the JVM exits to persist the cookies in long- term storage and read again the next time the application starts. In your application, if you don’t wish to deal with cookies, you can set your application’s cookie policy to ignore all cookie headers in requests and responses. Use the following code to do this: client.getParams().setParameter("http.protocol.cookie-policy", CookiePolicy.IGNORE_COOKIES); The default cookie policy is based on rfc2109. Retrieve all cookies for current client Print information about each cookie Licensed to Tricia Fu 40 JAKARTA COMMONS ONLINE BOOKSHELF 1.5 Summary From authenticating users to handling cookies to handling all HTTP methods, HttpClient provides a complete client-side implementation of the HTTP protocol. It’s one of the best-documented components of the Commons stable, and it has an active user community that keeps contributing and increasing its depth. In this module, we started with a simple teaser application that showed how to use HttpClient in the most basic manner. This gave you a taste of things to come and allowed us to digress a little to discuss the technologies behind HttpClient. These technologies form the basis of HttpClient; a discussion of HttpClient without HTTP, Basic and Digest authentication, and cookies would have left a void. We rounded the module off by showcasing the use of HttpClient. We began by using the new Preferences Architecture and continued with the functionality of HttpClient from basic to advanced. In the next module, we’ll continue with all things web-related and look at the FileUpload component. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 41 Index AuthChallengeException, 17 AuthChallengeParser, 17 AuthChallengeProcessor, 17 authenticating users, 9 Basic and Digest authentication, 33–38 org.apache.commons.httpclient.auth package, 16 understanding NTLM, 13 understanding rfc2617, 9 using NTLM authentication, 38 AuthPolicy, 17 AuthScheme interface, 16 AuthState, 17 Basic authentication, 33. See authenticating users BasicScheme, 16 Cache-Control header, 6 possible request values, 6 possible response values, 6 ConnectMethod, 27 ConnectResponse, 28 cookie policy, 39 cookies ignoring headers, 39 modifying, 39 org.apache.commons.httpclient.cookie package, 18 reading, 38 understanding rfc2965, 8 using, 38–40 CookieSpec, 18 CookieSpecBase, 18 credentials. See also authenticating users CredentialsNotAvailableException, 18 CredentialsProvider, 17, 37 DeleteMethod, 29 Digest authentication. See authenticating users Digest authentication parameters calculating response, 11 DigestScheme, 16 ExpectContinueMethod, 16 firewall, 6 GetMethod, 21 HostParams, 18, 19 HTTP. See Hypertext Transfer Protocol differences between HTTP 1.1 and HTTP 1.0, 7 HttpClient, 1 API, 14–19 changing User-Agent header, 19 configuration. See HttpClient: Preferences Architecture: creating tunneled connections, 27 determining supported services, 26 downloading resources, 22–24 methods, 21 Preferences Architecture org.apache.commons.httpclient.params package, 18 using, 19 sending data, 24–26 simple example, 1–2 supported RFCs, 3 uploading files, 26 using ConnectMethod. See HttpClient: creating tunneled connections using GetMethod. See HttpClient:downloading resources using in a multithreaded environment, 30–33 using MultipartPostMethod. See HttpClient: uploading files using OptionsMethod. See HttpClient:determining supported services using PostMethod. See HttpClient: sending data HttpClientParams, 18, 19 HttpConnectionManagerParams, 18, 19, 32 HttpConnectionParams, 18, 19 HttpMethodBase, 16 HttpMethodParams, 18, 19 HttpState, 36 hypermedia, 3 hypertext, 3 Hypertext Transfer Protocol. See also rfc2616 IgnoreCookiesSpec, 18 InvalidCredentialsException, 18 maintaining state. See cookies MalformedChallengeException, 17 MalformedCookieException, 18 multipart/form-data, 26 MultipartPostMethod, 16, 26 multithreaded environment, 30 MultiThreadedHttpConnectionManager, 14, 31 NetscapeDraftSpec, 18 NT Lan Manager. See authenticating users NTCredentials, 38 NTLM authentication, 38 OptionsMethod, 26 org.apache.commons.httpclient package, 14 org.apache.commons.httpclient.auth package, 16 org.apache.commons.httpclient.methods package, 15, 21 org.apache.commons.httpclient.params package, 19, 21 org.apache.commons.httpclient.protocol package, 19 org.apache.commons.httpclient.util package, 19 PostMethod, 24 Pragma header. See Cache-Control header Preferences Architecture component hierarchy, 22 parameters, 21 Licensed to Tricia Fu 42 JAKARTA COMMONS ONLINE BOOKSHELF proxy tunneling, 27 ProxyClient, 28 PutMethod, 29 realm, 36. See also authenticating users rfc2109. See cookies RFC2109Spec, 18 rfc2396, 13 rfc2616 features, 3 caching, 6 methods, 5 methods supported, 5 proxies and tunnelling, 5 request-response architecture, 4 stateless protocol, 5 rfc2617. See authenticating users RFC2617Scheme, 16 scope, 37 SimpleHttpConnectionManager, 14, 31 TraceMethod, 29 transparent proxy, 6 Uniform Resource Identifiers. See understanding rfc2396 URI class, 15 UsernamePasswordCredentials, 38 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Uploading files with FileUpload Vikram Goyal MODULE MANNING 2 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu 1 Module 2 Uploading files with FileUpload 2.1 HTTP and uploading files......................................................................................................................................... 1 2.2 FileUpload basics...................................................................................................................................................... 6 2.3 Uploading files without using FileUpload.............................................................................................................. 10 2.4 Uploading files using FileUpload........................................................................................................................... 13 2.5 Handling a complex input form.............................................................................................................................. 16 2.6 Uploading large files............................................................................................................................................... 20 2.7 Uploading files to a database.................................................................................................................................. 22 2.8 Summary................................................................................................................................................................. 27 Index ............................................................................................................................................................................. 29 The ability to transfer files from a user’s PC to a remote server over the Internet allows the user to share images, documents, and audio-video files with other users around the world. The FileUpload component is an integral part of this transfer and sits on the server end. It receives, validates, and processes upload requests in a seamless fashion. In this chapter, we’ll review the basics of transferring files from a user’s computer to a server, and you’ll learn about rfc1867, the standard on which FileUpload is based. You’ll then see an example of uploading files without using the FileUpload component. This will get you ready for the FileUpload component itself. We’ll discuss this component with some basic examples and advanced usage scenarios. 2.1 HTTP and uploading files File upload is the transfer of files from a user’s computer to a server via an Internet browser. It’s a subset of file transfer, which is the process by which files are transmitted between different machines. These machines must be interconnected by an underlying networking mechanism and must have software on either end that facilitates this transfer. A browser on a client machine, for example, is the high-level software that transfers files to a remote machine, which has server software running on it that handles the incoming files. To fully understand file transfer, it’s important to have a working knowledge of the underlying protocol that file transfer works with. Hypertext Transfer Protocol (HTTP) is this underlying protocol over which a browser transfers files. As opposed to HTTP, the File Transfer Protocol (FTP) is more specialized and geared toward the explicit transfer of files between machines. We’ll discuss FTP further in module 3, “Handling protocols with the Net component,” when we discuss the Commons Net component. HTTP works on the basis of a request/response architecture. The user’s browser sends a request message to a server for a particular resource, and the server responds with a response message to the original request. The format of these messages is standard. The browser sends the request to the server by encoding the user request in a simple fashion. This message contains the usual HTTP headers, and the user data is encoded using URL encoding. The server responds with a message that contains, in addition to the headers, the actual data in a Multipurpose Internet Mail Extension (MIME)–like format. File transfer is inherently bidirectional and implicit. Think about what happens when you request a web page over the Internet through your browser. The browser makes a request to retrieve the requested page from Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF the server. The browser then makes a separate request for each element within that page and transfers each of these elements to your machine. By transferring files from the server to your machine, the browser is acting as a file transfer agent. Each image, HTML file, or data file is retrieved from the server, transferred over the Internet, and stored on your machine. Thus file transfer is happening behind the scenes even though you may not realize it. When you request that a file be transferred from the server to your machine, you’re downloading files: This is explicit file transfer, as opposed to the implicit file transfer/download that the browser does on your behalf when you request a web page. But how is file transfer inherently bidirectional? The underlying protocol doesn’t put any restrictions on the order of transfer of files between machines. Files can be transferred in either direction and are transferred whenever a transaction takes place (it helps in this analogy to think about a file as blob of data). A browser can upload and download files; a server can receive and send files. Let’s look at how a simple file upload request is initiated and processed. A user requests a page that contains an HTML element of type and a form with an encoding of type multipart/form-data instead of the normal application/x-www-form-urlencoded. This page is rendered by the browser as part of a form element that allows the user to navigate his computer and select a file for transfer. The user then submits this form to the server. Now the browser takes over by converting this selected file into a MIME format for transfer to the server. At the server, this form is parsed, and the different parts of this form are processed. A Common Gateway Interface (CGI) program does this processing by interpreting the different parts according to their content type. The semantics of how file upload is handled are defined by rfc1867. 2.1.1 Understanding rfc1867 To understand how a browser handles a file-upload request, you need to first understand the rules that a browser follows to bundle the user’s request to upload files so that files can be transferred over the Internet using the HTTP protocol. These rules are defined by rfc1867. A Request for Comments (RFC) is an Internet Engineering Task Force (IETF) document that relates to creating standards for the Internet. RFCs define the many protocols, standards, and procedures that have become the guiding force of the Internet today. Protocols like HTTP (rfc1945), standards like MIME (rfc2045), and procedures like the Internet Standards Process (rfc2026) all began as RFCs published by the IETF. You can access these documents by logging on to www.faqs.org/rfcs. Rfc1867 defines how a browser handles file upload requests from page authors and how a browser is supposed to retrieve and parse this information from the client’s computer and send it to the server. It’s an extension of the original HTML and MIME formats and is the standard by which the server parses the files sent to it. This RFC enables file upload from HTML forms using two mechanisms. The first deals with what a page author needs to do to enable file upload and how such a request is represented by a browser to the end user. The second defines how a browser then formats the form for upload to the server. Both of these mechanisms are necessary for a fully functional and successful file upload. The program on the server that is responsible for parsing these formatted requests needs to be aware of the format and parse the request accordingly. The FileUpload component does this by adhering to this RFC. Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 3 Understanding multipart messages Before we delve into the two mechanisms, it’s important for you to understand multipart messages. A multipart message is a message that contains one or more other message parts of different content types as defined by rfc1521 for MIME messages. So, for example, a multipart message may contain a message that is of Content-Type text/plain, and it may contain a message of Content-Type image/gif. Each part of a message is separated by a boundary, which is explicitly defined by the message composer. This boundary must be unique and must not occur within the data defined in the message. Each part of this multipart message can itself be multipart with its own separate boundary. A simple multipart message is shown here: Content-type: multipart/mixed; boundary="AABB09" This is a preamble and is ignored --AABB09 content-type:text/plain; charset=us-ascii This is a plain text message --AABB09 content-type: image/gif content-transfer-encoding: base64 .. base64 encoded message .. --AABB09-- This is an epilogue and is also ignored The first line clearly identifies the message as multipart and defines the boundary. A boundary must start by itself on a line and be preceded by a double hyphen (–); in this case, the boundary is -- AABB09. Next, the first part’s headers identify it as text/plain and of charset US-ASCII. The second part is encoded as Base64 so that binary image file can be transferred easily. The end of the message (the last part) is indicated by a double hyphen at both the beginning and the end of the boundary: -- AABB09--. Note that each header must be followed by a new line (CRLF) that doesn’t have anything else on it. Mechanism 1: creating and displaying a file upload request To implement the first mechanism to upload files from HTML forms, the RFC mandates the following: ƒ The page author must specify an ENCTYPE of multipart/form-data for the FORM element. In the absence of an ENCTYPE, the FORM element defaults to application/x-www-form-urlencoded. The RFC stipulates that if the ENCTYPE isn’t set to application/x-www-form-urlencoded and the enclosing form contains an INPUT type of FILE, the behavior is unspecified. ƒ The page author must specify an INPUT element of TYPE FILE, and this INPUT type must be nested within a FORM element. There is no restriction on the number of these elements within a page, and this Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF type can be intermixed with other INPUT elements of different types. The RFC gives the browsers a lot of freedom in the way they implement the FILE attribute. The RFC recommends the introduction of an ACCEPT attribute for the INPUT element, but this attribute has been ignored by almost all browser implementations. This ACCEPT attribute was recommended to limit the types of files a user could select for uploading. ƒ The page author must specify the METHOD attribute of the FORM element as POST. Since a GET request tags the values of a form at the end of the ACTION URL, this makes sense. You wouldn’t send file data appended to the end of a URL. ƒ On encountering an INPUT element with the TYPE set as FILE, the browser must display a text field with a file-selection button next to it. The user can directly type the location of the file to upload in the text box or use the file-selection button to select the file from her computer. Mechanism 2: packaging a file upload request The second mechanism that rfc1867 uses to enable file upload from HTML forms defines a format for the browser to send this data to the server by mandating the following: ƒ The browser must create a multipart message with a subtype of form-data, with a message boundary that doesn’t occur in any of the data that is being sent. So, the Content-Type of this message must be of the type multipart/form-data, and a boundary must be defined that separates each message. Each piece of the form’s original data must be sent as a part of this multipart message. ƒ Each part of this message must correspond to a single data entry in the original form. This part must contain the Content-Disposition header, which must be equal to form-data and must contain the original name of the data field in the form. For example, the content-disposition header for an element for file input may be content-disposition: form-data; name="fileToUpload". Similarly, a Content-Disposition header for a normal text input may be content- disposition: form-data; name="textField". ƒ Each part may have an optional Content-Type header. As with MIME messages, Content-Type specifies the type of the relevant part, which helps identify the type of application that can parse this particular part. If no Content-Type is specified, it defaults to text/plain. ƒ Each part may have a Content-Transfer encoding specified as part of the headers. This is useful in cases where the normal encoding of 7-bit US-ASCII may not be relevant or plausible for transfer of data. ƒ When formatting file data, a browser may choose to inform the server about the filename of the original file by putting this value in the Content-Disposition header of the part in which the data is represented. This is done by setting the filename attribute in this header: for example, Content- Disposition: form-data; filename="somefile.txt". ƒ The browser must send all the data, and it must send this data in the order in which it occurs in the original form. An example message composed according to these rules is shown in the following code. Assume for this listing that the end user filled out a form that asked her to type her name in a textbox called username and to attach a file called userfile. You’ll notice that the syntax is similar to that of the code shown earlier in the sidebar “Understanding multipart messages”; only minor changes in the required headers are needed: Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 5 Content-type: multipart/form-data, boundary="AOFDE00" --AOFDE00 content-disposition: form-data; name="username" Fred Somebody --AOFDE00 content-disposition: form-data; name="userfile"; filename="file1.txt" Content-Type: text/plain ... contents of file1.txt ... --AOFDE00-- Note: What happens when one of the form elements allows for multiple values to be selected—for example, using checkboxes or lists? How is this reflected in a multipart message? Each value the user selects is reflected as a part in itself. If the user selects two values in a group of five checkboxes, for example, there will be two separate parts in the resulting multipart message, each with its own boundary but the same name (if the checkboxes were given the same name by the page author). Similarly, each value a user selects in a multiple-selection list is represented with its own part that includes the selected value and the name of the list. Although rfc1867 mandates most of the requirements for successful file transmission, it also suggests several recommendations. These recommendations aren’t mandatory, which leads to incompatibilities in the way different browsers handle the same recommendations and causes these recommendations to sometimes be misinterpreted or ignored. In the next section, we’ll look at some of these omissions. 2.1.2 Rfc1867 omissions It’s useful to point out at this stage that rfc1867 includes several recommendations for the way a browser behaves for file uploads and what a page author is allowed to do. However, these recommendations aren’t mandatory and have largely been ignored by most browser implementations. In addition to specifying required behaviors, an RFC suggests or recommends other behaviors, which may or may not be implemented by the people implementing the RFC. Sadly, this is the case for rfc1867. In this section, we’ll look at some omissions from this RFC in today’s browser implementations. One of the most important omissions from current browser implementations is the ACCEPT attribute of the INPUT element. An ACCEPT attribute would have allowed page authors to specify the types of files that a user may upload. This would then restrict the end user from selecting files that aren’t of this particular type. Such an attribute would have saved a lot of parsing time at the server end, where the server program must determine whether the user selected the right type of file before processing it further. The VALUE attribute allows the INPUT element to be prefilled with a default value. However, with a TYPE attribute of FILE, it serves no purpose. The RFC recommends that, if specified, the browser may preselect a default filename in this field. Unfortunately, this isn’t implemented in any browser. It would have been useful in the case where a user wanted to replace a preselected file with another one from his computer. The browser could display the name of the preselected file as a reminder to the user. The SIZE attribute in the INPUT element lets the page author specify the size of the specified element’s display. The RFC recommends that for FILE input types, the SIZE attribute should accept width and height values, separated by a comma. The width would work as it works for other types, but the height would allow Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF multiple files to be uploaded using just one INPUT element. The browser would show a multiline input field, and the end user would be able to select multiple files to be uploaded simultaneously. However, this option hasn’t been implemented by any of the browsers, and the page author must create several INPUT elements to upload multiple files. Only the width attribute is honored. The MAXLENGTH attribute specifies the maximum input allowed for a particular INPUT element. The RFC suggests that page designers may use this attribute for the FILE type of INPUT elements to restrict the size of the file that the user may upload. This feature would have been a great way to restrict end users to uploading file of allowable lengths. Again, however, this attribute hasn’t been implemented. These omissions from the browser implementations mean that the server must work harder to parse data that contains a file. Not only must the server parse the headers to determine if the file uploaded is of the right type, it must read all the data before it can determine whether the size of the uploaded file is more than an allowed practical limit. Page authors also must create multiple input elements to request multiple file uploads, and they must preserve state between transactions in the absence of the recognition of the VALUE attribute. FileUpload helps to overcome some of these limitations by removing the drudgery of repetitive file parsing. In the next section, we’ll look at the basics of this component. 2.2 FileUpload basics Before we delve into examples of FileUpload in action, it’s instructive to peek at the classes that make up this component. We’ve divided these classes into blocks to separate their functionality, which provides better organization and will help you understand the component. A good understanding of the classes and the interactions between them will enable you to use FileUpload effectively. FileUpload is a library of classes that allows you to parse file-upload requests seamlessly. The library is built around the semantics of dividing the work of parsing file-upload requests into separate blocks. A set of high-level classes lets you parse the upload request, a low-level API processes the multipart data streams, and interfaces and implementations represent uploaded files. A general high-level exception class represents errors in parsing or represents file uploads and files, respectively; and low-level exceptions represent errors at a more specific level. Figure 2.1 shows these blocks. As shown in the figure, FileUpload can be divided into the following blocks: ƒ Parsing and processing API—This is a mixture of a high-level API and a low-level API. An abstract class called FileUploadBase and two implementations (DiskFileUpload and FileUpload) are part of the high-level API. The low-level API is represented by the MultipartStream class, which processes data streams of multipart data. ƒ Representation API—This API represents the uploaded files and form elements. The uploaded files and form elements are represented by the interface FileItem and its default implementation, DefaultFileItem. The items are generated using a factory interface called the FileItemFactory and its default implementation, DefaultFileItemFactory. ƒ Exception and utility classes—The general catchall exception class is called FileUploadException. The FileUploadBase class defines three specific exception classes to deal with various errors related to input request, and MultipartStream defines two exception classes. There is one utility class called ThresholdingOutputStream, which is extended by the DeferredFileOutputStream class. The following sections examine these APIs in more detail. Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 7 Figure 2.1 FileUpload API conceptual groups 2.2.1 Parsing and processing API The parsing and processing API is composed of classes that parse and disassemble an HTTP multipart request. The data elements in the request, or the different parts of the multipart request, are dissected into individual FileItem objects. Note that a FileItem object is used to represent form data such as text fields and radio buttons as well as any uploaded files. The classes that make up this API are as follows: ƒ FileUploadBase—This is the base class for all the processing that is done on input requests. It’s an abstract class that provides an implemented parseRequest(HttpServletRequest req) method that dissects a multipart request and returns a List of separate FileItems, which are created using a factory class. Subclasses call this method before they implement specific functionality on the parsed items in the list. This class relies on using the MultipartStream class to parse the multipart stream within the request. It provides a static utility method called Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF isMultipartRequest(HttpServletRequest req) that application developers can use to check whether a request contains multipart content before the request is parsed. ƒ FileUpload—This class is a generic implementation of FileUploadBase. It serves as a template for other, more specific, implementations. ƒ DiskFileUpload—This is the only specialized implementation of the FileUploadBase class provided in the FileUpload API. It handles disk-based file operations: In a nutshell, it uses a factory class internally to create FileItems geared toward saving files to disk. For most usage scenarios, this is the ideal class to use. Its internal factory class, DefaultFileItemFactory (which should have been called DiskFileItemFactory) stores files to disk in a location called the repository. You can specify the maximum size of uploaded files, the size threshold of the files before they’re written from memory to disk, and the repository location in the parseRequest(HttpServletRequest req, int sizeThreshold, long maxSize, String path) method. This method calls the superclass parseRequest method after setting the class-specific values. ƒ MultipartStream—This low-level class handles the processing of data streams as per rfc1867. The FileUploadBase class delegates the processing of the input stream from the request to this class, which follows the semantics of rfc1867 in parsing the data. You’ll probably never need to call this class directly and will use the parseRequest method in FileUploadBase or its implementations. The MultipartStream class expects the input data stream to follow the rules defined in rfc1867 for composing messages that contain uploaded files. Before this class can be used, the calling class needs to identify the boundary of the multipart message and pass this boundary as a byte array to the MultipartStream class. This boundary is used to identify the various parts of the message and to identify when the message ends. Internally, the MultipartStream class uses a buffer with a default size of 4KB to process a request. You can override this size when the class is constructed by using the MultipartStream(InputStream stream, byte[] boundary, int bufSize) constructor. Think of the MultipartStream class as the workhorse of the FileUpload API. It doesn’t make any decisions about the data it’s reading; it simply reads the data as per rfc1867. It lets the calling class, FileUploadBase, make decisions about the data it reads. The FileUploadBase class points the MultipartStream class to the input stream and the boundary within that stream and then uses the part information that MultipartStream reads to decide what to do with that particular part. Consequently, MultipartStream has methods that enable such blind reading of data from within the stream—for example, skipPreamble(), readBodyPart(), readHeaders(), and so on. 2.2.2 Representation API The parsing and processing API helps you process the information coming in. The representation API helps to represent this information in an easy way that makes it possible for you to do something with the data. What you decide to do is up to you. The representation API includes the following classes: ƒ FileItem—This interface represents both an uploaded file and a form item. To distinguish between a normal form item, like a text field or checkbox, and an actual uploaded file, the interface provides a method called isFormField() that returns a boolean true if the item is a normal form item. The MultipartStream class writes the body of a file item part to its OutputStream. This interface defines several convenience methods that let you manipulate the data received in a way that is most convenient. For example, you can get the data received in its entirety by calling get(), which returns the contents of the data as a byte array; or you can open an InputStream on the data by Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 9 calling the getInputStream() method. For normal form items, you can get the value as filled by the user by calling getString(). For items that are represented in an encoding other than the default system encoding, you can use the getString(String encoding) method, supplying the encoding you want. For items that return multiple values for a single form item, as in a checkbox or a multiple- select list, multiple items will be created. You’ll need to call getString multiple times for each value selected by the user in order to get all the selected values. When called on an item that’s an uploaded file, this method returns the contents of the file as a string representation. Another useful method, write(File location), lets you save the contents of a file item to the location specified by the parameter location. This works for both normal form fields and uploaded files. ƒ DefaultFileItem—A DefaultFileItem is the default implementation of the FileItem interface. Internally, it uses a DeferredFileOutputStream as the OutputStream to which data is written by the MultipartStream class. (DeferredFileOutputStream is described in the next section.) DefaultFileItem is a concrete implementation of the FileItem interface and is geared toward storing files to disk. It holds the file data in memory until a size threshold is reached, after which it streams the data to disk in a temporary location. The default threshold size is 10KB, and the default temporary location is the system default directory as returned by the System.getProperty("java.io.tmpdir") method. ƒ FileItemFactory—This is a factory interface to create FileItems with a single method called createItem(String fieldName, String contentType, boolean isFormField, String fileName). This method returns the newly created FileItem with the specified parameters. ƒ DefaultFileItemFactory—This is the default implementation of the FileItemFactory; it creates instances of DefaultFileItem. If you want to change the size threshold of the items this factory creates or change the location where the files are stored temporarily, you can do so by invoking the setSizeThreshold(int sizeThreshold) and setRepository(File repository) methods, respectively. You can retrieve the current factory being used by calling the getFileItemFactory() method on the FileUploadBase class. 2.2.3 Exceptions and utility classes The overall exception class for FileUpload is FileUploadException. It has three direct subclasses that are used to indicate problems in parsing input data streams: ƒ InvalidContentTypeException—This exception is thrown when the incoming request isn’t a multipart request. This typically happens when the page author has forgotten to identify the ENCTYPE for a form with file-upload requests as multipart/form-data or has forgotten to specify that the METHOD for form submit is POST. ƒ SizeLimitExceededException—This exception is thrown when the size of the incoming data request is larger than the allowed value for file upload. Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF Caution: Is SizeLimitExceededException thrown when an individual file’s size limit is exceeded, or when the cumulative size of the request is exceeded? According to the FileUpload 1.0 release notes, the expected behavior of calling the function setSizeMax(long size) on FileUploadBase is to set the individual size limit. However, this isn’t the case—calling this function sets the size limit for the entire upload rather than for individual items. ƒ UnknownSizeException—This exception is thrown when the incoming request doesn’t have a Content-Length header or the content length can’t be determined. The MultipartStream class throws some exceptions of its own. They are as follows: ƒ IllegalBoundaryException—This exception is thrown when a browser has set a boundary on a mixed multipart message whose length is different from the length of the boundary of the parent message. This error occurs when multiple files are uploaded in the same page but the boundary length separating this mixed content is different than the boundary length of the overall multipart boundary. These boundaries themselves will be different; however, their lengths must be same. ƒ MalformedStreamException—This is the catchall exception class for the MultipartStream class. It indicates one of several errors resulting from streams ending unexpectedly (perhaps because the network connection was lost), invalid stream headers (perhaps because the browser didn’t format the message according to rfc1867), or IO errors (due to an IOException). The FileUpload package includes one abstract utility class, ThresholdingOutputStream, and one implementation, DeferredFileOutputStream: ƒ ThresholdingOutputStream—This is an abstract OutputStream that triggers an event when a specified number of bytes have been written to it. This functionality is very useful for implementing actions based on certain events during the writing of uploaded file data to a FileItem’s output stream. For example, if you wanted to limit the number of bytes of a file held in memory, you would use this class to set a threshold beyond which the underlying stream would be changed from ByteArrayOutputStream to FileOutputStream. ƒ DeferredFileOutputStream—The DeferredFileOutputStream implementation extends ThresholdingOutputStream. Internally, it initially builds a ByteArrayOutputStream. This stream is used to write data until the maximum size allowed for holding data in memory is reached. The stream then automatically switches to using a FileOutputStream with a file location that is specified at the creation time of this stream. The DefaultFileItem class uses this stream internally. 2.3 Uploading files without using FileUpload Before you learn how to upload files using the FileUpload component, let’s see how to upload files without using it. Why? Well, because this exercise will help you better understand and appreciate FileUpload by exposing you to the difficulties encountered in processing uploaded files without the use of a standard handling and parsing mechanism. Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 11 Listing 2.1 is an HTML file that contains a file-upload request we’ll use for this example. Only the relevant parts are shown. Listing 2.1 simpleform.html: HTML file that uploads files without using FileUpload
#1

Bad File Upload Example

Your Name:
Your File:
#2
There is nothing remarkable about this HTML file. It contains a simple file upload request; on submit, it sends its contents as a multipart request to badfileupload.jsp, which is shown in listing 2.2. Listing 2.2 badfileupload.jsp: an alternate way of handling file uploads <%@ page import="java.io.BufferedReader, java.util.Vector, java.util.Iterator, java.io.FileOutputStream, java.io.File" %> <% BufferedReader in = request.getReader(); Vector readData = new Vector(); String lineRead; Notice method, action, and enctype File that user is uploading c Read contents of request Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF while((lineRead = in.readLine()) != null){ readData.add(lineRead); } String DELIMITER = "--"; String CRLF = "\r\n"; StringBuffer fileDataBuffer = new StringBuffer(); Iterator itr = readData.iterator(); while(itr.hasNext()){ String data = (String)itr.next(); if(data.startsWith(DELIMITER) && !data.endsWith(DELIMITER)){ data = (String)itr.next(); int index; if((index = data.indexOf("filename")) != -1){ String fullFileName = data.substring(index + 10, data.length() - 1); itr.next(); itr.next(); String fileData; while(!((fileData = (String)itr.next()).startsWith(DELIMITER))){ fileDataBuffer.append(fileData + CRLF); } FileOutputStream fos = new FileOutputStream( getServletContext().getRealPath("/") + new File(fullFileName).getName()); fos.write(fileDataBuffer.toString().getBytes()); fos.close(); } } } %> c We begin processing the request by reading the contents of the request through a BufferedReader and holding the contents read in a vector line by line. The assumption made here is that the data is encoded in the default encoding of the system on which this JSP is being run. d These two constants define what we’ll use in the later version of the code for a delimiter and a carriage return and line feed (CRLF). A delimiter is used to find the start of an encapsulation boundary and the end of the data stream. Remember that different parts in a multipart message are separated by a boundary that begins with a double hyphen. Also remember that the message ends with --boundary--. We’ve used this in the program not only to find the beginning of each part but also to find the end of the entire multipart message. e We define a variable fileDataBuffer to hold the contents of our file data. f Iterating through the data that we’ve read from the request line by line, we look for lines that start with the delimiter and don’t end with the delimiter (which would indicate the end of the request data). The d Constants for delimiter and CRLF e Define fileDataBuffer variable f Iterate through g Look for filename variable in header h Read data until delimiter i Save data to local file c Read contents of request Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 13 delimiter at the start indicates that the encapsulation boundary of a part is beginning. Note that there will be headers before the data after this encapsulation boundary. g Remembering that for a file-upload request, the Content-Disposition header contains the name of the file as the value of the variable filename, we look for this variable in the header. If it’s found, then our current part will contain the contents of a file. We extract the filename from this header and ignore the next two lines (these contain the Content-Type header and then a blank line before the data begins). h We keep reading the data in the buffer we defined earlier. The data ends when we encounter a line that contains the delimiter. Notice that we need to introduce the CRLF in the buffer because while reading it through the request, the BufferedReader strips it out with each invocation of the readLine method. i After we’ve read the data, it’s a simple matter to save the data to a local file using the filename we extracted earlier. When you run this program in a servlet container and upload files, the problems with this approach become painfully obvious. Not only will this program not work for binary files, it also won’t work when multiple file uploads are requested in the same page. Relying on parsing the headers for file information is also incomplete and doesn’t give you any confidence when you run this program. In the next section, we’ll start exploring the FileUpload component with a basic example. 2.4 Uploading files using FileUpload To begin using FileUpload, we’ll use the same basic HTML file that we used in the previous section (see listing 2.1). Listing 2.3 shows how the FileUpload component can be used to process this file. If you contrast this with the way we handled uploading files in the previous section, you’ll immediately see that FileUpload is a consistent, reliable, and much easier method of handling multipart data. Note: To run this example and the later examples, you must have the commons-fileupload-1.0.jar file in your servlet container’s WEB-INF/lib directory. Listing 2.3 basicfileupload.jsp: basic example of using FileUpload <%@ page import="org.apache.commons.fileupload.*, java.util.List, java.io.File, java.util.Iterator" %> <% if(!FileUpload.isMultipartContent(request)){ request.setAttribute("message", "Request is not multipart!"); request.getRequestDispatcher("msg.jsp").forward(request, response); return; } DiskFileUpload fileUpload = new DiskFileUpload(); List items = fileUpload.parseRequest(request); Iterator itr = items.iterator(); FileItem item; c Is request multipart? d Create instance of DiskFileUpload e Extract FileItems Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF while(itr.hasNext()){ item = (FileItem) itr.next(); if(item.isFormField()){ String fieldName = item.getFieldName(); if(fieldName.equals("username")) request.setAttribute("message", "Thank You " + item.getString()); } else { if(item.getSize() > 0){ File fullFile = new File(item.getName()); File savedFile = new File(getServletContext().getRealPath("/"), fullFile.getName()); item.write(savedFile); } } } request.getRequestDispatcher("msg.jsp").forward(request, response); %> This example demonstrates the simplest way to run FileUpload. Although it shows the way FileUpload is used to handle a single file, extending it to handle multiple files and other form elements is relatively easy; we’ll cover that in the next section. c Before we start processing the request, it’s advisable to check whether the request contains multipart data. This part of the code checks for a valid multipart request and stops any further processing if it isn’t found. If we don’t do this at this stage (before processing the actual request), FileUpload will throw an InvalidContentTypeException when the request is parsed. So, it’s always a good idea to do this before you do anything else. However, there may be situations where the developer processing the upload request is different than the person who designed the upload request page. The page author is typically a web designer who may not realize that to make a successful upload request, the ENCTYPE of the form must be multipart/form-data or that the METHOD of form submission should be POST, not GET. d To begin processing the request, we need to create an instance of DiskFileUpload. DiskFileUpload is a concrete implementation of the FileUploadBase class; it’s geared toward holding the uploaded files in memory until they’re swapped to disk when the threshold size is passed. To specify this threshold, you can do either of two things: ƒ After creating an instance of DiskFileUpload and before parsing the request, specifically set this threshold: fileUpload.setSizeThreshold(10000); // set as 10Kbytes You can similarly set the path where the files will be temporarily stored, called the repository, by calling fileUpload.setRepositoryPath("C:\\data\\temp"); ƒ After creating an instance of DiskFileUpload, and while calling the parseRequest method, pass the sizeThreshold in to the method. You must also pass in the maximum size allowed for the upload request and the temporary location of the repository: f Check if Item is a normal Form field g Retrieve field name and value h Handle uploaded files e Extract FileItems Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 15 parseRequest(request, // the original request 1000, // 1KB for the size threshold 1000000, // 1MB for the max upload request System.getProperty("user.home")); //repository location Recall that DiskFileUpload uses an internal factory to create instances of FileItems. This DefaultFileItemFactory is created when you call the constructor for DiskFileUpload. If you want to use another factory, you can change this default factory by calling the setFileItemFactory(yourFactory) method. The parseRequest method returns a List of the items processed from the request. This list contains normal form items (text fields, checkboxes, and so on) and the uploaded files. e This code iterates over the FileItems and extracts them one by one for further processing. f To easily distinguish between normal form items and uploaded files, we call the method isFormField() on each FileItem. It returns true for normal form fields and false otherwise. g Having identified this as a form field, we use the method getFieldName to retrieve the form field’s original name as set using the NAME attribute by the page designer/author. This value is used to distinguish between the various other form fields that may be present in this form and to treat them accordingly. Having found that this is the username field, we retrieve the value set for this field using the getString method and set this value in the request scope for the next page to use. If this were an uploaded file, getString would return the contents of the file as a String using the default encoding of the platform on which this program was running. If you wanted to use a specific encoding, you would use the getString(String encoding) method instead. h This section deals with items that have been identified as uploaded files. Before attempting to save the file to disk, the code checks whether the user uploaded a file by calling item.getSize() and making sure it returns a value greater than 0. This method can also be used if the user types the name of the file to upload in the text box instead of browsing for it, and makes a mistake. The browser will still send a multipart request, but the size of this file will be 0. The item.getName() method returns the full name of the file as specified by the filename in the Content-Disposition header. In some cases, this represents the full path on the user’s machine for this file. If you use the getName() method on an item that isn’t an uploaded file, it returns null. To save this file to disk, we create a File object that points to the location where we want to save the file and then call the item.write(File file) method. In this case, we save the file to the root of the current web application in which this JSP is running. What happens if you call the write method on an item that isn’t an uploaded file? A file is created that contains the value of the form item! Try it by creating a File object for the items that aren’t uploaded files. Here’s an example: item.write(new File(getServletContext().getRealPath("/"), item.getFieldName())); You’ll see a file created with the field name specified here. It contains the value of this field. The form that you’ve just seen handled contains a single text field and one input file. This is a very simple case and doesn’t cover many real-world scenarios. In the next section, we’ll show you how to handle a more complex input form that includes all sorts of form elements and type-specific file uploads. Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF 2.5 Handling a complex input form Figure 2.2 shows an input form that contains several form inputs. These include form elements that can have more than one value associated with them. In addition, more than one file-upload request is handled in one form, and the file-upload requests are specific: This form requires the user to submit an image file and a Word file. Figure 2.2 A complex input form that we’ll process with the FileUpload API As you can see, we have one text field for submitting data, one radio button set, one checkbox set, one list of multiple items, and two requests for specific files to upload. The partial HTML for this page is shown in listing 2.4. Listing 2.4 Partial HTML for the complex form

Complex Form File Upload Example

This form submits to complexform.jsp. As before, to be able to run this example, you must have commons- fileupload-1.0.jar in the WEB-INF/lib directory of your servlet engine. Listing 2.5 shows how this form is handled in complexform.jsp. Listing 2.5 complexform.jsp: handles multiple file uploads and various form elements <%@ page import="org.apache.commons.fileupload.*, java.util.List, java.io.File, java.util.Iterator, java.util.Vector" %> <% if(!FileUpload.isMultipartContent(request)){ request.setAttribute("message", "Request is not multipart!"); request.getRequestDispatcher("msg.jsp").forward(request, response); return; } FileUpload fileUpload = new FileUpload(); fileUpload.setFileItemFactory(new DefaultFileItemFactory()); List items = fileUpload.parseRequest(request); Iterator itr = items.iterator(); FileItem item; String name = "Not answered"; String team = "Not answered"; String imageFileName = "None submitted"; String wordFileName = "None submitted"; String imageFileMessage = ""; String wordFileMessage = ""; StringBuffer software = new StringBuffer(); StringBuffer books = new StringBuffer(); while(itr.hasNext()){ item = (FileItem) itr.next(); String fieldName = item.getFieldName(); if(item.isFormField()){ String fieldValue = item.getString(); if(fieldName.equals("software")){ software.append(fieldValue + " "); }else if(fieldName.equals("books")){ books.append(fieldValue + " "); c Create FileItems and parse request d Create variables e Create StringBuffers to hold multiple-select data f Determine necessary processing g Create form items for multiple-value elements Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 19 }else if(fieldName.equals("username")){ name = fieldValue; }else if(fieldName.equals("team")){ team = fieldValue; } } else { if(item.getSize() <= 0) continue; String contentType = item.getContentType(); File uploadedFile = new File(item.getName()); if(fieldName.equals("imagefile")){ if(!contentType.startsWith("image")){ imageFileMessage = "Incorrect file uploaded! File is not of type image."; } imageFileName = uploadedFile.getName(); } else { if(contentType.indexOf("msword") == -1){ wordFileMessage = "Incorrect File uploaded! File is not of type Word."; } wordFileName = uploadedFile.getName(); } } } StringBuffer buffer = new StringBuffer( "Thank You! The result of your submission is shown below:
"); buffer.append("Your Name: " + name + "
"); buffer.append("Your Image file: " + imageFileName + ". " + imageFileMessage + "
"); buffer.append("Your Word file: " + wordFileName + ". " + wordFileMessage + "
"); buffer.append("Software used: " + software + "
"); buffer.append("Favourite team: " + team + "
"); buffer.append("Books read: " + books + "
"); request.setAttribute("message", buffer.toString()); request.getRequestDispatcher("msg.jsp").forward(request, response); %> c This code illustrates a different way of handling file uploads. Recall that there are two concrete implementations of the FileUploadBase class: DiskFileUpload and FileUpload. Whereas h Check whether user uploaded file i Decide what to do with uploaded file j Inform user about invalid file Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF DiskFileUpload is geared toward swapping files in memory until they become too big and are swapped to a temporary repository, FileUpload is a more generic implementation. DiskFileUpload creates a default item factory for creations of FileItems. FileUpload doesn’t have a FileItemFactory by default; you must set it explicitly. Here, we use DefaultFileItemFactory as the factory for creating FileItems and then proceed to parse the incoming request. Warning: If you don’t set the factory for creating FileItems while using the FileUpload class, you’ll get a NullPointerException when the program is run. d We create variables here to hold the data the user has submitted and to display information about the user’s upload request. e These StringBuffers hold the data for the multiple values of the software checkboxes and the list of books. We’ll append all the values the user selects to these buffers. f We use the fieldName in the original page to determine the kind of processing to be done on each file item. For form elements with single values, we assign the element value to the correct element. For form elements with multiple values, we append to the relevant buffer all the values selected by the user. g As stated earlier, for form elements with multiple values, each value is created as a separate form item with the same name but a different value. These values are assigned to the relevant buffer. h Before processing the uploaded files, it’s a good idea to check whether the user uploaded a correct file. i Using the Content-Type of the uploaded file, we need to make relevant decisions about what to do with the uploaded file. item.getContentType() returns the Content-Type as prescribed by the Content-Type header field in the uploaded file’s part. For image files, the Content-Type always begins with image followed by the type of the image. For example, if the uploaded file were an image of type GIF, the Content-Type would be image/gif. For Word files, the Content-Type is application/msword. For files with binary content, the Content-Type is application/octet- stream. j We use the information about the Content-Type to inform the user about the validity of the uploaded file. If the file isn’t of type image, the user is so informed. k Similarly, if the Content-Type isn’t equal to application/msword, the user is so informed. You may choose to delete this uploaded file instead of storing it. 2.6 Uploading large files Transferring large files from one machine to another involves understanding not just the software residing on each end but also the mechanics of maintaining session and user interactivity. Large files require that the network and the server be capable of holding data in memory while the transfer is taking place and that the server be capable of handling and storing these large files. Further, interaction with the user as the files are being transferred is paramount: Your user must be informed about the status of their upload request and should, theoretically, be able to cancel the request. The good news is that users can decide to cancel the request to upload files by closing the browser window. Doing so closes the connection; and, since HTTP is a stateless protocol, any data transfer from the browser to the server stops immediately and won’t be resumed. You can try this by attempting to upload a large file through any of the earlier examples and closing the browser before the full file has been transferred. The browser will shut down, and you’ll get an exception in your logs similar to javax.servlet.ServletException: Processing of multipart/form-data request failed. Stream ended unexpectedly. This is normal; it’s the server’s way of letting you know that processing of a file upload request failed either because the user closed the browser or Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 21 because the connection to the user’s machine was lost. You can also cancel the request by clicking the Stop button on your browser. Doing so immediately stops the browser from sending the file contents over the server, and you’ll get the error javax.servlet.ServletException: Processing of multipart/form-data request failed. Connection reset. You can deal with these exceptions in a way that is suitable for your application. For example, if you were writing data to a database, you might need to release the connection and free other system resources on encountering these exceptions. The bad news is that keeping the user informed about the state of their request isn’t easy. There are several areas where the user may perceive inactivity. However, during these periods, several things could be happening: ƒ The browser could be preparing the data to send by copying the contents of the files to a temporary location and forming a multipart request in memory. ƒ The browser could be transferring the contents of the large file to the server. A channel is established between the browser and the server, and data is pushed over this channel. This is the period in which you as a developer have the least control. You don’t know the size of the request, you don’t know the contents of the request, and you don’t know how many files are in the request. You can make an educated guess, based on the directions from the original page author’s specifications about the likely contents of the upload request. However, this doesn’t help you inform the user about the result and status of the upload request. ƒ The server—and, consequently, your FileUpload package—could be reading the contents of the uploaded file and saving them to disk or processing them. This is the area where you have control over the uploaded files, because you can analyze them. However, you still can’t inform the user about the status until processing of the uploaded files is complete. The easiest way to deal with this situation is to spawn a separate thread to process the request and send an immediate response back to the user. This way, if a post-processing error is encountered, the user can be informed about it via email or the next time they log in for a session. The uploaded files can be rejected and discarded. 2.6.1 Dealing with server timeouts Although keeping the user informed about the status of an upload request and giving them a way of canceling the request is important, for large uploaded files it’s equally important that the server should not time out on the user’s request. It can take a long time to transfer a large file over an Internet connection. Most web servers have a timeout for incoming requests, which is the time the server waits after accepting a connection for the complete request to come through. For large files, depending on the Internet connection, this time may vary. If the server’s waiting time expires before the full contents of the user’s file have been transferred, the server will close the connection, and the user’s browser will display a generic can’t find server error. However, this may not be the only timeout you have to deal with. The following list presents the various timeouts you must set in order to accept large file uploads from users: ƒ If you’re using an HTTP server as the front end to your servlet engine, set the timeout in its configuration file. For example, if you’re using Apache HTTPD, set the Timeout value in the httpd.conf file in the conf directory to a value that you think is reasonable. ƒ Change the connection timeout value in your servlet engine. For example, if you’re using Tomcat, search for the connectionTimeout parameter in the server.xml file in the conf directory. If you set this value to –1, Tomcat will never time out incoming connections. Alternatively, if you want to keep a regular timeout for normal nonmultipart requests and have a separate timeout for multipart requests Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF when large files are being uploaded, you can set the value of the disableUploadTimeout element to false. Thus large upload requests won’t be timed out, and your regular form submissions will retain a proper timeout figure. Note: Although we haven’t yet covered writing uploaded files to databases, it’s important to know that if a large file is being transferred to a database within a regular file upload, a timeout isn’t the main issue. You’re likely to use a connection pool to maintain connections, so you’ll almost always have a connection to the database. The primary issue is the request size of the data that’s being transferred to the database. Some databases limit the size of queries that they will execute. For example, MySQL uses the variable max_allowed_packet to limit the size of a query that it will execute; the default value is 10MB. So, if a file is uploaded that’s larger than 10MB, and you try to store the file’s contents in MySQL in a BLOB column, you’ll get an error similar to Packet for query is too large (12416348 > 1048576). You can increase the value of the variable by issuing this command in a SQL window: SET GLOBAL max_allowed_packet=. In the next section, you’ll see how you can upload files and write them to a database using the FileUpload component. 2.7 Uploading files to a database Uploading files to a database isn’t much different from uploading files and storing them to disk. You can use the same code to handle the file upload, and another component can deal with creating connections to the database and writing queries to insert uploaded files. For the sake of simplicity and to illustrate these points, we’ll use a simple JSP to do everything. Before we look at the code, let’s make sure we have all the pieces for a successful upload to a database: ƒ We require a database (naturally!) and connection-pool software to connect to the database and maintain a pool of connections. This example uses MySQL (version 4.0) and Commons Database Connection Pool (DBCP), respectively. You’ll learn more about DBCP in module 9, “Pool and DBCP: creating and using object pools”; you can download it from http://jakarta.apache.org/commons/dbcp. If you’re using Tomcat as the servlet engine, you don’t need to download DBCP—it’s already available as part of a Tomcat release. ƒ We need a driver to connect to the database. To use MySQL as the database, download the MySQL Connector/J driver from http://www.mysql.com/products/connector-j/index.html. Save this driver file in the lib directory of your servlet engine. For Tomcat, this is {$tomcat-home}\common\lib. ƒ We need to define a table structure in the database to allow us to store uploaded files. This is simple enough, as long as you realize that uploaded files may be binary in nature and must be stored as the SQL datatype BLOB. ƒ We must define a Java Naming and Directory Interface (JNDI) resource in the configuration file for the servlet engine. This resource will be used to maintain the connection pool to the database and to reference it from the code. We’ll show how this is done for the Tomcat servlet engine. ƒ We need to define a web.xml file for our web application that will make the JNDI resource available to servlets and JSPs. Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 23 Let’s start by creating a table in the database for uploading files. Listing 2.6 shows a sample script that we’ll use for the rest of this example. Listing 2.6 Creating a table in MySQL for uploading files create database if not exists commons; use commons; drop table if exists chapter2_fileupload; CREATE TABLE chapter2_fileupload ( id int(11) NOT NULL auto_increment, datecreated datetime NOT NULL default '0000-00-00 00:00:00', createdby varchar(20) NOT NULL default '', fieldname varchar(100) NOT NULL default '', filename varchar(200) NOT NULL default '', file longblob, PRIMARY KEY (id) ) TYPE=MyISAM; As you can see, this table allows you to store various values associated with a file upload, including the filename and the original fieldname. The file is stored as a LONGBLOB type, which lets you store large files with sizes of around 4GB. If you’re sure that your users aren’t going to upload files that are this huge, you may change this field type to TINYBLOB (256 bytes), BLOB (64KB), or MEDIUMBLOB (16MB). To access this table in the database, we now need to define and configure the connection pool. Make the entries shown in listing 2.7 in the server.xml file of your Tomcat engine. Listing 2.7 Creating a context and defining a connection pool factory org.apache.commons.dbcp.BasicDataSourceFactory maxActive 100 Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF maxIdle 30 maxWait 10000 username root password driverClassName com.mysql.jdbc.Driver url jdbc:mysql://localhost/commons?autoReconnect=true There is nothing very remarkable about listing 2.7; it creates a database connection pool entry in the server.xml file and references it using the JNDI resource name jdbc/commons. Note: This entry is made in the separate context of Commons. A bug in Tomcat + DBCP prevents you from making the entry as a GlobalNamingResource. (We’ll talk more about DBCP in module 9.) Next, let’s define a web.xml file that will be used to declare and utilize the JNDI reference that we just created for our Commons web application. Listing 2.8 shows the relevant portions of this file. Listing 2.8 web.xml for the Commons web application Chapter 2 - FileUpload DB Connection jdbc/commons javax.sql.DataSource Container Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 25 This web.xml resides in the WEB-INF directory of the Commons web application folder. Now that we’ve covered all the basic steps required to make sure we can store uploaded files in the database, it’s time to look at the code that lets us do so. Listing 2.9 shows part of a simple HTML file that contains three fields for file uploads. Listing 2.9 databaseupload.html: uploading multiple files to a database

Database File Upload Example

Your Name:
Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 17
Your Image file:
Your Word document:
Please select ALL open source software that you have used
Struts Tomcat Log4J Lucene
Please tell us your favourite team: Indians Yankees
Please select books that you have read:
Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF
Your Name:
Your File:
Your File:
Your File:
This file is similar to all the HTML files we’ve created for uploading files. This page lets the user upload as many as three files at a time and provide a name under which these files can be stored. This HTML file is processed by the code shown in listing 2.10. Listing 2.10 databaseupload.jsp: storing uploaded files in the database <%@page import="javax.sql.DataSource, javax.naming.*, java.sql.*, org.apache.commons.fileupload.*, java.util.List, java.io.File, java.util.Iterator" %> <% Context ctx = new InitialContext(); DataSource ds = (DataSource)ctx.lookup( "java:comp/env/jdbc/commons"); if (ds != null) { Connection conn = ds.getConnection(); if(conn != null) { PreparedStatement stmt = conn.prepareStatement( " INSERT INTO chapter2_fileupload " + " (datecreated, createdby, fieldname, filename, file) " + " VALUES (now(), ?, ?, ?, ?)"); DiskFileUpload fileUpload = new DiskFileUpload(); List items = fileUpload.parseRequest(request); Iterator itr = items.iterator(); FileItem item; String userName = ""; while(itr.hasNext()){ item = (FileItem) itr.next(); if(item.isFormField()){ userName = item.getString(); } else { if(item.getSize() > 0){ stmt.clearParameters(); stmt.setString(1, userName.length() == 0 ? "RemoteUser" : userName); stmt.setString(2, item.getFieldName()); c Create InitialContext and perform lookup d Return connection object e Insert file into database f Parse request g Check for username field i Fill in default field value h Populate fields in PreparedStatement Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 27 stmt.setString(3, item.getName()); stmt.setBytes(4, item.get()); stmt.addBatch(); } } } stmt.executeBatch(); if(stmt != null) stmt.close(); } if(conn != null) conn.close(); } %> c To access the DataSource that we’ve defined in server.xml and web.xml, we need to load the default context defined by the JVM we’re running in; and within this context, we need to use the key defined in web.xml to find the DataSource for connecting to the database. This piece of code does this by creating an InitialContext and using the key jdbc/commons to look up the DataSource for this example. d Once we have access to the DataSource, we can retrieve preset connections to the database from within this pool. This code returns a valid connection object from within the pool. e Here we use a PreparedStatement to insert the uploaded file and details about the uploaded file into the database. We’ll populate the fields in this PreparedStatement later. f By now, you should be familiar with this piece of code. It creates a DiskFileUpload object to parse our incoming request and uses the parseRequest method to parse it. g While iterating through the List of FileItems, the only form field is the username field. Recall that multipart requests form body parts in the order in which they’re present in the original form. So, we can be confident that the first item in the list will be the username field. We still check for it by calling the isFormField method. h This is where we populate the fields in the PreparedStatement with the form values and the uploaded file (only if the user actually uploaded a file). Since the original form includes three file-upload fields, we use a batched PreparedStatement to insert the uploaded files into the database in a single connection and query. i Since users using the form may decide not to enter their name, it’s a good idea to fill in a default value for this field. In practice, this would probably be a unique ID that identifies the remote user. 2.8 Summary The FileUpload Commons component allows you to handle upload requests in an effective and efficient manner. It does this by providing a flexible interface and by conforming to rfc1867. In this module, you’ve seen how FileUpload handles multipart requests and how it lets you manage individual form items. FileUpload has an extensible API for representing uploaded form items. These form items may be regular form fields or they may be uploaded files. FileUpload distinguishes between these by providing a simple flag. FileUpload can handle both complex forms and large files. Fields in a complex form are still represented as individual file items, and all selected values for multiple value fields are represented in the order in which they’re selected. Large file upload requests are handled efficiently by swapping them to disk. However, the success of large file uploads is partially based on external parameters. h Populate fields in PreparedStatement Licensed to Tricia Fu 28 JAKARTA COMMONS ONLINE BOOKSHELF FileUpload lets you retrieve the contents of an uploaded file as bytes. As a result, you can store these files in the database as BLOB fields rather than on disk. In the next module, we’ll look at the Net component, which is an Internet protocol suite library for various common protocols like Finger, FTP, Telnet, and others. Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 29 Index DefaultFileItem, 9 DiskFileUpload, 8 InvalidContentTypeException, 10 SizeLimitExceededException, 10 UnknownSizeException, 10 createItem method, 10 DeferredFileOutputStream, 11 file transfer, 2 file transfer process, 2 FileUpload accessing DataSource to store files in database, 28 API, 7 basics, 6 connection reset error, 22 exceptions and utility classes, 10 FileItem, 8, 9 FileItemFactory, 10 FileUploadBase, 8 FileUploadException, 10 handling complex forms, 16 handling timeouts on large files, 22 MultipartStream, 8 parsing and processing API, 8 representation API, 9 saving files to disk, 16 setting temporary file location, 15 setting the factory for FileItem creation, 20 setting upload limit, 15 simple file upload, 14 stream ended unexpectedly error, 21 ThresholdingOutputStream, 11 uploading files without FileUpload component, 11 uploading large files, 21 using Content-Type header, 21 writing files to database, 23 FileUploadBase, 8 get method, 9 getFileItemFactory method, 10 getInputStream method, 9 getString method, 9 IllegalBoundaryException, 10 isFormField method, 9 isMultipartRequest method, 8 MalformedStreamException, 10 multipart message, 3 boundary, 3 example, 3 multiple values, 5 MultipartStream constructor, 8 MySQL Connector/J, 23 MySQL JDBC Driver. See MySQL Connector/J parseRequest method, 8 readBodyPart method, 9 readHeaders method, 9 repository, 8 Request for Comments definition, 2 RFC definition. See Request for Comments rfc1521. See multipart message rfc1867, 1, 2 ACCEPT attribute, 3, 5 browser requirements, 4 Content-Disposition Header, 4 Content-Transfer encoding, 4 Content-Type header, 4 ENCTYPE attribute, 2 MAXLENGTH attribute, 6 mechanism, 2 METHOD attribute, 3 omissions, 5 SIZE attribute, 5 VALUE attribute, 5 setRepository method, 10 setSizeMax function, 10 setSizeThreshold method, 10 skipPreamble method, 9 System.getProperty method, 9 write method, 9 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Handling Protocols with the Net Component Vikram Goyal MODULE MANNING 3 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 3 Handling Protocols with the Net Component 3.1 Getting to know the protocols................................................................................................................................... 1 3.2 The Net API ............................................................................................................................................................ 13 3.3 Creating a multiprotocol handler ............................................................................................................................ 23 3.4 Summary................................................................................................................................................................. 32 Index ............................................................................................................................................................................. 33 The Net component brings together implementations for a diverse range of Internet protocols. It’s a feature- rich component and had been around a long time before it was open-sourced through Jakarta Commons. (It was originally built by ORO, Inc.) Most programmers have to deal with only a subset of the vast array of Internet protocols. The ones that are used most often include Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), and, perhaps, the mail protocols: Simple Mail Transfer Protocol (SMTP) and Post Office Protocol (POP3). However, several lesser-known protocols are also in use. The Net component features both the well known and the not so well known protocols in an easy-to-use interface. In this chapter, we’ll explore the Net component and look at the protocols it makes available. This will help you understand the basics of these protocols before you begin using them. Next, we’ll examine the structure of the Net component and explore its API. Finally, we’ll use all this information to develop a multiprotocol handler that uses each of the protocols. 3.1 Getting to know the protocols A protocol, in the real world, is a way something should be done. By adhering to a protocol, you’re following a strict procedure for doing certain things—for example, you shouldn’t take the last helping of dessert without asking a dinner guest if they’d like to have it! In the computer world, a protocol is a previously agreed upon way for two machines to exchange information with each other. Without a protocol to guide and define how machines talk to each other, there would be anarchy and confusion. A protocol may determine several things: how the two machines will handshake, how the transmitting machine will initiate transfer of data, what the format of the data will be, what the recipient machine will do to indicate that it has received the data, what the recipient machine will do to indicate an error condition, and so on. 3.1.1 TCP and UDP: the building blocks Before we talk about the protocols covered by the Net component, let’s discuss the protocols that are the building blocks of these protocols. A discussion of network technologies is incomplete without a basic understanding of the TCP and UDP protocols, which are the low-level protocols that act as the message carriers for the high-level, application-specific protocols. Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF The Transmission Control Protocol (TCP) is specified in rfc793.1 It’s a reliable, connection-oriented, complex protocol: ƒ It’s reliable because data sent by TCP can’t be lost: The protocol marks all packets of information that it transmits with a sequence number. This allows for retransmission of packets that go missing, because the receiving end can ask for those packets by looking up the sequence numbers of the packets it receives. ƒ It’s connection-oriented because a connection must be established between machines before data can be exchanged between them. ƒ It’s complex because it requires error correction and retransmission policies built in at the protocol layer itself. The User Datagram Protocol (UDP) is specified in rfc768. It’s a nonreliable, connectionless, simple protocol: ƒ It’s nonreliable because packets sent via UDP may or may not reach their destination: The packets may get lost along the route due to incorrect addressing or checksum errors. ƒ It’s connectionless because each packet is self-contained with the source host and destination host address. No direct connection stream is established between the source and the destination machines. ƒ It’s simple because it doesn’t require a connection, and no error checking or retransmission of packets is involved. This also means that if an application-level protocol is using UDP as the underlying protocol for transmission of data, it must be ready to accept loss of data or handle error checking and retransmission of lost data on its own. It helps to think of TCP as a continuous phone line communication channel and UDP as a standard postal letter communication channel. With TCP, a continuous full duplex (two-way) channel is established before any data is exchanged, similar to numbers being dialed before a phone call begins. With UDP, a letter is marked with a destination address and is posted; it will probably reach its destination, but it may not. Both TCP and UDP are protocols on the Transport layer of the standard TCP/IP four-layer model, which is shown in figure 3.1. For this reason, these protocols are low-level protocols as compared to, for example, HTTP, which is considered a high level protocol. Figure 3.1 The four-layer TCP/IP model, showing the different levels of protocols 3.1.2 High- and low-level protocols Whenever an application-level protocol is used, it’s likely to be running on top of other low-level protocols. Consider HTTP. Although this protocol is designed to transfer information between a web browser and a 1 See module 2, section 2.1.1, for the definition of an RFC. Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 3 web server, it relies on an underlying protocol (TCP) to gather together the packets to be transmitted to the server. HTTP (built into the web browser) at the client end talks with TCP on the client’s machine. TCP, in turn, talks to a protocol at a still lower level (IP), again on the client’s machine. This protocol talks across the physical layer to the server and transmits the packets. Conceptually, the web browser’s HTTP is taking with the HTTP on the web server end. Physically, the information goes through a series of protocols before being transmitted. A protocol at the top end of the chain is a high-level protocol—it’s the one making all the initial requests (if you ignore the user). The protocols it talks to in order to complete the user requests are low-level protocols. Note that HTTP, TCP, and IP are software protocols; they’re conceptual entities. For data transmission to occur, it requires the presence of a hardware protocol. Thus, continuing our example, IP talks with the local hardware/network combination to transmit the information. This combination is an actual physical or wireless entity that transmits the information over network channels. Examples of these network entities— which are hardware protocols—include Ethernet, AppleTalk, 802.11g, and so on. The protocols covered by the Net component are all high-level protocols like HTTP. They’re called application-level protocols because they’re designed to help applications on different machines to transfer information. These protocols don’t deal with issues that primarily involve creating and transferring packets, error recovery, and checksums; lower-level protocols such as TCP and IP (which aren’t covered by the Net component) take care of these tasks. 3.1.3 Protocols of the Net component As we said in the previous section, the Net component doesn’t provide support for low-level protocols. It only deals with application-specific protocols like HTTP, SMTP, and POP3. In this section, we’ll review the supported protocols. Let’s start with the simplest protocol: Discard. Discard protocol The Discard protocol is the simplest protocol you’re likely to encounter. It does what its name suggests—that is, a server running this protocol discards anything it receives. This protocol is useful for debugging and for measuring signal strength. It’s defined by rfc863. This protocol operates in both TCP and UDP mode, which means that you can use it either to make a connection on TCP port 9 and discard any data it receives in the connection, or to listen for packets on UDP port 9 and discard any packets it receives. Since TCP is a connection-oriented protocol, the connection is maintained until the client closes the connection either abruptly or by sending a close signal. The server doesn’t terminate the connection after discarding the data. Character Generator (CharGen) protocol The Character Generator protocol (CharGen for short) is slightly more useful than Discard; at least it responds with some data! This simple protocol, defined by rfc864, is used as a debugging and measurement tool for clients in a client-server environment. The server listens on port 19 for incoming connections. Any client that is to be tested makes a connection on this port. Once a connection is made, the server ignores any incoming data from the client and begins sending a long and unending stream of character data back to the client (hence the name, character generator). The type of data to be sent back isn’t specified, but the RFC recommends the pattern shown here: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefgh "#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghi #$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghij $%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijk Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF %&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijkl &'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklm '()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmn ()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmno )*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnop *+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopq +,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs -./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst ./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu /0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv 0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw 123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx 23456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy 3456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz This pattern consists of a series of 72 ASCII printable characters in a line. The first line consists of the first 72 printable characters out of a possible 95 printable characters in ASCII. The next line ignores the first character (!) on line 1, begins with the second character ("), and prints the next 72. This continues until the last printable character ( ), has been printed. The line after that starts again from the first printable character, and the pattern continues. The CharGen protocol can be run as either a TCP- or a UDP-based protocol. When you run it as TCP- based, the server listens for TCP connections on TCP port 19. Once a connection is established, the stream of data just described is sent until the client terminates the connection. When you run it as UDP-based, the server listens for UDP packets on UDP port 19. When a packet is received on this port, it’s ignored; an answering packet is sent back, containing a random number (between 0 and 512) of characters. No pattern is involved in the UDP version of the CharGen protocol—the server just keeps sending packets until the client stops sending packets on UDP port 19. CharGen, along with the Echo protocol, has been the vanguard of the hacker community. Hackers have used it to mount denial of service (DoS) attacks for several years. It’s an especially vulnerable tool because the server doesn’t stop sending data until the client closes the connection. A malicious hacker can simulate simultaneous multiple connections to the service and leave them running. Worse, a hacker can cripple two machines at the same time by invoking CharGen on one machine and redirecting the output to another machine on its CharGen port. Both machines will keep sending data to each other indefinitely, thus bringing down network traffic and, eventually, the machines themselves. You’ll find hardly any public servers today that have an ongoing CharGen service. Daytime protocol The Daytime protocol, defined in rfc867, is very simple. Clients use it to connect to a server running this protocol on port 13. When connected, using either TCP or UDP, the server ignores the input data. For TCP, the server sends an ASCII string containing the date and time back to the client. Unlike with the Discard and CharGen protocols, for TCP, the connection is closed after the server has sent the ASCII string back to the client. For UDP, the server sends a packet containing the date and time as an ASCII string. The date and time are local to the server, and the preferred format is Weekday, Month Day, Year Time- Zone. Echo protocol The Echo protocol is defined by rfc862. This protocol allows network engineers to test the robustness and reliability of their networks by sending back the data received on port 7; hence the name Echo protocol. Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 5 This protocol also works in both TCP and UDP modes. In TCP mode, once a connection is established on port 7, whatever data is received is sent back to the originating client, and then the connection is closed. In UDP mode, any packet that is received is sent back after the source and destination host information has been reversed. Time protocol The Daytime protocol sends the time in a human-readable format. This may not be satisfactory for machines, which can’t interpret the information. The solution to this problem is the Time protocol, defined in rfc868, which lets servers send the current time in a machine-readable format. This format is a 32-bit binary number: the number of seconds that have passed since midnight on January 1, 1900 GMT. This protocol lets clients quickly connect to a time server on the Internet (or on their own intranet) and correct their time based on the time sent back by the server. This process raises reliability concerns about the time server, which must calculate and maintain the correct time. Note: Several Internet time servers are available. For example, you can use those supplied by the U.S. government. To test the time-related protocols we’ve discussed so far (Daytime and Time), you can use the servers supplied by the National Institute of Standards and Technology (NIST). A list of servers is maintained at http://www.boulder.nist.gov/timefreq/service/time-servers.html. Whois protocol The Whois protocol was defined in rfc954 (in 1985) as a directory service protocol for looking up individuals in the then-growing network of people. The protocol is a bit like HTTP, in that it follows a query/response architecture whereupon the server closes the connection. A query is sent to a server to look up an individual registered with the directory service on the server on port 43. The server responds with details about the individual (if the individual is registered with the server’s directory) and closes the connection. The Whois protocol has, of course, grown considerably since then: It now looks up not only individuals but also (and perhaps more importantly) information associated with domain names. By making a Telnet connection on port 43 to the Whois servers at Network Solutions, you can get registration information for any domain. (To test this, make a connection on port 43 to the server whois.networksolutions.com. Once you’re connected, type a domain name and press Enter. The server will return the registration information to you and close the connection.) You can use a web page to gather the Whois information instead of making a Telnet connection. The RFC for Whois specifies several parameters that you can send to the servers to narrow the results of your search. However, not all Whois servers implement all the parameters specified in the RFC. The best way to determine which parameters are implemented is to send a query containing a single question mark (?). The result will be the list of commands that the particular Whois server understands. Figure 3.2 shows an example response from the Whois server at Network Solutions. Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF Figure 3.2 Help file for Network Solutions’ Whois protocol Finger protocol The Finger protocol is perhaps inappropriately named. It’s defined in rfc1288 as a protocol that, in essence, is very similar to Whois but differs in that it’s primarily targeted toward getting information about user(s) of a remote system rather than a domain. (Note that it’s possible for some Whois protocols to return information about people registered within the system they’re running on, thereby negating the use of the Finger protocol.) The Finger protocol defines the mechanism by which a client machine connects to a server on port 79. The connection is made by TCP; after connecting, the user on the client machine requests information about either a single user or all the currently logged-on users on the remote server. The server responds by processing the input query; it interfaces to a backend system to gather information about users, returns the data in a human-readable ASCII format, and closes the connection. What information the server returns about its users is left up to system administrators to decide. They may choose to include useful information like email addresses, contact phone numbers, and departments. All this information must be specified to the user interface program the Finger protocol talks to, so it can retrieve the information and send it back to the client. Individual Unix users can control part of this data by altering what is known as a plan file; however, this isn’t available on Windows. To see information about a specific user, the client makes a connection on port 79 and types the name of the user on a line by itself. The server then, if possible, returns whatever information it can gather. However, Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 7 after connecting to the server on port 79, if the client sends an empty line request, it indicates to the server that the user wants information about all currently logged-on users. The server must then respond with the details of all the connected users, which may include facts like when the users connected, what terminal they connected from, and how long they’ve been active. Of course, some server administrators may decide that disclosing such information is a security risk and may prohibit the Finger protocol from returning anything in response to such a request. The Finger protocol specifies that in such a case, the server must return a message that discourages further such requests, similar to Finger online user list denied. Since the data returned is human readable and, for the most part, written by humans, various other uses of the Finger protocol have come into practice. For example, by fingering for the user quake at the US Geological Survey servers at gldfs.cr.usgs.gov, you can get information about the current global seismicity in a human-readable format. Telnet protocol The initial use of the Telnet protocol was to provide dumb terminals with access to a mainframe computer. This idea is central to network computing where one powerful machine is timeshared by several users. Of course, workstations are no longer dumb, and access to central computers on a network is also done through PCs. Telnet is a type of protocol that lets networked machines remotely log in to servers. The protocol defines properties for a terminal emulation program, which allows the networked machines to act and behave as if the user was running commands directly on the server. However advanced, the Telnet protocol doesn’t allow for graphical user interaction and was designed for text-based processing. Telnet is defined in rfc854. Several other RFCs extend this protocol by defining options that enhance the user interface behavior of the terminals a remote client sees when connected via Telnet. Options are usually negotiated at the beginning of a session and can be overridden later by mutual agreement between the client and the server. A client making a connection on port 23 of the server machine over a TCP connection initiates a Telnet session. The server starts the session by invoking the simplest form of transaction window (full duplex), where both sides (client and server) agree on the minimum set of functionality supported by the client. This minimum set of functionality—known as the Network Virtual Terminal (NVT) mode—is a 7-bit ASCII stream with escape sequences for control functions. Once an NVT is in place, more options can be set up at any time (for example, the client may require the support of a character set other than ASCII). Setting up more options involves the client and the server negotiating for each option. Either side can initiate this process by asking the other side a simply query: either WILL XXX or DO XXX, where XXX is an option that either side wants to perform. Using WILL XXX conveys that the sender wishes to start using this option. The other side may accept or reject it by responding with DO XXX or DON'T XXX, respectively. Using DO XXX represents the sender’s desire for the recipient to start doing the requested option. The answer to this request is either WILL XXX or WON'T XXX, as the case may be. To prevent an endless loop, if one side receives a request for an option that it’s already implementing, it ignores the request. An example is the Echo option, which is defined in rfc857. With the Echo option in place, the end that is implementing this option is expected to transmit or echo the data characters it receives back to the end that sent those characters. Either end of a Telnet connection can implement the Echo option. Because a Telnet connection is initiated in an NVT mode, the Echo mode is off by default; either end can initiate it by sending either the WILL ECHO or DO ECHO command. Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF Rlogin protocol Telnet isn’t the only remote login protocol available for networked computers. Rlogin is available on BSD Unix, Linux, and OS X, and is specified in rfc1282. However, Telnet is the overwhelming choice for remote logins; several versions of Unix disable rlogin in favor of Telnet or implement special security restrictions on top of rlogin. Vulnerabilities in the rlogin protocol have also been responsible for its lack of widespread use. Rlogin lets users on a networked machine connect to the server on port 513. It differs from Telnet in that it allows trusted hosts to log in without requiring a password. But how does a host become trusted? Well, special configuration on the server side lets you mark specific IP hosts as trusted by making an entry in the hosts.equiv and rhosts files. An rlogin session initiates with a client sending a request for connection to the host. Once a connection is established, the client sends four lines to the server: The first line contains an empty string, the second line contains the username of the client, the third line contains the username on the server for the client, and the last line contains the client’s terminal type and speed. The server responds by sending a zero byte, which indicates that it’s ready to start working. If the client can’t be authenticated using the hosts.equiv and rhosts files, the server terminates the connection. File Transfer Protocol File Transfer Protocol (FTP) is perhaps the oldest Internet protocol still around. This first rudiment of this protocol was designed in 1971 as rfc114. It has grown via several more RFCs since then; the latest version in use today was specified in rfc959 in 1985. We’ll concentrate on that RFC. FTP complements the Telnet protocol and is designed for efficient transfer of files from one networked computer to another. Files can be transferred both ways in a client/server mode with the help of a small set of FTP commands. The process by which an FTP session is initiated and maintained is displayed in figure 3.3. The sequence of events is summarized here: 1. A user starts a user agent to make a connection to a remote server. The user agent, also called the user protocol interpreter (PI), establishes a connection with the remote server on port 21. This connection is called the control connection, and it’s separate from the connection that would be established for actual transfer of files. This control connection is similar to the user agent making a Telnet connection with the remote server, albeit on port 21. 2. The user initiates the transfer of a file from their machine to the server. The user PI passes the request to the remote server as a standard FTP command over the control connection. This request may contain several parameters, including the data type, the data transfer mode, and the actual data transfer request. The PI on the remote server interprets the command and sends a reply to the user PI indicating the readiness of the server data transfer process (DTP) to start receiving data from the user data transfer process. 3. The server DTP runs on a separate port (this port number is 1 less than the FTP port over which the control connection has been established) in a normally passive mode. This means that this data transfer connection is established permanently, but the data transfer may not actually happen. This passivity is enabled by default and can be turned off. 4. The file is transferred on the data connection from the client to the server. Again, note that this transfer takes place on an independent connection, which is separate from the control connection between the user PI and the server PI. 5. Once a file has been transferred, the user may either send other files and repeat the process or terminate the connection by sending a QUIT signal. The server replies with a code and a message, as it does for all commands sent from the user PI. The connection is then terminated. Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 9 Figure 3.3 FTP transfer process Trivial File Transfer Protocol Although FTP is an efficient and reliable mode of transferring files between computers, it requires a separate connection to be set up for control commands. There are several scenarios in which this may not be suitable. For example, transferring information to bootless workstations would be inefficient because of narrow bandwidth and limited channels. Trivial File Transfer Protocol (TFTP), defined in rfc783, is a simpler version of FTP and is suitable to use in these circumstances. It’s used mainly for bootstrapping diskless workstations and routers. TFTP differs from FTP in several ways that enable quicker, simpler transactions. For starters, TFTP uses UDP rather than TCP for data transmission. The utilization of the simpler protocol reduces TCP overhead. Any TFTP transfer begins with a request to read or write a file, and this request serves as a request for a connection. Thus a single UDP packet is responsible for both the initial connection request and file transmission details. Once the connection is established, the file is transferred in UDP packets, which are 512 bytes long. To close the connection and signal the end of file transmission, the sender sends a packet that is less than 512 bytes. The recipient of the file signals acknowledgement of various packets with an ACK packet. The recipient must acknowledge each packet sent by the sender. The sender waits to send a packet until it receives confirmation for the packet it sent previously. If a timeout occurs because of a lost packet, the recipient retransmits the last packet it sent (whether an ACK or a data packet). If an error occurs due to insufficient privilege or data or transfer errors, an ERROR packet is sent. Once this ERROR packet is received, the transmission of further packets stops immediately. The ERROR packet isn’t acknowledged. Network News Transfer Protocol Before the explosion in email, another messaging system—Network News Transfer Protocol (NNTP)—made information sharing between users around the world a matter of simply typing a message and posting it on Usenet. Usenet was (and, really, still is) the place where disparate communities come to vent their anger, to rejoice over silly things, to flame other users, and to share knowledge. NNTP makes Usenet possible. (Historically, Usenet relied on Unix to Unix Copy [UUCP], but NNTP is the preferred mechanism today.) Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF NNTP is described in rfc997. It’s similar to FTP or SMTP in that it works on the basis of a request/response architecture. Different news servers talk to each other, using this architecture to share and propagate messages. A user agent like Microsoft Outlook uses the same architecture to log in to a central news server and retrieve groups and messages. How does it work? Who owns a group? Better still, what is a group? Let’s begin with the last question. A group, in NNTP terms, is any online forum where users may post messages of their own and read messages posted by others. Messages aren’t addressed to any particular individual; instead, they’re addressed to the group as a whole, even though they may be in response to messages that other users post. A group can be created for any topic. But then who owns the group? Nobody and everybody. It’s almost like asking who owns the Internet. Although certain companies may start their own groups (for example, to support their products) and therefore, by extension, are responsible for starting that group, they don’t own the group after that. Each group must be created on a newsgroup server that may be local to the company/individual that’s trying to create it. Once it’s created, it’s propagated to other newsgroup servers using NNTP. Each news server tries to stay up to date with new groups and new messages in each group by querying the news servers that it is connected with. However, not all news servers want to remain up to date with all the new newsgroups and messages. (You may be interested in discussions about the latest series of “Survivor,” but not all users of Usenet may be.) Similar to new newsgroup creation and propagation, a new message in a newsgroup is propagated by your local news server talking to its nearest news servers. Your news server first asks these servers if they have the group in which you’ve posted a new message (using the NEWGROUPS command). If the group doesn’t exist, the other servers may create it based on their rules for creating new groups. Either way, your news server then seeks any new messages it wants for any of the groups it has (using the NEWNEWS command). It receives any new messages and stores them so you can see them as and when you require. Finally, your news server posts your new message to its nearest neighbors, which then continue the cycle with their neighbors. Your message ends up in all the news servers around the world (provided it isn’t filtered for unacceptable content!). NNTP runs on port 119. A user agent like Microsoft Outlook connects with a news server on this port to send and receive messages. News servers talk to each other on this port to propagate groups and messages. Simple Mail Transfer Protocol Simple Mail Transfer Protocol (SMTP) transfers email messages between different servers using a request/response architecture. The services that implement this protocol run on port 25 and are accessed not only by other servers but also by user agents, such as Microsoft Outlook. To easily explain the SMTP protocol, it’s instructive to consider an example scenario. Suppose you’re a mail sender, and you don’t have a user agent like Microsoft Outlook to send email to a friend, Joe Blow. You know that Joe Blow’s email address is jblow@manning.com. Your email address is sender@someisp.com. You know that when you signed up with your Internet Service Provider, it gave you a list of the host names for the SMTP and POP servers. Let’s say you still have this list and that it has your SMTP server set at smtp.someisp.com. To send an email to your friend at Manning, you now need to connect to this SMTP server. How do you do this? Well, using SMTP is like using Telnet: You can make a connection over a TCP network by talking to the designated port and then issue commands. For our example, the steps are as follows: 1. To make a connection to this server, open a command window and type telnet smtp.someisp.com 25 Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 11 2. If the connection goes through, you’ll see a greeting from someisp similar to this: 220 smtp.someisp.com ESMTP Sendmail 8.12.10/8.12.10; Fri, 12 Dec 2003 12:49:20 +1000 (EST) The first three numbers (220) identify a message code, which indicates that the SMTP service is ready on this port. Further message strings identify the type of SMTP service and the date and time on the server. 3. To send an email, you begin by typing the following in the Telnet window (the command is terminated by a new line [CRLF]): MAIL FROM:sender@someisp.com The MAIL command is an instruction to the SMTP server that it should get ready for a new mail transaction. The FROM argument specifies that this email comes from sender@someisp.com. This is also the return path of this email, in case the message is returned for some reason. This field is important, because most ISPs won’t allow an email to be sent from their SMTP server if the user or the recipient doesn’t have a mailbox at the server. You can probably still find some servers that relay messages, but these are few and far in between. (The problem is that spammers use this technique to hide their own tracks.) A response to this command from the SMTP server may look like this: 250 2.1.0 sender@someisp.com... Sender ok 4. Next, you need to specify the recipient of this message. You do this by typing the following (ending with a CRLF): RCPT TO:jblow@manning.com The server responds with a similar message for the sender: 250 2.1.5 jblow@manning.com... Recipient ok 5. Now that both the sender and the recipient have been specified and the server is ready to send the message, you can begin the text of the message by typing the following: DATA A simple DATA command indicates to the server that the user is ready to send the contents of this message. The server responds with something similar to 354 Please start mail input. Notice the message code 354: It’s an intermediate reply code indicating that the server is ready to accept the message and that all lines from now on will be considered part of the message. 6. The message should end with period (.) on a line of its own. Thus your next few lines of input may look Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF like this as you write your message: This is a test message. Please ignore. . Once the server receives the last line of data with a single period on a line by itself, the server understands that you have finished writing the message and that it should now try sending the message to the recipient. The server may respond to indicate successful receipt of the message with a message and status code like 250 Mail queued for delivery. 7. To deliver the message to the recipient, your SMTP server tries to have a similar dialogue with the destination SMTP server; or, if it can’t make a direct connection to the final destination SMTP server, it tries to talk to other SMTP servers to find one that can relay the message. The servers talk to each other on port 25 until the message is either delivered to the final destination or (if it can’t be delivered) returned to you at the address you specified as part of the MAIL FROM command. There is, of course, more to SMTP than we’ve shown in this simplistic scenario. But the guts of the system are as they’ve been described here. To get a list of all the commands and possible replies, refer to SMTP RFCs: rfc821 and rfc2821 (a recent consolidation of the original rfc821). Post Office Protocol 3 Post Office Protocol 3 (POP3) is the third enhancement of this popular protocol. It’s defined in rfc1725 and is used for storage and retrieval of email messages. Like a post office, this protocol allows users’ emails to be stored at a central location for pickup and delivery later. Users can log in any time to the central server and retrieve a list of messages waiting for them. Again, as with a real post office box, users must provide authentication in order to retrieve their messages. However, this authentication is transmitted in clear text over the Internet, where it’s liable to be snooped at by unsavory characters. Similar to SMTP, the POP3 service lets user agents connect to it and send commands. The POP3 service sends responses to these commands, which are preceded by either +OK or –ERR for successful and unsuccessful operations, respectively. Unlike SMTP, POP3 requires user authorization before anything else can be done. So, after connecting to the POP3 service on port 110, the user agent must send authorization information using the USER and PASS commands on separate lines. After a successful authorization, the server opens the authenticated user’s mailbox for a transaction. The user can then retrieve the waiting messages or get the number of messages using the RETR or STAT command, respectively. Messages can be listed using the LIST command, which returns the message ID and the size of the message in bytes. Both the RETR and LIST commands take as an argument the message number on the server. If no argument is supplied, then the command is expected to apply to all messages. A user agent normally follows a set sequence when retrieving messages from the POP3 server: 1. The user agent connects and authenticates itself to the server by using the USER and PASS commands. 2. It uses STAT to find out how many messages are waiting for the current authenticated user. 3. It uses the LIST command to determine the individual message numbers. 4. Using these message numbers, the user agent retrieves each message using the RETR MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 13 number> command. 5. When each message has been retrieved, the user agent issues the DELE command to delete each message on the server so that the next time the user agent logs on, the message is no longer there. Phew! We’ve discussed a total of 14 protocols. That’s a lot of theory without any information about how the Net component implements these protocols. In the next section, we’ll look at the structure of the Net component and show you the relationship between different packages and classes. 3.2 The Net API The Net API contains all the classes that make up the Net component. No external classes or libraries are required in order to use it. Most of the classes in the API are divided into two groups: One group deals with TCP-based protocols and the other with UDP-based protocols. Figure 3.4 shows a static UML diagram that depicts the classes in the TCP group. Similarly, figure 3.5 shows a diagram for the UDP group classes. These diagrams and the classes they contain are discussed in later sections. 3.2.1 The overall picture As you can see from the diagrams, all protocol implementation classes derive from an abstract class: SocketClient (for TCP) or DatagramSocketClient (UDP-based protocols). These abstract classes provide the basic operations required for connecting to remote hosts, setting timeouts, accessing input and output streams, and closing connections, for TCP; and opening local ports and setting timeouts, for UDP. However, you don’t create instances of these sockets yourself; the sockets are created using factory classes. You can either let the SocketClient and DatagramSocketClient classes manage their factories internally, which they do by using the default factory classes; or you can create your own SocketFactory to custom- implement the way the sockets to remote hosts are created. The default implementations of the SocketFactory interface, DefaultSocketFactory and DefaultDatagramSocketFactory, are simple wrapper classes around the Java-supplied Socket and DatagramSocket classes. Since all protocol implementation classes derive from either the SocketClient class or the DatagramSocketClient class, they inherit the common methods of connecting and disconnecting from remote servers for TCP and opening and closing sockets for UDP, whichever protocol it may be. (Note that since UDP-based protocols are connectionless, the socket that is being opened is on your machine.) A typical TCP protocol implementation class operates as shown here: XXXProtocol.connect(remoteHost); --- do some protocol specific work --- XXXProtocol.disconnect(); Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF Figure 3.4 TCP-based static class structure for the Net API Figure 3.5 UDP-based static class structure for the Net API Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 15 Both the connect and disconnect methods are inherited from the SocketClient class. Similarly, a typical UDP protocol implementation class operates like this: XXXUDPClient.open(); --- do some protocol specific work --- XXXUDPClient.close(); Again, the DatagramSocketClient class defines the open and close methods. This way, the implementation classes don’t have to worry about connection and disconnection methods. All they need to do is to look after implementing the basics of their protocol. 3.2.2 The Net API structure The Net API was designed around the principle of giving basic access to the low-level programmer and giving broad-based, easy abstractions to the high-level API user. Thus the API is designed around the packages: You can either access the low-level classes directly, for any protocol; or work around these classes and use the higher-level abstractions. Most developers fall in the second category of users and use the low-level classes only when they want to tweak the parameters more than the default implementations. The low-level class is named after the protocol it implements. For example, SMTP is implemented in the low-level mode in the class SMTP in the package org.apache.commons.net.smtp. As we said before, you probably won’t ever need to call or work on this class; you’ll almost always use the high-level SMTPClient in the same package. Note: Note the Client at the end of the protocol name that defines this class. All high-level protocol implementations have Client added to the end of the class name. This distinguishes between the low- and high-level classes for a protocol. Only complex protocols like POP3, NNTP, FTP, TFTP, SMTP, and Telnet have this distinction, though. The simpler protocols, like CharGen and Daytime, have only low-level classes called CharGenTCPClient and DaytimeTCPClient, respectively. Almost all protocol implementations have their own package in the Net component API. Table 3.1 lists each protocol and the corresponding package(s) in which its implementation classes are found. Table 3.1 Package information for protocols supported by the Net API Protocol Package BSD rlogin org.apache.commons.net.bsd FTP org.apache.commons.net.ftp, org.apache.commons.net.ftp.parser NNTP org.apache.commons.net.nntp POP3 org.apache.commons.net.pop3 SMTP org.apache.commons.net.smtp Telnet org.apache.commons.net.telnet TFTP org.apache.commons.net.tftp All other protocols org.apache.commons.net Support and utility org.apache.commons.net.io, org.apache.commons.net.util In the next few sections, we’ll look at some of the complex protocol package structures, as shown in table 3.1. Protocols like CharGenTCPClient are self-explanatory after you’re familiar with the complex ones. Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF org.apache.commons.net.bsd This package contains the following three classes, which implement the rlogin protocol discussed earlier: ƒ RExecClient—The base class, which provides functionality for executing commands on BSD Unix, equivalent to the rexec() command. ƒ RCommandClient—Extends RExecClient and provides facilities for executing rcmd(). This command allows remote trusted hosts to connect to a Unix server without requiring explicit authorization. An added constraint on the rcmd() utility restricts client machines to connect to the server only between port numbers 512 to 1023. The RCommandClient class enforces this restriction internally without your having to specify it. ƒ RLoginClient—A simple extension of RCommandClient. It logs in using the trust relationship established in the superclass. Notice that since all three classes’ names contain the word Client, the classes can be used individually at a higher level. org.apache.commons.net.ftp and org.apache.commons.net.ftp.parser The FTP classes have been divided into two packages as an enhancement of the older FTP implementation of Net. The new package org.apache.commons.net.ftp.parser contains classes that implement the FTPFileListParserImpl abstract class. These classes make it easier to handle different file lists. (We’ll talk about this later.) The main package org.apache.commons.net.ftp contains the FTP and FTPClient classes, which are the low-level and high-level implementations, respectively, for the FTP protocol. In addition to these two classes, there are several classes that supplement FTP behavior. The FTPCommand and FTPReply classes are constant classes that contain FTP command and reply codes, respectively, as constant integers. In addition, the FTPReply class contains several convenience methods that allow you to test and group the status of the reply code sent by an FTP server. Thus, if you wanted to test whether the reply sent by the server was preliminarily positive, you’d use the method boolean isPositivePreliminary(int reply). The complete list of methods is as follows: ƒ isPositivePreliminary(int reply)—The server sends a positive preliminary reply only if part of the original request has been completed and the user should expect another reply before sending a new command. All reply codes that start with 1 are examples of such a reply. ƒ isPositiveCompletion(int reply)—Determines whether the server sent a reply indicating that the original command was successfully completed. All reply codes that start with 2 indicate that the request was successfully completed. ƒ isPositiveIntermediate(int reply)—Determines the success of part of a series of commands sent to a server. The reply codes for such responses start with 3. For example, the server would send such a reply code for the USER command (if the user specified by this command was accepted by the server). The server would still wait for the user to send the PASS command. ƒ isNegativeTransient(int reply)—Indicates that the original command couldn’t be executed but that if the user sent the command again, it would probably succeed. These reply codes start with 4. Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 17 ƒ isNegativePermanent(int reply)—Indicates that the original command couldn’t be executed and that retrying the command is also likely to fail. All reply codes beginning with 5 are examples of such a reply. A file on the FTP server is represented using the FTPFile class. This class is an abstraction of the way a file could be represented over several servers and contains methods that let you gather meta information about the file. This information represents the attributes of a file as it’s stored on the server. For example, there are methods that allow you to gather the size of the file or its name or group. You can also get information about the permissions associated with the file. More information may be available from some FTP servers about the files on the server, so you may want to subclass the FTPFile class and add those enhancements. But how does the Net component get information about these files on the server? The Net component runs on the client side, and nothing in rfc959 allows an FTP server to disclose information about individual files. The Net component overcomes this problem by asking for a list of files in a directory by using the FTP command LIST and then parsing the reply from the server to create an array of individual FTPFile classes that more or less represent individual files on the server. This is where the parser package comes into picture. When the Net component was initially released, the FTP package contained only a single parser called DefaultFTPFileListParser. This parser implemented the FTPFileListParser interface, which contains a single method: public FTPFile[] parseFileList(java.io.InputStream listStream)throws java.io.IOException The DefaultFTPFileListParser class is suitable for most cases. However, using the parseFileList method of this default parser led to problems in cases where there were a number of files in a single directory: Because this default parser created FTPFile objects for each file in a directory, there were memory and performance issues with large listings. To resolve this problem, a new set of parsers was implemented. These parsers don’t create FTPFile objects until they’re utilized by the end user. Instead, a whole directory listing is represented using the FTPFileList class, which contains methods that let you extract or iterate over the files. In addition, a new interface called FTPFileEntryParser was defined. To maintain backward compatibility, FTPFileEntryParser was implemented using a class called FTPFileListParserImpl. This class implements the old interface (FTPFileListParser) and the new one (FTPFileEntryParser) and provides default functionality for the old interface. The new parsers are collected in the org.apache.commons.net.ftp.parser package, and all of them extend the FTPFileListParserImpl class. Finally, there is an exception class called FTPConnectionClosedException, which is dedicated to catching premature or unexpected closing of connections. This exception extends the IOException class and is thrown when the FTP server closes the connection due to inactivity. To make sure this exception isn’t thrown, you should regularly use the method sendNoOp(). This method sends the NOOP command to the remote server, keeping the connection active; it does nothing else. org.apache.commons.net.nntp This package contains classes for the NNTP implementation. The classes in this package are as follows: ƒ NNTP—A low-level NNTP implementation that provides direct access to NNTP commands ƒ NNTPClient—A high-level NNTP implementation that accesses the NNTP class and provides convenience methods for use by developers ƒ NNTPCommand—Contains NNTP command constants Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF ƒ NNTPReply—Contains NNTP reply constants ƒ NewsgroupInfo—Provides information about a newsgroup ƒ NewGroupsOrNewsQuery—A utility class that issues queries for new groups and new news ƒ ArticlePointer—Contains information about the location of newsgroup articles ƒ SimpleNNTPHeader—Lets you create message headers ƒ NNTPConnectionClosedException—An exception that is thrown when the server sends a 400 error code, indicating a closed connection for a command the client is sending to it While using this package, you’ll almost always work with the higher-level NNTPClient class. The low-level class provides convenience methods to send various NNTP commands directly; for example, the newgroups(date, time, GMT, distributions) method lets you send the NEWGROUPS command to a server and returns the reply code as a constant. There are similar methods for all the NNTP commands. The NewsgroupInfo class represents information about a newsgroup. Methods in this class allow you to get the count of articles in a newsgroup (getArticleCount()), get the first and last article numbers (getFirstArticle(), getLastArticle()), get the newsgroups name (getNewsgroup()), and determine whether posting of articles in this group is allowed (getPostingPermission()). A NewsgroupInfo instance is created as the result of invoking either the listNewsgroups() or listNewNewsGroup(NewGroupsOrNewsQuery query) method on the NNTPClient class. NewGroupsOrNewsQuery is a handy class because it abstracts the tasks of working out times, dates, and group names and leaves you to specify the date from which you want to retrieve the new groups or new messages. Recall that NNTP lets you retrieve new groups and messages by issuing the NEWGROUPS and NEWNEWS commands. However, these commands must be issued with respect to a timeline from which a group or message is considered new. Further, you may want to narrow the search to groups starting with a particular name. The NewGroupsOrNewsQuery class helps formulate such queries by allowing you to specify the date and group (called the distribution) you want to search for new groups or messages. Here’s an example of how you would use this: NewsGroupsOrNewsQuery query = new NewsGroupsOrNewsQuery(new GregorianCalendar(99, 3, 3), false); query.addDistribution("comp"); NewsgroupInfo[] newsgroups = client.listNewgroups(query); This code lists all newsgroups that are more recent than midnight of March 3, 1999 and that contain the prefix comp. The ArticlePointer class is a small structure that contains information about the location of an article. It’s used as a result holder when the STAT command is issued; this command returns the message (article) number and the message’s unique ID. The SimpleNNTPHeader class is used when an article (message) needs to be posted to a group and the appropriate headers for the article must be created in a format that is acceptable to the group. It contains methods to add the newsgroup to which the message is being posted and any other headers that may be required. A SimpleNNTPHeader instance is created by passing in the From address and the subject of the message being posted. org.apache.commons.net.pop3 This package contains the POP3 implementation classes. There is the POP3 low-level class and POP3Client, which is a high-level abstraction of the POP3 protocol and represents the client side of a POP3 protocol. The class POP3Command contains all the POP3 commands as constants. The POP3Reply class contains all the Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 19 replies as constants. Since the POP3 protocol supports only two types of responses, ERR and OK, this class contains only these two constants. Finally, the class POP3MessageInfo is used to represent meta information about messages stored on the POP3 server; it uses the properties number, size, and identifier. However, the class doesn’t provide any public getter methods for retrieving this information. You have to use the public class-level fields to get to this data. The information that can be gathered using this class differs based on the command used to create an instance, as illustrated in table 3.2. Table 3.2 POP3MessageInfo properties and their meanings Property Command issued Number Size Identifier STAT Returns the number of messages in the mailbox Returns the size of the mailbox in bytes Null LIST Returns the message number Returns the size of the message in bytes Null UIDL Returns the message number Undefined Returns the message’s unique ID There are no exception classes in this package. This class relies on the Net package’s generic exception class, MalformedServerReplyException. This is a catchall exception that is thrown whenever a protocol implementation class can’t understand the reply given by a server. It’s thrown only in extreme cases. A best effort is made to understand the replies of a server, and a lot of leniency is given in the interpretation of the replies. org.apache.commons.net.smtp As expected, this package contains the SMTP class and the SMTPClient class. The SMTP class provides direct interaction with the SMTP server via a variety of methods, each of which maps to an SMTP command. For example, the mail(String reversePath) method lets you send the SMTP MAIL command to the server. There are similar methods for all other SMTP commands. The SMTPClient class takes a higher-level view of creating and sending email messages via the SMTP interface. This is why you won’t find a method that directly mimics the SMTP MAIL command; the user- friendly method public boolean setSender(java.lang.String address) throws java.io.IOException essentially does the same thing. On the other hand, if you want to send a simple message to one person, you can use the method public boolean sendSimpleMessage( String sender, String recipient, String message) throws IOException where all the commands are set for you using the underlying SMTP class methods. The SMTPCommand and SMTPReply classes provide the constant fields for the SMTP commands and replies, respectively. This package also contains a class similar to SimpleNNTPHeader called SimpleSMTPHeader, which lets you easily create headers for sending messages. For example, to create a minimalist header, you can do the following: Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF SimpleSMTPHeader header = new SimpleSMTPHeader("johndoe@manning.com","Vikram@manning.com" "Book Status"); header.addCC("manager@manning.com"); In two lines, you can create a header that would otherwise take at least four lines because of the four header values. The first parameter in the SimpleSMTPHeader class is the sender’s email address, the second is the recipient’s email address, and the third is the subject of the email. The SimpleSMTPHeader class provides two more methods: addCC(String emailAddress) adds a standard CC header, and addHeaderField(String headerField, String value) adds any other nonstandard header. The RelayPath class is an encapsulation of the relay paths for mail messages. A relay path represents the different hosts a mail message may pass through en route to its final destination, or it may represent the different hosts the message may go through if it was returned. These are known as forward paths and reverse paths, respectively. Rfc821 specifies relay paths as follows: @host1,@host2:final@host3.com This relay path specifies that to reach the final mailbox at host3.com, the mail may have to go through host1 and host2. If specified as a reverse path, this may represent that if the message can’t be delivered to the specified host, it may be returned to final@host3.com through host2 and host1. To use the RelayPath class, you use the constructor to create the final destination address and then add all the hosts through which the email must travel in either the forward or reverse direction. An example is shown here: path = new RelayPath("final@host3.com"); path.addRelay("host1"); path.addRelay("host2"); Finally, the package contains an exception class, SMTPConnectionClosedException. This exception is usually thrown when a timeout has occurred and the server sends a reply code of 421, indicating that the service isn’t available. org.apache.commons.net.telnet The Telnet package is slightly different from the rest of the packages in that it doesn’t provide a low-level class called Telnet. Instead, to use it in its basic form, you use the TelnetClient class, which implements the most basic form of the Telnet protocol using the NVT format. On top of the NVT format, you can add any of the supplied options or create your own option classes by extending the TelnetOptionHandler class. The supplied option classes EchoOptionHandler, SimpleOptionHandler, SuppressGAOptionHandler, and TerminalTypeOptionHandler extend this class. The TelnetOption class contains constants for option negotiations as defined by rfc855. The TelnetCommand class houses the Telnet commands as constants. There is one exception class, called InvalidTelnetOptionException, which is thrown when an option that can’t be registered because of an invalid option code is added to the TelnetClient to extend its behavior by registering additional options. Finally, the interface TelnetNotificationHandler is used by clients to inform them when option negotiation commands are received. Option negotiation is the method by which a Telnet session is extended to achieve extra functionality. It’s such an important concept that it’s defined in a separate RFC (rfc855). The TelnetOptionHandler class lets you implement this RFC by providing methods to perform negotiations for adding options on to an NVT client. It’s an abstract class, and a concrete option class like the EchoOptionHandler must Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 21 implement the basics of option negotiation by overriding the methods startSubnegotiationLocal, startSubnegotiationRemote, and answerSubnegotiation. org.apache.commons.net.tftp The TFTP package is slightly different from the packages we’ve been discussing so far. TFTP is based on UDP, and therefore the base class in this package (TFTP) derives from the DatagramSocketClient class. However, as with other packages, you probably won’t need to use this class directly. Instead, you’ll use the TFTPClient class for almost all operations. The most important methods in this class are those that let you send and receive files without having to deal with the low-level details of creating and receiving packets. These methods include several variations of sendFile, where you provide an InputStream to specify the file to send, the hostname to specify the host to send to, and a constant to specify the mode in which to send the file. Similarly, to receive a file, you use variations of the receiveFile method; almost every parameter is the same as for sendFile, but instead of an InputStream, you specify an OutputStream to write the received file to. Before you can send or receive files, you use the methods derived from the DatagramSocketClient class to open and close sockets. You can also set the default timeouts using the setDefaultTimeout method of this class. The TFTPPacket class and its subclasses represent the different types of packets that can be created for transfer of file using the TFTP protocol. Five types of packets can be created. However, you probably won’t ever need to use these packets directly. The base class for packets, TFTPPacket, is abstract and defines an abstract method called newDatagram that subclasses must override for packet-specific creation of datagrams. It also defines a factory method called newTFTPPacket, which is the only way to create a new packet from an incoming or outgoing datagram. If the datagram doesn’t contain a proper TFTP packet, this method throws the TFTPPacketException. There are five implementations of the TFTPPacket class: TFTPReadRequestPacket, TFTPWriteRequestPacket, TFTPDataPacket, TFTPAckPacket, and TFTPErrorPacket. They corresponding to the five types of TFTP packets. TFTPReadRequestPacket and TFTPWriteRequestPacket derive from TFTPRequestPacket, which combines common functionality for packet creation. org.apache.commons.net.io and org.apache.commons.net These two packages aren’t protocol specific; rather, they contain a set of utility classes that are used internally within the Net component. The org.apache.commons.net package contains the implementations of several protocols that don’t justify separate packages because only one class is required to implement them. These classes are the protocol implementations of the CharGen, Daytime, Discard, Time, Echo (both TCP and UDP), Finger, and Whois (TCP only) protocols. The usage of these classes is the same as for the rest of the protocols; you’ll see this usage shortly when we implement a multiprotocol handler. The org.apache.commons.net package also contains a listener and a listener support class for firing events when protocol commands are received and sent via the protocol implementation classes. The listener interface is defined in ProtocolCommandListener, which has the following two methods: protocolCommandSent(ProtocolCommandEvent event) and protocolReplyReceived(ProtocolCommandEvent event). Clients that need to take specific actions on these events can be notified by implementing this listener. The ProtocolCommandSupport class is responsible for registering/unregistering listeners and firing events. A class wishing to receive notifications of these events registers with ProtocolCommandSupport class by calling the Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF addProtocolCommandListener(ProtocolCommandListener listener) method and specifying a listener that should receive event notifications. The org.apache.commons.net.io package contains several IO-related classes that are specific to socket-based operations. The SocketInputStream and SocketOutputStream classes represent streams that originate from a socket, with the advantage that you don’t need to have a handle to the original socket. Closing these streams also closes the original socket. Suppose you’re transferring files using TFTP between different operating systems. Further assume that one peculiarity of these differing operating systems is the way they define a new line. On one system, the new line is defined as CR (the ASCII symbol for a carriage return). On the second system, the new line is defined using CRLF, which is a combination of carriage return and line feed. When transferring files between these two systems, you must take care to synchronize the new-line characters; in addition, conversions will be necessary on both ends to make sure the new lines appear in the right place for each operating system. The actual transfer takes place using a standard called netascii. The org.apache.commons.net.io package contains classes to convert between these characters and provide consistency: ToNetASCIIOutputStream and ToNetASCIIInputStream to convert to the consistent form of netascii (where CR and LF both are used) from a platform-specific format, and FromNetASCIIOutputStream and FromNetASCII- InputStream to convert from netascii to a native format. DotTerminatedMessageReader and DotTerminatedMessageWriter are similar streams for character-based data. Recall that for protocols like SMTP, POP3, and NNTP, a period (.) on a line by its own is significant because it represents the end of data. A line that starts with a period and contains other characters needs to be treated with special care. For these lines, an extra period is added to the start of the line to discriminate between the end of data and lines that are part of the data. The DotTerminatedMessageReader and DotTerminatedMessageWriter streams help read and write streams taken from clients of these protocols. DotTerminatedMessageReader reads messages from a server and strips duplicate periods for lines starting with two dots. It also ensures that you can’t read past the end of the data on encountering the period. DotTerminatedMessageWriter does the opposite of these functions: It adds an extra period for lines that start with a dot. It also sends the only period on a line by itself when you signal the end of data. Finally, the Util class provides methods to copy the contents of one stream to another. You can transfer the data from one stream to anther using the static methods provided in this class. A default buffer of size 1KB is used internally for transfers, but you can use any size that you want by overriding it. As each buffer sized data is transferred from the source to the destination stream, interested parties can listen in and act on it by registering listeners, which receive events when a buffer load of data is transferred or copied. To enable this, Net provides two classes—CopyStreamEvent and CopyStreamAdapter—and an interface. The interface, CopyStreamListener, provides two methods: bytesTransferred(CopyStreamEvent evt) bytesTransferred( long totalBytesTransferred, int bytesTransferred, long streamSize) The first method is invoked by a CopyStreamEvent source when a buffer load of bytes is transferred from one stream to another. The second method is similar, except that instead of transferring information about the data being sent through an event, it transfers it through this method itself, for performance reasons. The CopyStreamEvent class returns the same information: bytes transferred, size of the stream being copied, and total bytes transferred. The CopyStreamAdapter class allows interested parties to register/unregister CopyStreamListeners and informs them whenever bytes are transferred using the bytesTransferred methods. Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 23 It’s time to cut some code! Let’s use our knowledge of all the protocols and the Net component to create a multiprotocol handler. 3.3 Creating a multiprotocol handler The best way to learn something is to put it into practice. To understand the Net component better, we’ll develop a multiprotocol handler in this section. Protocol Runner is an application that allows us to exercise several protocols using one interface. To begin, let’s discuss the basic requirements of such an application; then we’ll describe some of its features. The Protocol Runner application lets the user connect to remote servers for TCP-based operations. Once a connection is made, the application enters a transaction mode based on the protocol being used. For some protocols, like CharGen, Daytime, and Time, a simple connection with the remote server will suffice, and an expected result will be shown in the server response field. For others, like Echo, Finger, and Whois, a successful connection will end in a query being thrown open to the user for further input. For example, when an Echo connection has been made, the user will have to enter characters to be sent to the server so that the server may echo them back. The remaining protocols—FTP, NNTP, POP3, and SMTP—require user interaction on a higher level. This means that once the connection is made, the application will let the user either send commands to a remote server via a list of predefined commands, which are reflective of the protocol being used, or send commands to the server through a free-flow command area. This process should be similar to typing commands on a Telnet screen and receiving the response from the server. Some of the predefined commands require further interaction from the user, and these should be gathered from the user using pop-up screens. The same is true for UDP-based operations, except that no connection will be made. For the CharGen, Daytime, and Time protocols, no user input is expected; the user should be able to send packets to the remote server by clicking a Start button. For Echo, once the user clicks Start, the application should ask the user for characters to send. Finally, for TFTP, the user should get a choice of two preset commands: one to send a file and the other to receive a file. After the user selects either one, the application should prompt the user for the location of the files. The list of desired features for such an application is as follows: ƒ The application lets the user select a protocol and connect to a remote server using that protocol. The protocols are divided into two screens to allow for TCP- and UDP-based protocols. ƒ The user specifies the remote server to connect to, or the remote server to send packets to (in case of UDP). ƒ For TCP screens, the application automatically populates a port field with the default port for the current protocol selection. The user can change this port if desired. There is no such default population of the remote server field; the user must specify this port before a connection can be made or packets sent. ƒ The application lists the protocols alphabetically at the top of the screen for both TCP and UDP. For TCP, the protocols to be listed are CharGen, Daytime, Echo, Finger, FTP, NNTP, POP3, SMTP, Time, and Whois. For UDP, the protocols are CharGen, Daytime, Echo, TFTP, and Time. In both cases, the application defaults to the CharGen protocol. ƒ The application provides fields for entering the remote server details. The application accepts both domain names and resolved IP addresses. For TCP, there is a port field, as explained earlier, with a Connect button next to it. Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF ƒ The application has an area where the server response is shown. This area is noneditable and shows fresh data for each new connection. This means that a history of server responses need not be shown for different protocols. This is the same for TCP and UDP. ƒ Whenever possible, the application uses the built-in client methods in the Net component to execute commands. For example, if the user wishes to change the working directory for the FTP protocol, the changeWorkingDirectory(String pathName) method of the FTPClient must be used. This is far from an exhaustive set of features. Several things can be added. For example, we haven’t considered error handling or logging mechanisms. The point here is to illustrate the use of the Net component without including any excess baggage. With this in mind, let’s look at some of the assumptions and omissions we’ve made: ƒ As we said before, there is no proper error-handling mechanism except for catching exceptions locally and reporting them back to the user. This serves as the logging mechanism as well. ƒ We’ve omitted the use of the Telnet and BSD-based rlogin protocols. The rest of the protocols adequately reflect the use of the Net component. Having said this, it’s not difficult to implement these on top of the other protocols. ƒ Timeouts have been ignored, and the default timeouts are used. If this becomes a concern, especially for UDP-based protocols, you can set the timeout by calling setDefaultTimeout(int timeout) method of the DatagramSocketClient class. ƒ Since content is more important than style, we used the popular NetBeans open-source IDE (http://www.netbeans.org) to develop the GUI responsible for the application. This allowed us to concentrate on writing the plugs around this GUI rather than worry about the tedious development of the GUI. For that reason, in the code explanations, we ignore the GUI code and concentrate on the actions behind it. You can download the full source code and the final executables for this application from the book’s web site. Figure 3.6 shows the final application. This figure shows a message being posted to an NNTP server via a pop-up window. 3.3.1 Application classes The Protocol Runner application is based on three classes: ProtocolRunner, ProtocolHandler, and ClientFactory. ProtocolRunner is the main class that contains the GUI code. The ProtocolHandler and ProtocolRunner classes share the burden of working behind the scenes of this GUI to handle the various details of connecting, maintaining state, and executing commands. The Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 25 Figure 3.6 Protocol Runner in action: posting a message to an NNTP server ProtocolHandler class is responsible for handling free-flow TCP commands and several simpler protocols. To maintain simplicity, the ProtocolRunner class handles several protocols and their predefined commands that require input from the user. The ClientFactory class is a singleton factory for creating instances of various protocol clients. 3.3.2 Creating the GUI As we said before, the GUI isn’t the main code in this application. Therefore, this section describes the GUI’s elements without getting into much detail about its construction. The main screen is divided into two parts using a JTabbedPane (a Swing class for creating tabbed windows). The first (default) part deals with TCP, and the second deals with UDP. The TCP part is further divided into five sections. The top section contains a single element: a JComboBox containing the list of all the protocols available for execution. The next section contains two JTextfields for the server name and port number, and a JButton titled Connect. This button’s label changes to Disconnect once a connection has been made. The section on the left after this section changes based on the protocol that is currently selected using the combo box from the first section. If the protocol doesn’t offer any preset commands, this section says No Preset Commands. Simply Connect. Otherwise, this section shows a list of preset commands available for the particular selected protocol using JRadioButtons grouped together in a ButtonGroup. The section on the right of this list of preset commands contains a free-flow command section area using a single JTextField. Users can type a command in this area and press Enter to execute this command. The final section is a vast JTextArea that shows the responses from the server. Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF The UDP part is similar, but it doesn’t contain the port number field and the Connect button. It also doesn’t let the user type free-flow commands. Instead, the command section contains a single Start button that is replaced with a list of preset commands for the TFTP protocol. There are several other elements that aren’t visible when the application opens. These elements include the JPanels that contain the set of preset commands for FTP, NNTP, POP3, and SMTP. Several dialogs are also used; they’re instances of JDialog and are reused across various protocols. These dialogs are the loginDialog, the singleValueDialog, and the multipleValueDialog for handling input from the user for login, for single text field input, and for multiple-value input, respectively. 3.3.3 Handling the commands We’ll concentrate on the TCP protocols to explain the way the commands are handled. Handling of the UDP protocols is almost identical. When the user selects a protocol from the list, two things happen: The port number field changes to reflect the default protocol for the currently selected protocol, and the preset command section changes to reflect available preset commands for the selected protocol. This is done in the updateTCPPanels method of the ProtocolRunner class, as shown in listing 3.1. Listing 3.1 updateTCPPanels method of the ProtocolRunner class private void updateTCPPanels(int index){ presetPanel.removeAll(); boolean isPanelDifferent = false; switch(index){ case 4: { presetPanel.add(ftpPanel); isPanelDifferent = true; portNumberField.setText("21"); break; }// ftp case 5: { presetPanel.add(nntpPanel); isPanelDifferent = true; portNumberField.setText("119"); break; } // nntp case 6: { presetPanel.add(pop3Panel); isPanelDifferent = true; portNumberField.setText("110"); break; } case 7: { presetPanel.add(smtpPanel); isPanelDifferent = true; portNumberField.setText("25"); break; } // smtp case 0: { portNumberField.setText("19"); break; Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 27 } // chargen case 1: { portNumberField.setText("13"); break; } // daytime case 2: { portNumberField.setText("7"); break; } // echo case 3: { portNumberField.setText("79"); break; } // finger case 8: { portNumberField.setText("37"); break; } // time case 9: { portNumberField.setText("43"); break; } // Whois } if(isPanelDifferent) { commandBox.setEnabled(true); } else { commandBox.setEnabled(false); presetPanel.add(defaultPanel); } repaint(); } There is nothing magical about this code. It removes everything that might have been there from the preset command panel and adds a new panel based on the protocol that has been selected. If no new panel needs to be added, it adds the default panel. It also disables or enables the free-flow command box based on the same criteria. A connection attempt isn’t made until the user clicks the Connect button. Once the user clicks this button, and after some error checking, the application tries to make a connection, as shown in listing 3.2. Once a connection is made, the ProtocolHandler class is called into action using the handleTCPProtocol method. Listing 3.2 ActionPerformed method for the Connect button in the ProtocolRunner class private void tcpConnectButtonActionPerformed(ActionEvent evt) { if(connected) { try { handleDisconnect(); }catch(IOException e) { showError(e.getMessage()); } return; Licensed to Tricia Fu 28 JAKARTA COMMONS ONLINE BOOKSHELF } String server = serverNameField.getText(); String port = portNumberField.getText(); int portNumber; try{ portNumber = Integer.parseInt(port); }catch(NumberFormatException nfe){ showError("Invalid port number: " + port); return; } if(server == null || server.trim().length() == 0 || portNumber < 1){ showError("Invalid connection attributes"); return; } try{ tcpClient = ClientFactory.getTCPClientInstance( tcpSelectionBox.getSelectedIndex()); tcpClient.connect(server, portNumber); }catch(SocketException se){ showError(se.getMessage()); return; }catch(IOException ioex){ showError(ioex.getMessage()); return; } connected = true; tcpConnectButton.setText("Disconnect"); tcpSelectionBox.setEnabled(false); jTabbedPane1.setEnabledAt(1, false); try { ProtocolHandler.handleTCPProtocol( tcpSelectionBox.getSelectedIndex(), tcpClient, this); }catch (IOException io) { showError(io.getMessage()); io.printStackTrace(); } } The important pieces of code in this method are as follows: c The instance variable tcpClient is instantiated with a client instance that is fetched out of the ClientFactory class. This factory method is passed the current selected index that marks the current selected protocol. The factory returns an instance of that protocol class using a singleton pattern, thus c Fetches the tcpClient instance d ProtocolHandler handles commands Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 29 keeping only one instance of each client. Note that the tcpClient variable is marked as a SocketClient. Once an instance of the client is obtained, the SocketClient method connect() is used to connect to the remote server. d Once a successful connection has been made, the ProtocolRunner class delegates to the ProtocolHandler class the task of doing something with the connected tcpClient. The handleTCPProtocol method is responsible for handling interaction immediately after a successful connection. The handleTCPProtocol method exists in two forms and is overloaded. One form takes a command string as an additional argument to allow free-flow commands. Other than that, the method is a traffic router that, based on the protocol being used, calls an internal method to handle the details of the protocol. For example, consider listing 3.3, which shows how the CharGen protocol is handled in the handleTCPChargen method. Listing 3.3 handleTCPChargen method private static void handleTCPChargen( CharGenTCPClient client, ProtocolRunner runner) throws IOException { BufferedReader chargenInput = new BufferedReader(new InputStreamReader(client.getInputStream())); int lines = 500; while(lines-- > 0) { runner.getTCPServerResponse().append( chargenInput.readLine() + "\r\n"); } } Recall that by this stage, a connection has already been made by the ProtocolRunner class. This method is called after the connection has been made to handle the details of the concerned protocol, in this case TCP CharGen. The CharGenTCPClient class provides a method to get the input stream from which the server- generated data can be read. Using this, the method prints the first 500 lines from the data in the server response area. Listing 3.3 shows how a simple protocol is handled. Listing 3.4 shows how a more complex protocol like FTP is handled. Listing 3.4 handleFTP method private static void handleFTP( FTPClient client, ProtocolRunner runner, String commandString) throws IOException { if(commandString == null) { if(!FTPReply.isPositiveCompletion(client.getReplyCode())) { runner.handleDisconnect(); return; c Check reply from server Licensed to Tricia Fu 30 JAKARTA COMMONS ONLINE BOOKSHELF } } else { client.sendCommand(commandString); } runner.getTCPServerResponse().append(client.getReplyString()); } Note that this method is used only for handling free-flow commands and once the user has connected to an FTP server. c The assumption is that if the command string is null, the user has just connected; therefore the reply string from the server is checked to see whether the server sent a positive completion code. If not, the server protocol layer didn’t accept the connection, even though the socket layer made a successful connection, and the ProtocolRunner class is asked to handle the disconnection. d If the command string isn’t null, it means that after a successful connection, the user has typed a free-flow command. It’s sent to the server here. e The response from the server is shown in the server response area. To handle the preset commands, which the ProtocolRunner class depicts as ButtonGroups of JRadioButtons, the relevant JRadioButton action-performed method is called. Each preset command’s action-performed method determines whether the user needs to provide data in order for the command to be completed. This data is expected to be in one of three formats; therefore, one of the three JDialogs— loginDialog, singleValueDialog, or multipleValueDialog—is used to gather the information before the command is executed in the action-performed method of the dialog’s OK button. Listing 3.5 shows part of the action-performed method for the OK button of the singleValueDialog method for FTP commands. Listing 3.5 FTP part of the action-performed method for the singleValueDialog OK button if(selection == 4) { if("New Directory:".equals(valueLabel)) { ((FTPClient)tcpClient).changeWorkingDirectory(valueText); } else if( "New Directory Name:".equals(valueLabel)) { ((FTPClient)tcpClient).makeDirectory(valueText); } else if( "Full path to local file:".equals( valueLabel)) { FileInputStream input = new FileInputStream(valueText); ((FTPClient)tcpClient).storeFile( new File(valueText).getName(), input); input.close(); } else if("Remote file:".equals(valueLabel)) { FileOutputStream output = new FileOutputStream( System.getProperty("user.dir") + System.getProperty("file.separator") + valueText); ((FTPClient)tcpClient).retrieveFile(valueText, output); output.close(); d Send user’s typed command e Show response from server Differentiate between possible commands Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 31 } tcpServerResponse.append( ((FTPClient)tcpClient).getReplyString() + "\r\n"); The JDialog, singleValueDialog, is shown to the user by the action-performed method of a radio button after changing the label on the singleValueDialog to reflect the input required from the user. An example is shown in listing 3.6. This same label text is then used as shown at the beginning of listing 3.5 to differentiate between the various possible commands and execute them accordingly. The last line shows the response from the server in the server response area. Listing 3.6 Handling the change-working directory for FTP private void jRadioButton1ActionPerformed(ActionEvent evt) { if(!checkValidConnection()) return; singleValueLabel.setText("New Directory:"); showDialog(singleValueDialog); } In a similar vein, look at listing 3.7, which shows the handling of the OK button for the NNTP multipleValueDialog. In this case, the only multiple-value entry required for NNTP is when the user is posting a message to the NNTP server. Listing 3.7 Handling posting a message to an NNTP server if(selection == 5) { writer = (DotTerminatedMessageWriter) ((NNTPClient)tcpClient).postArticle(); if(writer == null) { tcpServerResponse.append( ((NNTPClient)tcpClient).getReplyString() + "\r\n"); return; } SimpleNNTPHeader headers = new SimpleNNTPHeader(fromText, subjectText); headers.addNewsgroup(variableText); writer.write(headers.toString()); writer.write(messageText); writer.close(); ((NNTPClient)tcpClient).completePendingCommand(); tcpServerResponse.append( ((NNTPClient)tcpClient).getReplyString() + "\r\n"); } The NNTPClient is instructed that the user wishes to post an article to a news server. The client responds with a DotTerminatedMessageWriter. Before writing data to this writer, it needs to be sure the writer isn’t null. A simple NNTP header is constructed by specifying the sender and the subject of the message. This Response from server Licensed to Tricia Fu 32 JAKARTA COMMONS ONLINE BOOKSHELF data is gathered from the multipleValueDialog screen. In this header, the newsgroup to which this message is being posted is specified using the addNewsgroup() method. This information is also gathered using the multipleValueDialog, where the user must specify the newsgroup. The message is then posted to the newsgroup by first sending the headers and then sending the actual message, which is again gathered from the user using the multipleValueDialog. Finally, the client is instructed to complete this posting of the message. In a nutshell, this is all there is to our multiprotocol handler. The UDP section is handled similarly, and so are the other TCP protocols. Download the source code and the classes from the book’s web site, and give the application a go. 3.4 Summary We’ve covered a lot of ground in this module. The Net component comprises a set of libraries that handle a suite of Internet protocols. These protocols are covered in well-represented packages and offer flexibility in the way a user interacts with these protocols. The module started with a look at all the protocols covered by the Net component. We examined the basics of these protocols to be sure you understood them before we tackled the Net component’s API. The protocols discussed are both TCP and UDP based. We also explained the differences between TCP and UDP. The Net component provides its functionality through a well-designed API. This API is divided into various packages; most of the packages handle one protocol each. We looked at how and when to use the classes in these packages. The SocketClient class provides the basic infrastructure for TCP-based protocols, and DatagramSocketClient provides the same for UDP-based protocols. You should now understand how the NNTP, SMTP, POP3, and FTP classes differ from the rest of the classes and their similarities as well. In the final section of this module, we created a multiprotocol handler called Protocol Runner. This single application can become a point of reference for using the Net component within another application. We created this application using three classes that share the burden of handling protocol commands. The application contains code for handling most of the protocols you’ve seen in this module and which exercise the Net component. The next module covers the Digester component, which lets you read XML configuration files and invoke predefined rules for initialization of Java objects. Licensed to Tricia Fu MODULE 3: HANDLING PROTOCOLS WITH THE NET COMPONENT 33 Index Character Generator protocol definition, 4 issues with hacking, 5 recommended pattern, 4 CharGen Protocol. See Character Generator protocol data transfer process, 9 datagrams, 3 Daytime protocol definition, 5 preferred format, 5 Discard protocol definition, 3 dumb terminals, 7 Echo protocol definition, 5 File Transfer Protocol definition, 9 Net classes, 17 process, 9 Finger protocol definition, 7 plan file, 7 hardware protocol, 3 high-level protocols, 3 low-level protocols, 3 National Institute of Standards and Technology, 5 Net API, 14 structure, 16 TCP classes, 15 UDP classes, 16 API org.apache.commons.net, 23 org.apache.commons.net.bsd, 17 org.apache.commons.net.ftp, 17 org.apache.commons.net.io, 23 org.apache.commons.net.nntp, 19 org.apache.commons.net.pop3, 20 org.apache.commons.net.smtp, 21 org.apache.commons.net.telnet, 22 org.apache.commons.net.tftp, 22 creating a multiprotocol handler, 24 creating TCP connection, 30 fundamental class usage, 16 handling Chargen connection, 31 handling FTP commands, 32 handling FTP connection, 31 handling message posting via NNTP, 33 SocketClient, 14 using, 24 Utility classes, 23 Network News Transfer Protocol definition, 10 Net classes, 19 Network Virtual Terminal, 8 newsgroup servers, 11 NNTP. See Network News Transfer Protocol NVT. See Network Virtual Terminal POP3. See Post Office Protocol 3 Post Office Protocol 3 definition, 13 Net classes, 20 protocol, 1 protocol levels, 3 Protocol Runner, 24 classes, 26 creating the GUI, 27 features, 25 handling commands, 28 omissions, 25 requirements, 24 rfc114. See File Transfer Protocol rfc1288. See Finger protocol rfc1725. See Post Office Protocol 3 rfc1822. See rlogin protocol rfc2821. See Simple Mail Transfer Protocol rfc768. See UDP rfc783. See Trivial File Transfer Protocol rfc793. See TCP rfc821. See Simple Mail Transfer Protocol rfc854. See Telnet protocol rfc855, 22 rfc857, 8 rfc862. See Echo protocol rfc863. See Discard protocol rfc864. See Character Generator protocol rfc867. See Daytime protocol rfc868. See Time protocol rfc954. See Whois protocol rfc959. See File Transfer Protocol rfc997. See Network News Transfer Protocol rlogin Net classes, 17 rlogin protocol definition, 8 Simple Mail Transfer Protocol definition, 11 Net classes, 21 process, 11 software protocols, 3 TCP complex protocol, 2 connection-oriented, 2 definition, 2 phone line analogy, 2 reliable protocol, 2 Licensed to Tricia Fu 34 JAKARTA COMMONS ONLINE BOOKSHELF TCP/IP four-layer model, 2 Telnet protocol definition, 7 Echo option, 8 Net classes, 22 option, 7 option negotiation, 8 session, 8 terminal emulation, 7 TFTP. See Trivial File Transfer Protocol Time protocol definition, 5 time servers, 5 Transmission Control Protocol. See TCP Trivial File Transfer Protocol definition, 10 Net classes, 22 UDP connection-less protocol, 2 definition, 2 non-reliable protocol, 2 post office analogy, 2 simple protocol, 2 US Geological Survey servers, 7 Usenet, 10 User Datagram Protocol. See UDP UUCP, 11 Whois protocol definition, 6 example, 7 Whois servers, 6 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF XML parsing with Digester Vikram Goyal MODULE MANNING 4 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 4 XML parsing with Digester 4.1 The Digester component........................................................................................................................................... 1 4.2 The Digester stack .................................................................................................................................................... 5 4.3 A match made in heaven?......................................................................................................................................... 6 4.4 Conforming to the rules!........................................................................................................................................... 7 4.5 Resolving conflicts with namespace-aware parsing............................................................................................... 22 4.6 Rule sets: sharing your rules................................................................................................................................... 23 4.7 Externalizing patterns and rules: XMLRules ......................................................................................................... 24 4.8 Summary................................................................................................................................................................. 26 Index ............................................................................................................................................................................. 27 Configuration files are used in all sorts of applications, allowing the application user to modify the details of an application or the way an application starts or behaves. Since storing these configuration details in the runtime code is neither possible nor desirable, these details are stored in external files. These files take a variety of shapes and forms. Some, like the simple name-value pair, let you specify the name of a property followed by the current assigned value. Others, like those based on XML, let you use XML to create hierarchies of configuration data using elements, attributes, and body data. For application developers, parsing configuration files and modifying the behavior of the application isn’t an easy task. Although the details of the configuration file are known in advance (after all, the developers created the basic structure of these files), objects specified within these configuration files may need to be created, set, modified, or deleted. Doing this at runtime is arduous. The Digester component from Jakarta Commons is used to read XML files and act on the information contained within it using a set of predefined rules. Note that we say XML files, not XML configuration files. Digester can be used on all sorts of XML files, configuration or not. The Digester component came about because of the need to parse the Struts configuration file. Like so many of the Jakarta Commons projects, its usability in Struts development made it a clear winner for the parsing of other configuration files as well. In this module, we’ll look at the Digester component. We’ll start with the basics of Digester by looking at a simple example. We’ll then look at the Digester stack and help you understand how it performs pattern matching. This will allow us to tackle all the rules that are prebuilt into Digester and demonstrate how they’re useful. We’ll use this knowledge to create a rule of our own. We’ll round out the module by looking at how the Digester component can be externalized and made namespace-aware. 4.1 The Digester component As we said, the Digester component came about because of the need to parse the Struts Configuration file. The code base for the parsing the Struts configuration file was dependent on several other parts of Struts. Realizing the importance of making this code base independent of Struts led to the creation of the Digester component. Struts was modified to reuse the Digester component so as not to include any dependencies. In essence, Digester is an XML Æ Java object-mapping package. However, it’s much more than that. Unlike other similar packages, it’s highly configurable and extensible. It’s event-driven in the sense that it lets Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF you process objects based on events within the XML configuration file. Events are equivalent to patterns in the Digester world. Unlike the Simple API for XML (SAX), which is also event-driven, Digester provides a high- level view of the events. This frees you from having to understand SAX events and, instead, lets you concentrate on the processing to be done. Let’s start our exploration of the Digester component with a simple example. Listing 4.1 shows an XML file that we’ll use to parse and create objects. This XML file contains bean information for the JavaBeans listed in listings 4.2 and 4.3. Listing 4.4 shows the Digester code required to parse this file and create the JavaBean objects. Note: To be able to run these and all the other examples in this module, you need to have two supporting libraries in addition to commons-digester in your CLASSPATH. These libraries are common-logging.jar and commons-collections.jar. Listing 4.1 A simple XML file (book_4.1.xml) Vikram Goyal Listing 4.2 Book JavaBean package com.manning.commons.chapter04; import java.util.Vector; public class Book { private String title; private Vector authors; public String getTitle() { return this.title; } public void setTitle(String title) { this.title = title; } public Vector getAuthors() { return this.authors; } public void setAuthors(Vector authors) { this.authors = authors; } public void addAuthor(Author author) { if(authors == null) authors = new Vector(); authors.add(author); } public String toString() { StringBuffer buffer = new StringBuffer("Title: " + getTitle() + "\r\n"); if(authors != null) { for(int i = 0; i < authors.size(); i++) { buffer.append( "Author " + (i + 1) + ": " + authors.get(i)); } } List of book’s authors held in this Vector Book bean returns toString representation that includes title and all authors Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 3 return buffer.toString(); } } Listing 4.3 Author JavaBean package com.manning.commons.chapter04; public class Author { private int id; private String name; public int getId() { return this.id; } public void setId(int id) { this.id = id; } public String getName() { return this.name; } public void setName(String name) { this.name = name; } public String toString() { return "Author: " + this.id + ", " + this.name; } } Listing 4.4 Digester code to parse the XML file in listing 4.1 package com.manning.commons.chapter04; import java.io.IOException; import org.xml.sax.SAXException; import org.apache.commons.digester.Rule; import org.apache.commons.digester.Digester; import org.apache.commons.digester.ObjectCreateRule; public class SimpleDigester { public static void main(String args[]) { Digester digester = new Digester(); digester.setValidating(false); Rule objectCreate = new ObjectCreateRule("com.manning.commons.chapter04.Book"); digester.addRule("book", objectCreate); try { Book book = (Book)digester.parse("book_4.1.xml"); System.err.println(book); } catch(IOException ioex) { System.err.println(ioex.getMessage()); } catch(SAXException saex) { System.err.println(saex.getMessage()); } } } c Create Digester instance d Make Digester nonvalidating e Create rule f Add rule to Digester instance g Parse in try-catch block Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF The code in listing 4.4 creates a simple Digester instance and uses it to parse the code listed in listing 4.1 for the XML file called book_4.1.xml. Let’s examine the steps involved in doing so: c A new Digester instance is created by a call to the constructor. In this instance, the constructor takes no arguments and defaults to using the basic properties. The alternate constructors, Digester(javax.xml.parsers.SAXParser parser) and Digester(org.xml.sax. XMLReader reader), let you specify the parser or the reader used by Digester internally. The empty constructor uses the Java API for XML Processing (JAXP) parsers and readers by default. This is suitable in most cases, except where there is a conflict with the underlying environment as reported in BEA WebLogic application server. Note: The Digester class extends the org.xml.sax.helpers.DefaultHandler class; this implies that Digester uses SAX2 to parse XML documents and handle callbacks. d By setting this Digester instance to be nonvalidating, we make the Digester instance bypass validation of the input XML file against a schema. The schema location must be specified in the XML data file that is being parsed. In our case, the validation flag is set to false, so the validation isn’t done. What happens if the validation is set to true, and a schema isn’t properly specified in the XML data file? Well, you’ll get parse exceptions during runtime for each and every element in your XML data file, because Digester defaults to using the W3C_XML_SCHEMA. Validation is turned off by default, so you can bypass this step if you don’t want validation of your XML data. e Here we’re creating a rule that specifies the creation of a Book object. When will this Book object be created? It doesn’t matter—at least, it doesn’t matter for the rule creation itself. Logically, we want this rule to be followed or executed when a corresponding Book element is encountered within the XML data file, but we’ll deal with that later. Notice the syntax of the ObjectCreateRule constructor; when creating this rule, you need to specify the fully qualified name of the class to be instantiated. f Now we add the rule created in step 3 to the Digester instance created in step 1. The syntax of this method should be clear: digester.addRule(String pattern, Rule rule) specifies that when the pattern specified by the first parameter is found in the XML data file, the rule specified by the second parameter should be executed. In this case, the pattern is "book", and the rule is the ObjectCreateRule listed in step 3. In a nutshell, the Digester is now set to create an object of type com.manning.commons.chapter04.Book whenever it encounters the pattern "book" in an XML data file. g The actual parsing is encapsulated in a try-catch block. Parsing of XML documents throws two types of errors, and as shown in this example, they’re caught separately. The IOException is for errors that relate to problems while reading the XML data document. The SAXException is thrown for all errors relating to parsing of the XML document itself. Having specified the rules and patterns in step 4, it’s time to parse the XML document. This is done by calling Digester’s parse method and passing in the location of the file to be parsed. There are several variants of this method, allowing for different input sources (input stream, reader, URI, and file). All these methods return the top-level object created, which in this case is a Book object. We print to the screen the final created Book object’s toString method. If you run this example, you’ll see the following on the screen: Title: null Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 5 This is the expected result. It proves that Digester created the Book object successfully after reading the example XML data file, although the correct title wasn’t set. In our SimpleDigester code, all we wanted to do was to provide an example of the way Digester is run. We’ll extend this simple example later to make sure not only that the Book object is created properly, but that its title and authors are printed as well. 4.2 The Digester stack Before we move on, let’s examine the following statement more carefully from listing 4.4: Book book = (Book)digester.parse("book_4.1.xml"); We said earlier that this statement returns the top-level object created. Since it’s possible that several objects may be created in the course of parsing of an XML document, Digester maintains a stack of these objects. The stack operates on the principle of last-in-first-out (LIFO). Internally, the stack is maintained using org.apache.commons.collections.ArrayStack, which is a utility class in the Jakarta Collections library. (We’ll talk more about this class in module 7, “Enhancing Java core libraries with Collections.”) If you’re familiar with how a stack operates, you know that objects are pushed to the bottom of the stack by newer objects that come in. This way, the topmost object in the stack is more recent than the others that were put in the stack, which is the basic idea of a LIFO list. This has a strong impact on your understanding of Digester. Why? Because as the beginning of a newer element in the XML data file that matches a rule for object creation is encountered, the Digester stack fills with this newer object on top of another object whose end tag hasn’t yet been encountered. These objects are popped off the stack only when their corresponding end tag is encountered. Thus when the parsing of an XML document is finished, technically, the only reference available at that point is the last object created. This may or may not be the desired effect. Consider listing 4.1, the simple XML data file. Two distinct objects can be created by Digester’s parsing of this file. One is the Book object, which we created in listing 4.4. The second is the Author object, which we didn’t create. Let’s assume that we add another rule to listing 4.4 to create this Author object as well, as shown here: Rule objectCreateAuthor = new ObjectCreateRule("com.manning.commons.chapter04.Author"); digester.addRule("book/author", objectCreateAuthor); Run the code in listing 4.4 again after you add these lines. It doesn’t matter whether you add them before or after the corresponding Book creation lines. The output on the screen remains the same: Title: null In a way, this contradicts the operation of a stack. If the code to add the Author rule was after the corresponding Book rule, and therefore the Author object should have been the last object on the stack, the Book object on the stack should have been pushed down, and we should have gotten a ClassCastException. We didn’t. There are two reasons. First, the objects are pushed on the stack in the order they’re created and not when the rule is added. The book element precedes the author element in the XML data file. The rules associated with the book element are fired before the corresponding rules for the author. Thus a Book object is created before an Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF Author object, regardless of when the code to create an Author is specified. Second, and more important, Digester maintains a handle to the first object pushed into the stack and returns it at the end of a parse operation. Since, in this case, Book is the first object pushed in the stack, Digester returns this referenced object at the end of the parse operation. There are very few reasons why you would want to manipulate the Digester stack yourself. You might want to preset the stack with objects that were created separately from the XML data file you want to parse. For example, suppose you had your own instance of the Book object. Instead of adding a rule to create the Book object based on the XML data file, you could push this Book object onto the stack before any parsing was done, as shown here: Book book = new Book(); book.setTitle("My book"); digester.push(book); This lets you preempt the book element in the XML file. When the final parse is complete, the referenced object that is returned is this Book object and not the one specified in the XML data file. Of course, the rest of the elements are created as they’re specified in the XML. In addition to push(Object object), you can perform three other operations with the stack: ƒ clear() clears the contents of the stack. ƒ peek() returns the next object on the stack without removing it. ƒ pop() removes the next object on the stack. These operations are useful when you create your own rules. 4.3 A match made in heaven? Like a real-world dating agency, Digester matches elements in an XML file based on patterns specified by the user. However, unlike real-world dating, the Digester supports wildcard matching. When the parse method of the Digester is called on an XML file, the Digester starts to traverse the element hierarchy contained within the XML file. This means it examines every element and looks up a List to see if any rules are associated with this element. This examination is done from the top-level element to the innermost nested element in the order they’re listed in the XML file. Once a match is found, all rules associated with that match are fired in the order that they were registered in the first place. The element matching roughly follows the XPath naming convention. However, unlike in XPath, there are no relative paths—only absolutes. Each path/pattern starts with the root element, and the root element must be specified. Using listing 4.1 as an example, the following element matching patterns are valid: book, book/author, and book/author/name. There is an exception to this rule, which relates to wildcard patterns: You can use the wildcard character (*) to match more than one pattern. For example, the pattern book/*/name matches the names of all authors, if any, for listing 4.1. With this pattern, you can also match the name element for all elemental depths. So, the wildcard isn’t only a wildcard for elements; it also substitutes for the depths of nesting within an XML document. Continuing with the book/*/name example, this pattern matches all patterns where a name element follows immediately after the book element, as well as book/author/name, book/author/dog/name, book/publisher/name, and so on. The wildcard can be substituted for the Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 7 root character, as well. Thus you can use */name, which matches all name elements that appear at any depth within the document. XPath XPath is a W3C standard for describing the contents of an XML file in a universal way. It specifies a set of rules that let you define any element in an XML file. It’s like a tree of files and folders and is similar to the way you describe files and folders on your computer. Thus you can use an XPath structure like /usr/tomcat/bin to describe elements in an XML file. For more details about XPath, visit http://www.w3cschools.com; or see module 5, “JXPath and Betwixt: working with XML,” where it’s discussed in section 5.2.1. 4.4 Conforming to the rules Having defined how the Digester component makes a match, let’s look at how to specify what happens when a match is made—that is, how a rule is executed. In this section, we’ll also discuss the prebuilt rules specified by Digester, and we’ll make a rule of our own. You add rules to a Digester instance by using the following method: addRule(String pattern, org.apache.commons.digester.Rule) The syntax is straightforward. As you saw in listing 4.4, you add the rule by specifying an instance of the Rule interface to be executed when the specified pattern is matched. In a way, when you add the rule, you specify not only the rule but also the pattern to be matched in order for that trigger to take place. You’ve already come across one of the prebuilt rule specified by Digester. The ObjectCreateRule appears in listing 4.4; we used it to create a Book object. All rules in Digester extend the org.apache.commons.digester.Rule abstract class. This class defines four methods that are called at specific instances by the underlying SAX parser: ƒ begin(String namespace, String name, org.xml.sax.Attributes attributes)— This method is called when the beginning of an XML element is encountered. There is a deprecated version that ignores the namespace and name parameters. We’ll look at namespace-aware parsing later in this module. For the moment, suffice to say that this method is called for all rules that match the registered pattern for that rule, when the pattern is first encountered. In this method, most rules perform housekeeping chores that are useful for rule processing. ƒ body(String namespace, String name, String bodyText)—This method is called when the body of the matching pattern/element is encountered. If the element doesn’t contain a body (for example, in the case of an empty element), this method is not called. The main parameter is, of course, bodyText, which is the text contained within this pattern/element. ƒ end(String namespace, String name)—This method is where most of a rule’s action happens. Once the body of an element/pattern has been read, Digester is confident about creating and executing any objects or methods. The parser calls this method when the end of the element/pattern is matched. As is the case with begin and body, there is a deprecated version of this method, which isn’t namespace-aware and takes no parameters. Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF ƒ finish()—This is a cleanup method that’s called when rule processing is finished. It lets you release any resources that may have been allocated when the beginning of the method was encountered, perhaps in the begin method. A rule defines what needs to be done when a matching pattern is found. But how is the matching performed? It’s easy to assume that a match is made when the exact pattern specified by the user while adding the rule is found. However, that is just the default (and perhaps the most expected) way of matching patterns. You can change that behavior by extending the interface Rules, which defines the way matching is performed. Digester uses a default implementation of this called the RulesBase class. Digester calls the match(String namespaceURI, String match) method of this class to get a list of all registered rules for that particular pattern. The RulesBase class maintains a HashMap of rules (with the pattern as the key) that have been registered with the Digester before the parsing begins. When the match method is called, the rules associated with this pattern are returned in the order that they were registered with Digester. By extending the RulesBase class, you can override this match method to provide a different implementation, which matches patterns based on other matching policies. It’s time to look at the standard rules provided with Digester. For clarification purposes, we’ve divided these rules into three categories: element creation, property setters, and stack manipulation rules. The rules are listed in table 4.1, and the following sections examine them in more detail. Table 4.1 Conceptual categories for Digester rules Category Category summary Rules in the category Element creation Rules that let you create objects or elements ObjectCreateRule FactoryCreateRule NodeCreateRule Property setters Rules that let you modify properties of objects SetPropertiesRule SetPropertyRule CallMethodRule CallParamRule PathCallParamRule ObjectParamRule Stack manipulation Rules that let you manipulate the objects on the Digester stack SetNextRule SetTopRule SetRootRule 4.4.1 Element creation rules Rules in this category let you create an object or an element. Basically, an entity is created with the help of these rules. You’ve already encountered one of these rules, ObjectCreateRule. Let’s begin by examining this rule in greater detail. ObjectCreateRule The ObjectCreateRule allows you to create an object and push it onto the Digester stack. This happens when the begin() method is fired when an element matching the patterns is encountered. When the end() method is fired, which happens when the end of the element is encountered, the same object is popped off the stack. Typically, you pass in the name of the class that needs to be created when you’re creating an ObjectCreateRule, as we did in listing 4.4. However, you may also create this rule so that an override is used to create the instance of the class at runtime. This override is taken from the attributes passed to the Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 9 begin method. These are the attributes of the element that is being processed, so among them should be an attribute that corresponds to the override you specify. This is better explained with an example. Suppose you have a specialized Book class called TechnicalBook. This specialized class extends the original Book class and provides an extra property called technicalCategory. This class is shown in listing 4.5. Listing 4.5 TechnicalBook, which extends the Book class package com.manning.commons.chapter04; public class TechnicalBook extends Book { private int technicalCategory; public void setTechnicalCategory(int technicalCategory) { this.technicalCategory = technicalCategory; } public int getTechnicalCategory() { return this.technicalCategory; } } To create this class instead of the Book class, we need to make a change in listing 4.4 and in listing 4.1, which contains the sample XML data file. Listing 4.1 is altered to include an attribute that specifies the class to be created: Listing 4.4 is changed to specify the attribute that will override the original Book class: Rule objectCreate = new ObjectCreateRule("com.manning.commons.chapter04.Book", "realclass"); What is the advantage of being able to specify an override? Well, consider the cases where you don’t know in advance which class needs to be created when a book element is encountered. The XML data file may be an input from disparate sources, and some sources may want the creation of the Book object while others may want the creation of the specialized TechnicalBook. Because the attribute is an override only if it’s present, you can cater to all sorts of Book creation cases in the modified listing 4.4. All the XML data file must contain is the attribute realclass, which points to a resolvable class name. If it’s missing, the parent class Book object is created. The ObjectCreateRule contains constructors that let you pass the base class to be created as a Class or as a String, along with variations for the overriding attribute name, which is always specified as a String. FactoryCreateRule FactoryCreateRule is similar. ObjectCreateRule is useful when the object that needs to be created supplies a no-argument constructor, which can be used to create the object without any parameters. But how do you create the object when the underlying class doesn’t supply such a constructor? You use the FactoryCreateRule. Using the FactoryCreateRule is slightly more involved than using the ObjectCreateRule. As the name suggests, an object is created not directly but by using a factory. This implies that when this rule is constructed, you need to specify the factory that will create the object rather than the object itself. Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF The factory class in question must implement the ObjectCreationFactory interface. This interface specifies a method called createObject(org.xml.sax.Attributes attributes), which your factory must implement in order to create the required object. Listing 4.6 shows an example using the Book class. Listing 4.6 BookCreationFactory: a factory for creating Book objects package com.manning.commons.chapter04; import org.xml.sax.Attributes; import org.apache.commons.digester.AbstractObjectCreationFactory; public class BookCreationFactory extends AbstractObjectCreationFactory { public Object createObject(Attributes attributes) { Book book; if(attributes != null) { book = new Book(); book.setTitle(attributes.getValue("title")); } else { throw new IllegalArgumentException("No Attributes!!"); } return book; } } As you can see, using the FactoryCreateRule lets you configure and manipulate your object before it’s created. This allows for greater flexibility in the way objects are created, as compared to the ObjectCreateRule. Let’s change listing 4.4 to use this rule: import org.apache.commons.digester.FactoryCreateRule; ... Rule objectCreate = new FactoryCreateRule("com.manning.commons.chapter04.BookCreationFactory"); After you make the changes and add the code showed in listing 4.5, running listing 4.4 will create a Book object with its title set as specified by the XML data file. Digester uses this rule to create rules when these rules are read from an XML file rather than being listed in code. We’ll cover this topic later in this module. NodeCreateRule The final rule in this category is NodeCreateRule. It’s useful for creating XML representations of either part of an XML data document or the whole document. This rule creates a node based on the match that triggers it. The default node is the element itself; therefore the return type of the entity that is created is an ElementNode. The element is pushed onto the stack when the start of the element is encountered and popped off the stack at the end of the element. As an example, change listing 4.4 and modify the creation of the objectCreateRule as shown in the code snippet here: import javax.xml.parsers.ParserConfigurationException; import org.apache.commons.digester.NodeCreateRule; Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 11 ... Rule objectCreate = null; try { objectCreate = new NodeCreateRule(); digester.addRule("book", objectCreate); Object book = digester.parse("book_4.1.xml"); System.err.println(book); } catch(IOException ioex) { System.err.println(ioex.getMessage()); } catch(ParserConfigurationException px) { System.err.println(px.getMessage()); } catch(SAXException saex) { System.err.println(saex.getMessage()); } } When you run this code, you’ll see the following output: Vikram Goyal The output is a string representation of the Element node for the book object, based on our XML data file. The same Element node can be input to another DOM document for use in creating a separate representation of the XML document that we already have. 4.4.2 Property setter rules The rules in this category allow for either the manipulation of object properties or modifying existing objects by calling their methods. SetPropertiesRule We’ll start with the simplest of these rules, SetPropertiesRule. As the name suggests, this rule sets all the settable properties of the current object on the Digester stack. Note the emphasis on settable properties. Digester uses reflection to find out which attributes of the current matched element have a corresponding setter in the object on the stack. It uses the JavaBean convention (setXXX(datatype xxx)) to match attributes to settable methods within the object on the stack. If an attribute can’t be matched to a settable property, it’s ignored. You can override this natural mapping by specifying alternate mappings between attributes and properties. The SetPropertiesRule class provides two constructors to do this: One allows for a single attribute to be matched, and the second allows for a whole array of attributes to be matched to a whole array of properties. These constructors are SetPropertiesRule(String attributeName, String propertyName) and SetPropertiesRule(String[] attributeNames, String[] propertyNames), respectively. Generally, you’ll use default constructor, SetPropertiesRule(). Using this rule is straightforward; in fact, nothing is stopping you from reusing it to set the properties of several objects. To create this rule for the simplest case, create it and then add it to the Digester at the point where you want the object properties to be set. Remember that rules are executed in the order in which they’re added for a particular match, so don’t add this rule to a match until the rule for creation of that object has been completed. For example, you can create this rule at any point in listing 4.4, as long as it’s created before it’s to be added for a match and it’s added after the object creation rule for that particular Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF object/match. In the following code, the SetPropertiesRule is added to both the Book object and the Author object: digester.addRule("book", objectCreate); Rule propertiesSet = new SetPropertiesRule(); digester.addRule("book", propertiesSet); Rule objectCreateAuthor = new ObjectCreateRule("com.manning.commons.chapter04.Author"); digester.addRule("book/author", objectCreateAuthor); digester.addRule("book/author", propertiesSet) As you can see, the same SetPropertiesRule is used for both the Book and the Author objects. Also, these rules are added after the rule for creation of these objects. Note: The use of this rule and the next one requires the presence of commons-beanutils.jar in your CLASSPATH. SetPropertyRule The SetPropertyRule is similar to the SetPropertiesRule—except, as you may have guessed, it’s only called for a single property. The name of the property as well as the name of the attribute that contains the value of the property must be specified when this rule is created. Suppose that we need to add a new property to our Book class that requires us to keep information about the publisher of the book. Further suppose that this value is held in our Book object as boolean flags, one for each possible publisher. For example, if the book is published by ABC publishing house, this flag is set to true. If the book is published by XYZ publishing house, this flag is set to true, and so on. Thus, our Book object contains the following boolean flags, along with their getter and setters: private boolean ABCPublishingHouse; private boolean XYZPublishingHouse; To make the task of specifying these properties in the XML data file generic, we need to allow XML data file publishers to specify the property they’re setting. This should be the same for all publishers, because we don’t want to create separate Document Type Definitions (DTDs) for different publishers. The publishers need to set the value of the same attribute—let’s call it publisherName—equal to their name within the system, which may be supplied to them by you. Further, they must specify the value of another attribute, which in this case (since it’s a boolean value) will be equal to true. This is shown here: To use this rule, all you need to do is create it with the publisherName and publisherValue values, as shown here: digester.addRule("book", new SetPropertyRule("publisherName", "publisherValue")); Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 13 Digester looks for the attribute publisherName when it matches the book element. It uses the value of this attribute to look for a property by this name in the Book object. When it finds that property, it looks at the value of the second attribute, publisherValue, and uses this value to set up this property. CallMethodRule The CallMethodRule is useful when you want to call a method on the object on top of the stack. You can call this method only for the object that is currently being processed, which means this method will be called only on the object whose end element has not yet been reached; therefore, the triggering element must be a nested element of this object. As an example, let’s add a new element to the XML data file from listing 4.1, as shown here: Vikram Goyal 10 The chapters element is nested within the book element; therefore, if we set a CallMethodRule on this element, the method being called must be present in the Book class. Let’s add a method to calculate the number of pages in the book based on this value and a slightly superficial calculation (total pages = number of chapters * 30): public void calculateTotalPages(int chapterCount) { System.err.println("Total Pages: " + (chapterCount * 30)); } How do we call this method when we’re configuring Digester? Add the following lines to listing 4.4 after the ObjectCreateRule for the Book object has been added: digester.addRule("book/chapters", new CallMethodRule("calculateTotalPages", 0, new Class[] {int.class})); The CallMethodRule must be told what method is to be called, how many parameters this method takes, the data type of each method, and where to collect the values of these parameters. In the previous code, CallMethodRule is told that the calculateTotalPages method needs to be called, that it takes one parameter, and that this parameter is of type int. Note: How does a value of 0 in the code translate into a single parameter? Well, a 0 is of special significance in a CallMethodRule constructor. It means the method to be called takes one parameter and that the value of this parameter must be collected from the body text of the matching element that triggered this rule. Let’s examine the various constructors of CallMethodRule and their significance: Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF ƒ CallMethodRule(String methodName)—This constructor accepts the name of the method that will be called. The fact that there are no other arguments to this constructor implies that the method that will be called takes no parameters. The body text of the triggering element is ignored. ƒ CallMethodRule(String methodName, int paramCount)—This constructor is the same as the previous one, but in this case, the number of parameters to be passed is specified. Each parameter must be of type String. If paramCount is set to 0, as in the previous example, then only one parameter is to be passed, and the value of this parameter must be taken from the body text of the triggering element. However, if paramCount is greater than zero, a second rule—CallParamRule— comes into the picture. For each parameter, the value is set with a corresponding CallParamRule. (We’ll go into this later with an example.) ƒ CallMethodRule(String methodName, int paramCount, Class[] paramTypes)— Again, this constructor is similar to the others; but in this case, the types of each parameter are specified by the paramTypes array. For each type, this array contains corresponding type information. There is a variation of this constructor, where the parameter types are specified using fully specified class names in a String array. For either of these variations, if you specify these arrays as null or empty, the CallMethodRule defaults to using the equivalent of the first constructor (the method is called without any parameters). These constructors may seem a little confusing, but they cover all possible scenarios and are a complete solution to invoking an arbitrary method. CallParamRule The CallParamRule supplements the CallMethodRule by providing the value of parameters when the value of these parameters isn’t taken from the body text of a triggering element for a CallMethodRule. However, the CallParamRule can itself save a parameter for use by the surrounding CallMethodRule by taking this value from the body text of the element that triggers it. Again, an example will make things clearer. Consider our Author object. The XML data file in listing 4.1 shows that the author element contains a nested element for the name of the author. The name should really be an attribute, because conceptually it is an attribute of the Author. However, in this case we have made it a nested element and we need to set the name of the Author by using a combination of CallMethodRule and CallParamRule. (To test that the setName method of the Author class will be called, insert a debugging statement similar to System.err.println("Called Author:setName() with: " + name);.) Insert the following lines of code in listing 4.4 after the rule for the creation of the Author object (recall that CallMethodRule is called on the current object on the stack): digester.addRule("book/author", new CallMethodRule("setName", 1, new Class[] {String.class})); digester.addRule("book/author/name", new CallParamRule(0)); The first line in this code lets you set a CallMethodRule specifying the name of the method to be called on the current stack object, that it takes one parameter, that the value of this parameter will come from a CallParamRule to follow, and that the data type of this parameter is of type String. The second line sets up the expected CallParamRule on the element book/author/name. The constructor for CallParamRule, in this case, takes a single argument: the index of the parameter that is being set up (note that the index starts at 0). If a CallMethodRule isn’t followed by the right number of CallParamRules, no errors or exceptions are thrown; the rule exits silently without calling the method. Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 15 However, if the index of the corresponding CallParamRules isn’t set correctly, you’re likely to get an ArrayIndexOutOfBoundException at runtime. There are three other constructors for the CallParamRule, as listed here. Note that in all the constructors, the common thread is the paramIndex, which specifies where the parameter will fit within the surrounding CallMethodRule: ƒ CallParamRule(int paramIndex, String attributeName)—Instead of taking the value of the parameter from the body text of the element that triggers this rule, you can take it from the attributes of the element. This constructor lets you do that by specifying the name of the attribute from which the value is to be taken. For example, you could have the XML and use the CallParamRule to trigger on the name element. This way, if the Author object had separate parameters for the name (setName(firstname, lastname)), the same matching trigger on name could be used to set the different parts of the name. The CallMethodRule in this case would change to new CallMethodRule("setName", 2, new Class[String.class, String.class]) and there would be two CallParamRules after it: new CallParamRule("book/author/name", new CallParamRule(0, "firstname")); new CallParamRule("book/author/name", new CallParamRule(1, "lastname")); ƒ CallParamRule(int paramIndex, boolean fromStack); CallParamRule(int paramIndex, int stackIndex)—We’ve lumped these two constructors together because they essentially do the same thing. The first constructor takes the value of the parameter from the top object on the current stack, which is the current object on which this method will be called anyway. The second constructor lets you pass an index for the stack. Thus, instead of the top object on the stack, as with the previous constructor, you can specify which object on the stack you want the parameter to be set to. Remember that the stack index starts from zero, so calling CallParamRule(int paramIndex, 0) is the same as calling CallParamRule(int paramIndex, true). Note that when you use this constructor, you must specify the class of your parameter correctly to be the object on the stack when constructing the CallMethodRule. PathCallParamRule PathCallParamRule is an extension of the CallParamRule . It’s similar to the CallParamRule, in that it provides a surrounding CallMethodRule with the value of a parameter. Unlike the CallParamRule, this value is provided as the full path of the element that triggered it. Thus, if you constructed a PathCallParamRule as new PathCallParamRule(0) and added it to the digester as addRule("book/author/name", pathCallParamRule), it would pass the value "book/author/name" to the surrounding CallMethodRule’s first parameter when it was triggered. It may not seem useful in situations where the full path is known in advance, as in this case; but the value of this rule is manifest in situations where the matches are triggered using wildcards. Because the user doesn’t know which path may trigger this rule, the actual value of the path can be picked up using the rule. Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF ObjectParamRule Finally, ObjectParamRule is another extension to the CallParamRule. This rule lets you save an arbitrary object for a surrounding CallMethodRule by using the constructor ObjectParamRule(int paramIndex, Object param). You can make this setting conditional on the presence (or absence) of an attribute by using the alternate constructor ObjectParamRule(int paramIndex, String attributeName, Object param). 4.4.3 Stack manipulation rules Although the rules in this category don’t really manipulate the stack, they manipulate the objects on the stack. These rules let you establish parent-child relationships between the different objects that are adjacent at the top of the stack. Note: These rules apply only to the top object on the stack and the one just below it. SetNextRule The SetNextRule is used to establish a one-to-many relationship between the top object on a stack and the one just below it (which is why it’s called the set-next rule). Consider the Book object. It’s possible that a Book object has several authors. Thus, there is a one-to-many relationship between the Book and the Author objects. This rule is therefore set to trigger on the author element. Whenever an author element is found, Digester looks for the next object on the stack, which in our case is Book, and invokes a method that you specify on this Book object to add an Author to it. To add this rule, you use the constructor SetNextRule("addAuthor"). It assumes that the class of the object being added to the parent is the same as the class of the object on the top of the stack. This assumption is true in most cases. However, if it isn’t true, and if you wish to make the class different, you can use the alternate constructor: SetNextRule("addAuthor", String paramType). This constructor lets you specify the parameter type as a fully qualified argument to the exact class. So, to add authors to the Book object, you need to add the following line of code to listing 4.4: digester.addRule("book/author", new SetNextRule("addAuthor")); SetTopRule Similar to SetNextRule, but working in the opposite direction, is SetTopRule. Instead of setting the top object on the stack to the next-to-top element on the stack, this rule lets you set the next-to-top object on the stack to the top object on the stack. Consider that instead of specifying the parent-child relationship as Book Æ Author, you can specify it as Author Æ Book: An Author writes a Book, instead of a Book is written by several Authors. In this case, you’re setting the parent of the Author object. Using this rule is similar to the SetNextRule. However, since now you’re acting on the Author object, we expect addBook(Book book) or setParent(Book book) or some similar method in the Author class to allow us to set an Author’s parent. Once we have this method, adding this rule to the Digester is simple: digester.addRule("book/author", new SetTopRule("addBook")); Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 17 SetRootRule The SetRootRule is similar to the SetNextRule in the sense that it acts on the top object on the stack. However, it doesn’t necessarily add the top object to the next-to-top object on the stack. Instead, it adds it to the root object for the XML file, which may or may not be the next object on the stack, depending on whether a nested hierarchy pushes that parent object down the stack. Note: The Stack manipulation rules are called in their respective end methods. This allows the objects on which these rules are called to be fully configured before being passed to their parent/child. An example This brings us to the end of our discussion of Digester’s predefined rules. Before we show you how to make your own rules, let’s apply some of these predefined rules to listing 4.4 to create our Book object properly. Listing 4.7 shows how to do this. Although not all the rules we’ve discussed are used here, this example gives a fair example of most of the rules you’re likely to encounter. Listing 4.7 Extending listing 4.4 to completion package com.manning.commons.chapter04; import java.io.IOException; import org.xml.sax.SAXException; import org.apache.commons.digester.Rule; import org.apache.commons.digester.Digester; import org.apache.commons.digester.SetNextRule; import org.apache.commons.digester.CallParamRule; import org.apache.commons.digester.CallMethodRule; import org.apache.commons.digester.SetPropertyRule; import org.apache.commons.digester.ObjectCreateRule; import org.apache.commons.digester.SetPropertiesRule; public class SimpleDigester { public static void main(String args[]) { Digester digester = new Digester(); digester.setValidating(false); Rule objectCreate = new ObjectCreateRule("com.manning.commons.chapter04.Book"); digester.addRule("book", objectCreate); Rule propertiesSet = new SetPropertiesRule(); digester.addRule("book", propertiesSet); Rule objectCreateAuthor = new ObjectCreateRule("com.manning.commons.chapter04.Author"); digester.addRule("book/author", objectCreateAuthor); digester.addRule("book/author", propertiesSet); digester.addRule("book/author", new CallMethodRule("setName", 1, new Class[] {String.class})); Set properties of Book object Create Author object Set Author properties Set Author name Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF digester.addRule("book/author/name", new CallParamRule(0)); digester.addRule("book/author", new SetNextRule("addAuthor")); try { Object book = digester.parse("book_4.1.xml"); System.err.println(book); } catch(IOException ioex) { System.err.println(ioex.getMessage()); } catch(SAXException saex) { System.err.println(saex.getMessage()); } } } However, there is one sore point in the example: Each rule that is created must be added to the Digester separately, in a two-step process. First, the rule has to be created; and next, the rule has to be added to the Digester. Considering that this is such a repetitive process, it would be nice if it could be bundled into one shorthand method. Fortunately, Digester comes prebuilt with a one-step process for most of the rules. This not only reduces the lines of code in our example program, but also makes the code much cleaner to read and maintain. In listing 4.8, we rework SimpleDigester.java from listing 4.4 into SimpleDigesterV2.java. Notice that we removed the import statements for the rules in this code, because they’re no longer required. Listing 4.8 SimpleDigesterV2.java: uses the shorthand method for adding rules package com.manning.commons.chapter04; import java.io.IOException; import org.xml.sax.SAXException; import org.apache.commons.digester.Rule; import org.apache.commons.digester.Digester; public class SimpleDigesterV2 { public static void main(String args[]) { Digester digester = new Digester(); digester.setValidating(false); digester.addObjectCreate( "book", "com.manning.commons.chapter04.Book"); digester.addSetProperties("book"); digester.addObjectCreate( "book/author", "com.manning.commons.chapter04.Author"); digester.addSetProperties("book/author"); digester.addCallMethod( "book/author", "setName", 1, new Class[] {String.class}); digester.addCallParam("book/author/name", 0); digester.addSetNext("book/author", "addAuthor"); Add Author object to Book object Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 19 try { Object book = digester.parse("book_4.1.xml"); System.err.println(book); } catch(IOException ioex) { System.err.println(ioex.getMessage()); } catch(SAXException saex) { System.err.println(saex.getMessage()); } } } 4.4.4 Live by your rules After having gone through the list of available prebuilt rules, if you feel that none of them satisfy a unique need of yours, you may roll your own rule. In this section, we’ll roll a rule of our own to handle a problem that can’t be solved by the current crop of available rules. Consider the XML data file in listing 4.9 and the Grocery object in listing 4.10. Listing 4.9 XML data file for map-based values Sugar 3 Rice 5 Bakedbeans 2 Listing 4.10 Grocery object that needs to be added as a value to the supermarket HashMap package com.manning.commons.chapter04; public class Grocery { private int id; private String name; private int price; public int getId() { return this.id; } public void setId(int id) { this.id = id; } public String getName() { return this.name; } public void setName(String name) { this.name = name; } public int getPrice() { return this.price; } public void setPrice(int price) { this.price = price; } Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF public String toString() { return this.name + "-" + this.price; } } Looking at these listings, you’d think there is nothing different about them—processing this list using Digester should be simple. However, before we process this file, we must consider a restriction on the structure of the supermarket element. To make it easy to look up grocery items, we’re told to make the supermarket a HashMap. Then, grocery items within this supermarket are added as key/value pairs, with the id attribute of the grocery element as the key and a fully qualified Grocery object as the value. How would we do this with the current rules? The simple answer is, we can’t. No rule lets you add objects, either to the first object on the stack or to the one after it, using a key mapping. We can’t use the SetNextRule or the SetTopRule, because they don’t cater to mapped objects. We can’t use the CallMethodRule, because it would need to be called on the Grocery object rather than the supermarket object. If we used it on the supermarket object, it would be called before the Grocery objects were created. To solve this problem, we need to create our own rule. AddGroceryRule, shown in listing 4.11, is a custom rule that solves this problem. Listing 4.11 AddGroceryRule.java: adds groceries to the supermarket HashMap package com.manning.commons.chapter04; import java.util.Map; import org.apache.commons.digester.Rule; public class AddGroceryRule extends Rule { public void end(String namespace, String name) throws Exception { Grocery grocery = (Grocery)digester.peek(0); Map supermarket = (Map)digester.peek(1); supermarket.put(new Integer(grocery.getId()), grocery); } } It’s easy to add a custom rule in Digester, as you can see from this example. Your rule must extend the org.apache.commons.digester.Rule abstract class. Once you do this, you need to provide your own implementation of any of the begin(), body(), end(), or finish()methods. In this rule implementation, we need to extend the end method, because this rule should operate only on the fully formed Grocery object. Other methods don’t need to be extended because there is nothing else to be done when the beginning, body, or finish is reached for this element. We start this method by getting the top object on the stack, which should be the Grocery object. The next object to which the Grocery object needs to be added is the supermarket HashMap. This is extracted by looking at the next object on the stack. Finally, the supermarket HashMap is updated with the current Grocery object, using the Grocery object’s ID as the key and the wholly formed Grocery object as the value. Using this rule now is no different than using the rest of the rules in Digester. This is illustrated in listing 4.12. Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 21 Listing 4.12 Putting it all together: custom rule usage package com.manning.commons.chapter04; import java.util.HashMap; import java.util.Iterator; import java.io.IOException; import org.xml.sax.SAXException; import org.apache.commons.digester.Rule; import org.apache.commons.digester.Digester; public class NewRuleTest { public static void main(String args[]) { Digester digester = new Digester(); digester.setValidating(false); digester.addObjectCreate("supermarket", "java.util.HashMap"); digester.addObjectCreate( "supermarket/grocery","com.manning.commons.chapter04.Grocery"); digester.addSetProperties("supermarket/grocery"); digester.addCallMethod( "supermarket/grocery/price", "setPrice", 0, new Class[]{int.class}); digester.addCallMethod("supermarket/grocery/name", "setName", 0); digester.addRule("supermarket/grocery", new AddGroceryRule()); try { HashMap supermarket = (HashMap)digester.parse( "supermarket_4.9.xml"); Iterator itr = supermarket.keySet().iterator(); while(itr.hasNext()) { Integer key = (Integer)itr.next(); System.err.println(key + "-" + supermarket.get(key)); } } catch(IOException ioex) { System.err.println(ioex.getMessage()); } catch(SAXException saex) { System.err.println(saex.getMessage()); } } } Our AddGroceryRule is highlighted in bold. As you can see, once this rule is constructed, adding it to Digester is similar to adding the prebuilt rules. The output of this code, which is the key-value pair listing of the supermarket HashMap, is shown here using the sample XML data file from listing 4.9: 1002-Rice-5 1003-Bakedbeans-2 1001-Sugar-3 Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF 4.5 Resolving conflicts with namespace-aware parsing Namespaces are used in XML documents to resolve conflicts between same-named elements. By specifying a namespace for a particular element, you’re binding an element in that namespace. Some companies declare their namespaces within their XML data so that it isn’t confused with a competing company’s data, which may include same-named elements. Doing so also helps alleviate confusion between similar element names that refer to different things. For example, the XML data file for a business dealing in sports might have an element named — which is, of course, completely different from the XML data for a company that organizes dancing events and has a totally different meaning for a . To overcome this problem, the element is made namespace-aware: It’s qualified with a prefix to make sure that its meaning is unambiguous in all circumstances. Thus, the element for the sports company might be made into , and a likely transformation for the organizer of dancing events would be . Digester works two ways with XML documents that are namespace-aware. The first way, which is the default, is to specify namespace-aware elements in the patterns that are matched and acted upon. You don’t have to change any settings in Digester itself. However, you must specify the namespace prefix for all your elements and rules. For example, suppose listing 4.13 represents the combination of XML data files from the sports and dance companies. Listing 4.13 Namespace-aware XML data ... ... To parse this file the first way, each element-matching pattern must have the namespace prefix. Thus, to create a Ball object for the separate companies, you would use the following code: digester.addRule("dancenation:ball", "DancenationBall"); digester.addRule("sportsco:ball", "SportsCoBall"); The first rule will be triggered only for the Dancenation ball, and the second rule will be triggered only for the SportsCo ball. The same rule applies for any other rules and for nested elements as well. However, if the element matching isn’t prefixed correctly, or if it isn’t prefixed at all, no rules will be triggered (unless, in the case of no prefix, there is indeed a ball element without a prefix in the file, representing some other namespace). The second way you can make Digester work with namespace-aware XML documents is to set the namespace-aware property to true. You can do this by calling the setNamespaceAware(true) method on the Digester instance before adding any rules. You then set the URI of the namespace so that it can be matched in the XML data file. Finally, you add the rules corresponding to this namespace to the Digester. Once this is done, you repeat this process for any other namespaces in your XML data file. The following code snippet shows how to do this for the example in listing 4.13: Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 23 digester.setNamespaceAware(true); digester.setNamespaceURI("http://www.dancenation.com/namespace"); .. ... digester.setNamespaceURI("http://www.sportsco.com/namespace"); ... ... First we sets the Digester to recognize the namespace for the dance company. Then we add the rules to this Digester instance normally, but they will be executed only for the current namespace. After we’re finished adding rules for the dance company, we define the namespace for the sports company and add its rules. As with the first approach, if the XML data file contains data from other namespace or data that doesn’t have any namespace, this data won’t be recognized and the rules won’t be fired, even if a pattern-match happens. The second approach is better than the first because it removes the dependency on the prefix used for namespaces. This is important because companies are more likely to provide their namespace URI than the prefix used in their XML files. However, this approach requires you to bind your Digester instance to a particular namespace. 4.6 Rule sets: sharing your rules A single Digester instance can be reused to process a variety of XML files. However, this process is repetitive. You have to create a Digester instance, initialize it with its properties, add all the rules, and finally parse it. If you have multiple XML data files that contain different data but have the same elements and are processed by separate Digester processes using the same rules across enterprise-wide applications, you repeat the same process across all these applications. It would be nice to have a central repository of these rules. Separate applications would then need to initialize their own Digester instance from this repository rather than repeating the same code to add the rules. Digester helps in this process by introducing the concept of a RuleSet. RuleSet is an interface that defines a method called addRuleInstances(Digester digester). If you want to save your rules into a single place from which other applications can access them, you must implement this method. Then, the calling applications will call the method digester.addRuleSet(new YourRuleSet()); to enable the Digester instance with the right set of rules. Digester provides a convenience base class that implements this interface: RuleSetBase. Let’s create an implementation of this class to generalize the conversion of the supermarket XML data from listing 4.9. Listing 4.14 shows SupermarketRuleSet.java, which encapsulates the rules illustrated in listing 4.12. Listing 4.14 SupermarketRuleSet.java: encapsulates rules for parsing the supermarket XML package com.manning.commons.chapter04; import org.apache.commons.digester.Digester; import org.apache.commons.digester.RuleSetBase; public class SupermarketRuleSet extends RuleSetBase { public SupermarketRuleSet() { this(""); c Declare SupermarketRuleSet Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF } public SupermarketRuleSet(String namespaceURI) { super(); this.namespaceURI = namespaceURI; } public void addRuleInstances(Digester digester) { digester.addObjectCreate("supermarket", "java.util.HashMap"); digester.addObjectCreate( "supermarket/grocery", "com.manning.commons.chapter04.Grocery"); digester.addSetProperties("supermarket/grocery"); digester.addCallMethod( "supermarket/grocery/price", "setPrice", 0, new Class[] {int.class}); digester.addCallMethod("supermarket/grocery/name", "setName", 0); digester.addRule("supermarket/grocery", new AddGroceryRule()); } } c The SupermarketRuleSet extends the RuleSetBase class and provides an implementation for the addRuleInstances method. d Even though we’re encapsulating the rules within this class, they’re still namespace-aware. The namespaceURI is defined in the superclass and is declared protected to enable subclasses to set it. e This is the core of the class. Rule addition is encapsulated in this method with a handle to the Digester instance on which the rules will act. With this class in place, we can remove the rules from listing 4.12 and replace them with a single line: digester.addRuleSet(new SupermarketRuleSet()); This line must be added after the Digester has been configured and before it’s called to parse the actual document, just as you would do with the direct addition of rules. Any instance of Digester across applications can use this class as a central repository of the way a supermarket XML data should be processed. This is quite handy! In the next section, we’ll look at the ways Digester makes it even easier to encapsulate patterns and rules by externalizing them to XML files. 4.7 Externalizing patterns and rules: XMLRules So far, we’ve discussed how Digester lets you programmatically set the rules and patterns that drive it. However, this is a bit of a contradiction to the principle of Digester itself. The fact that Digester simplifies configuration of Java objects based on arbitrary XML input should alert you to the fact that the same principle can be applied to Digester! Digester is an object, and it should be easy to define the rules and d Define namespace URI e Encapsulate rule addition Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 25 patterns that drive a Digester application using external XML data files. In this section, we’ll look at how you can do that. There is, of course, an inherent advantage to externalizing the rules and patterns to XML files: Doing so lets you change the rules or the patterns at runtime without having to recompile your code. Another advantage is that XML rules files can be included or referenced within one another; this is advantageous if you want to place the configuration of different objects in separate files. However, the disadvantage of externalizing is that your rules and patterns are available in plain text for anybody to see. This may or may not be a good idea, depending on the confidentiality of the way your objects are created and related to each other. Support for externalizing Digester patterns and rules is provided in the package org.apache.commons.digester.xmlrules. This package contains three classes: ƒ FromXMLRuleSet creates a rule set from an XML file. ƒ DigesterLoader uses this class to load a rule set and adds it to a Digester instance. ƒ DigesterRuleParser is used internally by FromXMLRuleSet to load and create the rule instances. The DigesterRuleParser class contains internal factory classes to manage the creation of the rules found within the XML file. For each rule that is possible within Digester, a corresponding element can be defined in the XML file. Let’s look at an example to see how it’s done: We’ll externalize the creation of our original Book class to an XML file. Listing 4.15 shows how we can translate the rules defined in listing 4.8 into an external XML file. Listing 4.15 Externalizing the patterns and rules for creating the Book object into an XML file The top-level element in the XML file must be . A DTD lists this and the other requirements of an XML file that contains patterns and rules. This DTD is stored in the org/apache/commons/digester/xmlrules package in your Digester library; you’ll need to consult it to get correct syntax for the elements. This DTD is also listed in the appendix. The patterns in the example file can be specified in two ways. This element lists it in such a way that the rules associated with this pattern are nested within the pattern element. The other way lets you list the pattern as an attribute of the rule to which that pattern is applicable. Digester lets you freely mix and match both these techniques, although from a best-practice point of view, sticking to one or the other is the better way to go. Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF You can also include and reference other XML files that contain rules and patterns by using the element . You can do this for programmatically created rules, as well. For example, instead of using the path to other XML files, you can use the include element with the class attribute: . This class must implement the org.apache.commons.digester.xmlrules. DigesterRulesSource interface. This interface contains a single method, getRules(Digester digester), which should contain the rules as they’re added to the Digester instance. Notice that the patterns and rules translate in an elegant manner into the XML file in listing 4.15. It’s clear by looking at this file that there is a one-to-one transition from programmatically adding these rules and patterns to listing them in this file. How do we now use this file to parse our XML data file? Simple! We use the DigesterLoader class to load the rules and patterns defined in listing 4.15 and create an instance of Digester that’s fully configured with these rules, as follows: Digester digester = DigesterLoader.createDigester(new File("book_rules_4.15.xml").toURL()); Note: There is one problem with this scenario. Specifying Digester’s rules and patterns in an external file is good as long as the rules are prebuilt. If you want to specify your own rule, you can either add it after you get your instance of Digester from the DigesterLoader class or make your rule part of Digester’s prebuilt rules. However, this is a tedious process. You must first make an entry in the Digester’s DTD to specify the attributes of your rule. Next, you need to define an ObjectCreationFactory that allows DigesterRuleParser to create an instance of your rule. Finally, you must extend the DigesterRuleParser class and override the addRuleInstances method to add your own rule. Needless to say, this isn’t recommended; you can easily accomplish the same thing by adding your rules programmatically. 4.8 Summary Digester is an XML-to-Java mapping tool. However, it’s much more than just a mapping tool. It allows you to create, configure, and collate objects based on values specified in a configuration file. It does this by hiding the complexities involved in SAX parsing and callbacks, thereby making the interface easier and more configurable. Digester is configurable to the extent that you can have it use its own rules to set its own configuration environment. This is one of the stronger points of Digester, because it lets you specify the mapping between XML and Java at runtime rather than at compile time. Digester contains a rich set of rules and is configurable with external rules as well. By being namespace aware, Digester can bring together data from multiple organizations and provide a unified interface to object mapping. In the next module, you’ll learn about Betwixt, which is a data-binding tool; and JXPath, which is Java- based interpreter for XPath traversal of objects. Licensed to Tricia Fu MODULE 4: XML PARSING WITH DIGESTER 27 Index Digester changing rules for pattern matching, 8 creating own rules, 21 creating RuleSets, 25 externalizing own rules, 28 externalizing patterns and rules, 27 including rules from other files, 28 manipulating stack, 6 match() method, 9 matching patterns, 7 namespace aware parsing, 24 overview, 2 prebuilt rules CallMethodRule, 14 CallParamRule, 15 FactoryCreateRule, 10 NodeCreateRule, 11 ObjectCreateRule, 9 ObjectParamRule, 17 PathCallParamRule, 17 SetNextRule, 17 SetPropertiesRule, 12 SetPropertyRule, 13 SetRootRule, 18 SetTopRule, 18 reusing rules, 25 Rules interface, 8 simple example, 2 Stack, 5 presetting, 6 standard rules, 9 understanding matching, 8 understanding rules workflow, 8 wildcard patterns, 7 XML file traversal, 7 Digester rules methods, 8 DigesterLoader, 27, 28 DigesterRuleParser, 27 FromXMLRuleSet, 27 LIFO list, 5 RuleSetBase, 25 XML Æ Java object-mapping, 2 XMLRules, 27 XPath, 7 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF JXPath and Betwixt: working with XML Vikram Goyal MODULE MANNING 5 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 5 JXPath and Betwixt: working with XML 5.1 The JXPath story....................................................................................................................................................... 1 5.2 The Betwixt story.................................................................................................................................................... 21 5.3 Summary................................................................................................................................................................. 34 Index ............................................................................................................................................................................. 35 The need to cover two components in this module originated from the consolidation of similar components. JXPath and Betwixt, although different in concept, are complementary technologies that enable a higher degree of interaction between Java objects and XML data. The two technologies, used together, go a long way in simplifying the Java developer’s life. Betwixt resembles Digester, which we covered in the previous module, by providing a similar functionality of converting XML data to Java objects. JXPath is a handy tool if you want shorthand syntax for complex object access. JXPath and Betwixt solve small but very important pieces of a puzzle. Betwixt helps convert XML data to and from Java objects, whereas JXPath provides an expression language for making sense of complex data structures and objects. Betwixt can convert your data into a readable format, and JXPath can help by accessing this data—even the most complex parts of it. We’ll start with JXPath. It’s based on XPath, which is the syntax for traversing XML documents. Therefore, we’ll cover the basics of XPath before we dive into JXPath. We’ll explore Betwixt from two angles: the XML-to-Java path and the Java-to-XML path. We’ll help you understand how Betwixt operates by first explaining the concept of data binding and then diving into its structure. All through this discussion are plenty of examples to demonstrate how Betwixt operates. 5.1 The JXPath story JXPath, like the other Commons components, solves a very focused problem for the Java developer. It lets you simplify access to complex nested objects through a well-defined architecture. This architecture derives heavily from XPath, which we’ll introduce in section 5.1.2. Let’s start our introduction of JXPath by looking at a real-world problem that we’ll use as the basis for discussing this Commons component. As you understand more about JXPath, we’ll enhance the application to handle more complex functionality. 5.1.1 Understanding the problem: the Meal Plan application The whole idea of JXPath is to easily define paths to complex objects. Keeping this in mind, let’s create an application that tracks objects, which are hard to access unless you use complex iterations and loops. The goal is to create a Meal Plan application. The application specifies the meals to be eaten by a person on a diet and tracks the ingredients of each meal. The MealPlan class has start and end dates that default to Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF the current date and seven days from the current date, respectively. You can add meals to the meal plan. Each Meal class has an attribute for the day that meal is to be eaten (Sunday = 0, Saturday = 6), the type of meal it is (breakfast = 0, lunch = 1, dinner = 2, other = 3), and the name of the meal. Each meal can contain several ingredients. A key, a name, and an alternate qualify each Ingredient class. Figure 5.1 shows a simplistic relationship diagram for these domain objects. Figure 5.1 Class diagram for the Meal Plan application problem that shows the relationships between domain objects Let’s look at the full source code for these classes. This is our first attempt at creating these objects, and we’ll refine them as we explain more about the application and JXPath. We’ll start with the MealPlan class, shown in listing 5.1. Listing 5.1 The MealPlan class package com.manning.commons.chapter05; import java.util.Map; import java.util.Date; import java.util.Iterator; import java.util.Calendar; import java.util.ArrayList; import java.util.Collection; import java.util.GregorianCalendar; public class MealPlan { private Date startDate; private Date endDate; private Collection meals; public Date getStartDate() { return this.startDate; } public void setStartDate(Date startDate) { this.startDate = startDate; } public Date getEndDate() { return this.endDate; } public void setEndDate(Date endDate) { this.endDate = endDate; } public Collection getMeals() { return this.meals; } public void addMeal(Meal meal) { if(meal == null) throw new IllegalArgumentException("Meal cannot be added null"); c Declare meals collection Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 3 meals.add(meal); } public MealPlan() { this(null, null); } public MealPlan(Date startDate, Date endDate) { if(startDate == null && endDate == null) { startDate = new Date(); Calendar calendar = new GregorianCalendar(); calendar.setTime(startDate); calendar.add(Calendar.WEEK_OF_YEAR, 1); endDate = calendar.getTime(); } if(startDate == null || endDate == null || startDate.after(endDate)) throw new IllegalArgumentException("Please check the dates"); this.startDate = startDate; this.endDate = endDate; meals = new ArrayList(); } public static void main(String args[]) { MealPlan plan = new MealPlan(); plan.createMeals(); System.err.println(plan); } public void createMeals() { Meal pastaMeal = new Meal(0, 0, "Spicy Pasta"); Ingredient pasta = new Ingredient("PAS", "Pasta", "None"); pastaMeal.addIngredient(pasta); Ingredient tomatoes = new Ingredient("TMT", "Tomatoes", "Tomato Paste"); pastaMeal.addIngredient(tomatoes); this.addMeal(pastaMeal); Meal chickenMeal = new Meal(1, 2, "Honey Chicken"); Ingredient chicken = new Ingredient("CHK", "Chicken", "None"); chickenMeal.addIngredient(chicken); Ingredient honey = new Ingredient("HNY", "Honey", "Sugar Syrup"); chickenMeal.addIngredient(honey); this.addMeal(chickenMeal); } public String toString() { StringBuffer buffer = new StringBuffer(" **** Your Meal Plan **** "); buffer.append("\r\nStart Date: " + this.startDate); buffer.append("\r\nEnd Date: " + this.endDate + "\r\n\r\n"); Iterator itr = meals.iterator(); d Application’s main method Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF while(itr.hasNext()) { buffer.append(itr.next() + "\r\n"); } buffer.append(" ----------------------------------------------\r\n"); return buffer.toString(); } } c The meals collection can’t be set, because there is no setter method. This is a read-only property. d The main method lets us run this application with some default data. A plan is created, sample meals are added to it, and the resulting plan is printed on the screen. Listing 5.2 introduces the Meal class. Listing 5.2 The Meal class package com.manning.commons.chapter05; import java.util.Map; import java.util.HashMap; import java.util.Iterator; public class Meal { private int weekday; private int mealType; private String mealName; private Map ingredients; public int getWeekday() { return this.weekday; } public void setWeekday(int weekday) { this.weekday = weekday; } public int getMealType() { return this.mealType; } public void setMealType(int mealType) { this.mealType = mealType; } public String getMealName() { return this.mealName; } public void setMealName(String mealName) { this.mealName = mealName; } public Map getIngredients() { return this.ingredients; } public void addIngredient(Ingredient ingredient) { ingredients.put(ingredient.getKey(), ingredient); } Day meal is to be Type of meal Map-based collection of meal ingredients Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 5 public Meal(int weekday, int mealType, String mealName) { this.weekday = weekday; this.mealType = mealType; this.mealName = mealName; ingredients = new HashMap(); } public String toString() { StringBuffer buffer = new StringBuffer( "Meal name: " + this.mealName + ", Meal Type: " + this.mealType + " Meal Day: " + this.weekday); buffer.append("\r\nIngredients:\r\n"); Iterator itr = ingredients.values().iterator(); while(itr.hasNext()) { buffer.append(itr.next() + "\r\n"); } return buffer.toString(); } } Finally, let’s look at the Ingredient class in listing 5.3. Listing 5.3 The Ingredient class package com.manning.commons.chapter05; import java.util.Collection; public class Ingredient { private String key; private String name; private String alternate; public String getKey() { return this.key; } public void setKey(String key) { this.key = key; } public String getName() { return this.name; } public void setName(String name) { this.name = name; } public String getAlternate() { return this.alternate; } public void setAlternate(String alternate) { this.alternate = alternate; } Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF public Ingredient(String key, String name, String alternate) { this.key = key; this.name = name; this.alternate = alternate; } public String toString() { return "[Key=" + this.key + ", Name=" + this.name + ", Alternate=" + this.alternate + "]"; } } If you compile these three classes and run the MealPlan class, you’ll get a listing of the sample meal plan as created in the createMeals method of the MealPlan class. Note that this is only sample data; in a real- world application, this data would probably be gathered from the user. This is a decent application. However, it doesn’t give us fine-grained control over the constituents of the plan. For example, this application provides no way for the end user to know which ingredients are required for a particular day; this method could be handy, because it would let the user plan in advance. Being the good software developers that we are, let’s add a method for this purpose to the MealPlan class. We’ll call this method getIngredients; it takes an integer as a parameter, representing the day for which we want the ingredients. This method is shown in listing 5.4. Listing 5.4 getIngredients method to retrieve the ingredients required for a particular day public void getIngredients(int mealDay) { if(mealDay < 0 || mealDay > 6) throw new IllegalArgumentException("Invalid day!"); StringBuffer buffer = new StringBuffer(); Iterator itr = meals.iterator(); while(itr.hasNext()) { Meal meal = (Meal)itr.next(); if(meal.getWeekday() == mealDay) { Map ingredients = meal.getIngredients(); Iterator inner_itr = ingredients.values().iterator(); while(inner_itr.hasNext()) { buffer.append(inner_itr.next() + "\r\n"); } } } System.err.println("Ingredients are: \r\n" + buffer.toString()); } Phew! Although the method does what it’s supposed to do, it isn’t pretty. There are two iterations, and one of them is nested within the other. Could something help us make it easier to traverse nested elements and access objects buried within each other? The answer is yes, and the superhero in this case is JXPath. JXPath lets you access objects in cases like this with a simple traversal mechanism, which derives heavily from the XPath syntax (the traversal mechanism for elements within an XML document). JXPath not only lets you access simple objects and their properties, it allows access to collections, maps, DOM, and any other complex data structure you can imagine, including Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 7 mixed objects. For all these objects, common access uses the XPath syntax. We’ll get into XPath shortly, but listing 5.5 shows how you would transform the code in listing 5.4 to use the JXPath syntax. Listing 5.5 Transforming listing 5.4 to use JXPath public void getIngredients(int mealDay) { if(mealDay < 0 || mealDay > 6) throw new IllegalArgumentException("Invalid day!"); JXPathContext context = JXPathContext.newContext(this); System.err.println("Ingredients are: \r\n" + context.getValue("meals[weekday='" + mealDay + "']/ingredients")); } The output from running this listing is exactly the same as the output you get by running listing 5.4. Thirteen lines of code in listing 5.4 have been reduced to two lines in listing 5.5, and the logic has been greatly simplified. The advantage of JXPath should be obvious from this example. To appreciate JXPath fully, you need to first understand how objects are traversed. For this, as we said before, JXPath uses the XPath mechanism. Let’s look at XPath in greater detail. 5.1.2 Understanding XPath XML Path language (XPath) is a W3C standard for referencing and locating elements within an XML document. Using XPath, you can use a standardized method of naming parts of an XML document. In addition to providing a standard vocabulary for referencing parts within an XML document, XPath supplies a standard list of functions that can be used in a variety of situations. Let’s extend the example of the Meal Plan application further by looking at some sample data in XML format (see listing 5.6). Listing 5.6 A sample XML file that mirrors the createMeals method from listing 5.1 Pasta Tomatoes Chicken Honey Chicken Butter If you had to consistently reference a part of this XML document, how would you do it? One way is to say “The ingredient is a node of the meal element, which itself is a node of the mealplan, which is the root element of this file.” Further, how would you reference specific parts of this document? For example, would Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF you say, “The alternate to the Honey ingredient for the meal that is named Honey Chicken and that is eaten on weekday 1 is Sugar Syrup,” if you had to get to the Honey ingredient’s alternate? Probably not. XPath uses a standardized syntax for referencing the elements within an XML document. This syntax is very similar to the way file paths are specified. For example, to specify the directory called temp in your root folder in a Windows environment, you would probably write C:\temp. This way, everyone who reads this syntax is made aware that C:\ represents the root of your folder structure and that temp is a directory under it. XPath works in a similar way. Using the XPath syntax, the root element is written /[root element name]. Thus, using listing 5.6 as an example, the root element is /mealplan. Note: When you’re using JXPath, because the root element is used to create and initialize the JXPath context, you never use the / element or the root element name. Therefore, none of the examples using JXPath will ever start with a /, because it would technically represent the root element. As you follow the chain of elements, referencing other elements is simple. Thus, /mealplan/meal represents all the meal elements in listing 5.6, and /mealplan/meal/ingredient represents all the ingredients of all the meals in this document. To select attributes, you use the character @. Thus, to select all the attributes named key, you use the syntax /mealplan/meal/ingredient/@key. You can also use wildcards in XPath. Thus, /mealplan/*/ingredient is similar to /mealplan/meal/ingredient and matches all ingredients in all the meals in our meal plan. If the mealplan root element contained other elements besides meal, which themselves contained the element ingredient, this syntax would select those ingredients as well. To select actual elements, you can use square brackets, much as you use them in arrays. The major difference is that the first element is referenced using 1, not 0. Thus the first ingredient of the first meal is /mealplan/meal[1]/ingredient[1]. Using the square brackets, you can also use conditional element access: /mealplan/meal[ingredient] represents all meals that have an ingredient element. Although in our case this is true for all meals, this is a useful expression. You can make the expression more specific, to select an individual meal, by putting a conditional on the element within the brackets: /mealplan/meal[ingredient=Pasta]. This selects all meals that contain the pasta ingredient. The value of the ingredient element is on the right side of the equal sign. To select the ingredient itself, you can further qualify this as /mealplan/meal[ingredient= 'Pasta']/ingredient. The same principle can also be applied to attributes: /mealplan/meal[@weekday=1] selects all meals for weekday 1. The XPath standard also specifies several built-in functions that can help you traverse parts of a document. For example, if you don’t know beforehand as to how many meal elements are in the document, you can use the syntax /mealplan/meal[last()] to reference the last meal element in the document. Similarly, to get only the text contained within an element, you can use the text() function; for example, /mealplan/meal/ingredient/text() selects the text (the name) for all the ingredients in all the meals. A full list of XPath functions and possible expressions is listed at http://www.w3.org/TR/xpath. We recommend that you visit this site when you get stuck trying to figure out a possible path to a tricky XML document expression. Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 9 Now that you’re familiar with XPath and the problem JXPath solves using XPath as its base, it’s time to see how to use JXPath. We’ll introduce the JXPath Context component, which provides the basic services in a JXPath application. 5.1.3 The JXPath Context The JXPath Context component provides the services required to evaluate, retrieve and set logical values out of an XPath expression. This is the central component in JXPath and the one that you’ll use most while creating applications that use JXPath. You specify a JXPath context using the org.apache.commons.jxpath.JXPathContext abstract class. The org.apache.commons.jxpath.ri.JXPathContextReferenceImpl class provides a reference implementation. In your code, you create a context by calling the method newContext(Object contextBean) or the method newContext(JXPathContext parentContext, Object contextBean). In each case, the context bean becomes the root node around which the rest of the paths are evaluated. Recall that in listing 5.4, the context bean was the MealPlan class (using the this keyword). For most cases, the first method is the only one you’ll use. However, the second method is useful in cases where you want to create multiple instances of contexts with separate base classes for each of them. You can abstract the creation of the common properties in a parent context that has no base class (by creating the context as newContext(null)) and then use this parent context in all the child contexts via the second method. Behind the scenes, a new context is created using the implementation of the class org.apache.commons.jxpath.JXPathContextFactory. Thus, when you call either of the newContext() methods, the actual task of creating the context is delegated to the factory implementation that is registered with the system. The default implementation is org.apache.commons. jxpath.ri.JXPathContextFactoryReferenceImpl and is used if no other implementations are found. You can supply your own implementation for creation of contexts by extending the factory class and registering it by setting the system property org.apache.commons.jxpath.JXPathContextFactory (either by putting this property on the CLASSPATH or by putting this value in a file called jxpath.properties, which should be in your system’s JAVA_HOME). In most cases however, the reference implementation is sufficient. The relationship between these classes is shown in figure 5.2. Figure 5.2 The relationship between the classes that create a JXPath context Once a context is created, any path based on the root class can be evaluated or modified. However, before any operation can be performed on the path (and the underlying object), this path must be evaluated against the XPath specification. To do this, JXPath uses the concept of a compiler. Any operation on a path is first sent to a compiler, where the path is parsed to see if it crosschecks against the XPath specification. If it does, it’s evaluated as an expression and represented as a node. Also, to allow for faster reuse, successful expressions are Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF cached in a HashMap to enable lookups the next time the same path is operated on. This allows the compiler to be bypassed. Compilation is done using the classes in the org.apache.commons.jxpath.ri.compiler package; the classes in the org.apache.commons.jxpath.ri.parser package perform the parsing of a path. JXPathContext uses the parseExpression(String path, Compiler compiler) method in the class org.apache.commons.jxpath.ri.Parser to initiate the compilation. Note: It’s possible for you to call the compiler directly and initiate compilation and caching of an XPath. This is useful in cases where you have a small number of XPaths to work with that are called repeatedly, and precompiling them will give you a performance enhancement when they’re worked on. Since, as noted earlier, JXPath stores and looks up all compiled expressions in a HashMap, using the XPath itself as the key, compilation won’t be attempted on precompiled XPath expressions. To compile an XPath expression yourself, use the static method compile(xpath), which is in the JXPathContext class. All this processing happens behind the scenes without the end user realizing it. All the user has to do is to call one of the methods provided by the JXPathContext class to perform operations on a path. We’ve already come across one of these operations for getting the value out of an XPath when we showed you the code in listing 5.4. In the next few sections, we’ll explore more of these operations. In this process, we’ll expand on the application discussed in section 5.1.1 by adding new features and enhancements. 5.1.4 Reading data You know that JXPath can be used to read data from objects that are nested and difficult to access—you saw an example in listing 5.5, where 13 lines of code from listing 5.4 were reduced to 2 lines. Let’s take a step back and see how to read data from a simple object—or, rather, how to access a simple JavaBean property. Reading a simple object To read data, you first need to have a context (discussed in section 5.1.3). A new context is created around a context bean. Continuing with the Meal Plan application, this context bean is the MealPlan class, and the context is created as follows: JXPathContext context = JXPathContext.newContext(this); You can access properties from either this context bean itself or any of the nested beans within it. Keeping it simple, we access the startDate property of the MealPlan class using context.getValue("startDate"); or context.getValue("@startDate"); Both cases are equivalent and produce the same result, which is the value of the startDate property of the MealPlan class. The second method makes it explicit, since the value we’re after is an attribute (property) of the current context bean (this difference is important when you’re parsing XML data instead of a JavaBean, as Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 11 you’ll see later). This value is returned as a Date object, which is the type of the startDate property. If the property type had been a primitive, it would have been converted to an equivalent object type. For example, if the property was of type int, it would be converted to Integer. The method Object getValue(String xpath) has an overloaded method, Object getValue(String xpath, Class requiredType). This method can be used when the result needs to be cast to a particular class type. Of course, you can still use the first method, but you’ll need to do the casting yourself. If JXPath can’t find a property within the context bean with the name you’ve specified, it throws an exception and stops further processing. However, you can configure JXPath to be lenient by calling the method setLenient(true) on the context. After you call this method, any incorrect calls return null instead of throwing an exception. On its own, the ability to access the property of a context bean isn’t useful. The real power of JXPath comes from the ability to access nested beans and their properties (and maps and collections). To access nested properties (and objects), the correct XPath to that object needs to be created. However, the basic access is through the same method, getValue, and therefore the real power here lies in the XPath that is created. Reading a nested object To show an example of nested bean access, let’s add a new property to the MealPlan class, which in itself is a bean. This new bean is called snack, is of type Meal, and contains a single ingredient: private Meal snack; : : private void setSnack(Meal snack) { this.snack = snack;} private Meal getSnack() { return this.snack; } : : // in the createMeals() method, 7 indicates that this meal does not occur on // any particular day snack = new Meal(7, 3, "Fries"); snack.addIngredient(new Ingredient("POT", "Potatoes", "None")); To access the nested bean snack, we use the following syntax: context.getValue("snack"); or context.getValue("@snack);. As you can see, there is no difference between accessing a simple property or a property that is itself a nested bean. After all, a nested bean is a property. To access the properties of the nested bean, we can use any of the following methods: context.getValue("snack/@ingredients"); context.getValue("snack/ingredients"); context.getValue("@snack/@ingredients"); context.getValue("@snack/ingredients"); These methods return the same thing: all the ingredients of the nested bean snack. Of course, it helps to be consistent in your programming, and you should decide and stick to one format. Note that the nested bean access can be infinitely deep. Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF Caution: Remember not to put a forward slash (/) at the end of your XPath expression. It would be considered as an invalid expression. It’s easy to forget this and start putting a slash at the end of your expressions, but JXPath will complain when you run your program because such expressions are considered incomplete. What happens if we want to access the alternate property of the ingredients of the snack? Can we do the following to get it? context.getValue("snack/ingredients/alternate"); // INCORRECT No! There is a problem with the contained XPath. Because the ingredients property is of type HashMap, we need to construct an XPath to the specific ingredient reference; otherwise JXPath has no way of knowing which value to return. But how do you reference a value contained in a HashMap? Why, with its key, of course. The same is true for other map types and collections. Reading maps and collections To access specific properties within a map, you need to know the key; and to access specific properties within collections, you need to know the index of the property within the collection. Thus, to access the alternate for the Potato ingredient, you can use the following syntax: context.getValue("snack/ingredients/POT/alternate"); or context.getValue("snack/ingredients[@name='POT']/alternate"); The first syntax treats the Potato ingredient the same as if it was a node within the expression. The second syntax assigns a name property to the HashMap ingredients for lookup purposes. You can use the same syntax with normal properties. The first syntax treats POT as an identifier. Identifiers in JXPath can’t contain illegal characters, so you can’t use the first syntax in places where you know the key may contain these characters. For example, if there may be a key for an ingredient that contains characters like %, you should use the second syntax to guarantee that you won’t get a runtime error. You can use either syntax, but be consistent. Note that the key is case sensitive, as expected for map-based access. To access collection- or array-based properties, you can use the index of the specific property you’re after. However, remember that the access starts with 1, not 0. Thus to access the first meal name in our MealPlan class, we use the following syntax: context.getValue("meals[1]/mealName"); Combining map- and collection-based access; we can use the following syntax to get the alternate for tomatoes for the Spicy Pasta dish: context.getValue("meals[1]/ingredients/TMT/alternate"); Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 13 Accessing Properties of Map-Based JavaBeans How do you access properties of a map-based JavaBean? Consider that instead of using the standard HashMap, we use a custom implementation of HashMap for storing ingredients. Further, this custom implementation, which extends HashMap, has a property called noAlternateCount, which is a simple counter that tracks the number of ingredients for which no alternate is specified. How can we access this property? We can try this: context.getValue("meal[1]/ingredients/noAlternateCount"); // INCORRECT Unfortunately, this code won’t work for two reasons. The first should be obvious from the fact that the access is being performed on a general value rather than a specific value. Even if we were to make it specific by, for example, using the ingredients as the context bean, the second reason still won’t let it work. The second reason is subtle. Let’s say we use one of the custom HashMap-based ingredient objects as our context bean. Now, we should be able to use the following: context.getValue("noAlternateCount"); // STILL INCORRECT But this doesn’t work either. Why? Because there is no way for JXPath to know whether we’re looking for a value based on a key or looking for a property of the ingredients object. The default—and the safer option that JXPath chooses—is to assume we’re looking up the value for a key; so, it tries to look it up from its internal HashMap. It may or may not find it, based on whether there is a key in the HashMap called noAlternateCount. There is no way to switch between this behavior and the alternate (and preferred, in this case) behavior of looking for a property of this Map-based object. Therefore, unfortunately, there is no way in JXPath to retrieve the value of the property noAlternateCount. We said earlier that access to collection- and map-based values must be specific. Although this is technically true, in some cases you may want to get all the possible values. This is true not only for collection- and map- based access but in cases where there may be multiple return values Of course, getValue isn’t useful in these cases, because it only returns a single value; and, in cases where multiple results are expected, it only returns the first value. In the next section, you’ll see how to retrieve multiple values from a JXPath path. Reading multiple values In listing 5.5, we used getValue erroneously. That gave us only the first correct result. Instead, we should have used the equivalent of getValue for multiple results and called iterate, as shown in listing 5.7. Listing 5.7 Using iterate when multiple results are expected public void getIngredients(int mealDay) { if(mealDay < 0 || mealDay > 6) throw new IllegalArgumentException("Invalid day!"); JXPathContext context = JXPathContext.newContext(this); Iterator itr = context.iterate("meals[weekday='" + mealDay + "']/ingredients"); System.err.println("Ingredients are: \r\n"); Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF while(itr.hasNext()) { System.err.println(((HashMap)itr.next())); } } As you can see, the iterate method returns an Iterator over the expected results. The type of the expected results is the same for a single result, which in this case is a HashMap. If no results are found, an empty Iterator is returned. One problem with our Meal Plan application is that it uses an internal method (createMeals()) to create and initialize the meal plan. In reality, this wouldn’t be the case. In the next section, you’ll see how to create the Meal Plan application from an external file. Reading from a file A valid scenario for creating our meal plans would involve reading from an external XML file (which could be generated by reading it from a database). You saw an example of such a file in listing 5.6, which shows a sample XML file for our Meal Plan application. What happens if we want to traverse this XML file rather than the MealPlan created using the createMeals method? The answer is in an interface called org.apache.commons.jxpath.Container, which is implemented using org.apache.commons.jxpath.xml.DocumentContainer. The concept of a container lies in the fact that for JXPath, the value of a container-based node is the data contained within that container, not the container itself. This concept lets you associate container-based nodes within your XPaths; JXPath treats these paths transparently. The container may point to some data in memory, in a file, or over the network, and it’s used to point to the actual data when it’s parsed. JXPath comes with a default container called DocumentContainer, which can be used to parse XML documents. To use listing 5.6 as our initialization data, we need to modify the MealPlan class. First we need a method to load the XML file. This is shown in listing 5.8. Listing 5.8 Loading a container using DocumentContainer public Container loadMealPlan() { URL url = getClass().getResource("/sampleXML_5.6.xml"); return new DocumentContainer(url); } Using the DocumentContainer is easy: You point it to a URL that contains the XML data to be loaded. Using it now is no different from the way we used the previous MealPlan: Container alternatePlan = plan.loadMealPlan(); JXPathContext context = JXPathContext.newContext(alternatePlan); System.err.println(context.getValue("mealplan/meal[1]/@name")); As you can see, nothing much has changed. Instead of using the current MealPlan object, we’re using the alternatePlan loaded from the XML file using the loadMealPlan method. What has changed significantly is that we need to specify the root node in XPaths we create, which in the XML file is mealplan. Of course, this is only necessary based on the structure of our XML file. An alternate scenario may involve loading only the meal information from an XML file. In that case, the getMeals() method of the MealPlan class would load the XML file, which would only contain meal elements bounded by a generic root node. Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 15 Note that the JXPath uses an internal XMLParser to parse XML files. It loads and parses these files only on the first attempt at reading a node within this file, not when the DocumentContainer is constructed. Now that you know how to read data using JXPath, it’s time to learn how to modify this data. The next section covers this topic. 5.1.5 Modifying data The easiest way to modify data is to use the setValue method. You need to specify what to modify and what to change it to. The rules for specifying what to modify are the same as the rules for reading data, which we covered in the last section. Modifying simple properties Continuing with the first example from the previous section, to modify the startDate property of the MealPlan class, we use the following method: setValue("startDate", new Date()); This lets us change the startDate. The type of the new value must be the right type, or we’ll get a runtime exception. Also, a setter method must be available in the bean we’re trying to modify. Similarly, we can change the alternate for the Tomato ingredient for the first day’s meal: context.setValue("meals[1]/ingredients[@name='TMT']/alternate", "Ketchup"); JXPath also allows you to create and set objects as nodes. This is done using an abstract factory class called org.apache.commons.jxpath.AbstractFactory. This class defines two nonabstract methods, both of which return false. The method of interest for creating objects is the createObject method. In the next section, we’ll show you how to create objects using this method. Creating objects To create objects, you must first extend the AbstractFactory class. For example purposes, let’s say that we provide a factory for creation of snacks. This factory is shown in listing 5.9. Listing 5.9 Factory class for creating snacks: creating objects using JXPath package com.manning.commons.chapter05; import org.apache.commons.jxpath.Pointer; import org.apache.commons.jxpath.JXPathContext; import org.apache.commons.jxpath.AbstractFactory; public class SnackFactory extends AbstractFactory { public boolean createObject(JXPathContext context, Pointer pointer, Object parent, String name, int index) { if((parent instanceof MealPlan) && "snack".equals(name)) { ((MealPlan)parent).setSnack(new Meal(7, 3, "Fries")); return true; Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF } return false; } } The createObject method must check for the validity of the data that is passed. In this case, it ensures that we only create snacks for the MealPlan parent. We create a Meal and set it as a Snack on the parent, but we haven’t yet set any ingredients on it. On successful creation of objects, the method must return true To use this factory, we need to set it on the current context by calling the method context.setFactory(new SnackFactory()). This allows the current context to know that if there is a null value for the snack property, it can be created using this factory. Note the emphasis on the null value: JXPath doesn’t create objects if they already have a value. Once this factory is set in the current context, we can use it to create a snack for our MealPlan class by using createPath("snack"). Note that we must remove the creation of the snack property from the createMeals method for this to work, because, as we said before, this method won’t work if there is already a current snack value. Even though this method works for creating the snack object, there is a shortcoming in the snack that we created: It doesn’t contain any ingredients! This situation can be rectified by either setting the ingredients in the createObject method or setting this value using the setValue method, as shown here: context.setValue("snack/ingredients[@name='POT']", new Ingredient("POT", "Potatoes", "None")); Because creating objects and setting their value are common occurrences, there is a shorthand method for doing both: context.createPathAndSetValue("snack/ingredients[@name='POT']", new Ingredient("POT", "Potatoes", "None")); This method combines creation of the snack property and the setting of its ingredient in an easy-to-use signature. How do you create a factory for properties that are based on Collection, Map, or Array? The next section illustrates by creating a factory for Meal creation. Creating Collection-based factories Recall that meals are kept as a collection (ArrayList) in the MealPlan class. Therefore, when we’re creating the factory, we need to be sure that a collection exists in the parent and then create an empty meal and add it to this collection. This process is shown in listing 5.10. Listing 5.10 Factory for creating collection-based properties for meals in the MealPlan class package com.manning.commons.chapter05; import java.util.ArrayList; import java.util.Collection; import org.apache.commons.jxpath.Pointer; import org.apache.commons.jxpath.JXPathContext; import org.apache.commons.jxpath.AbstractFactory; public class MealFactory extends AbstractFactory { Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 17 public boolean createObject(JXPathContext context, Pointer pointer, Object parent, String name, int index) { if((parent instanceof MealPlan) && "meals".equals(name)) { if(index < 0) return false; ArrayList meals = (ArrayList)((MealPlan)parent).getMeals(); meals.add(index, new Meal(0, 0, "Unknown Meal")); return true; } return false; } } Notice that the name parameter must always be equal to the expected property of the parent. The index parameter tells us where this value should be inserted in the collection. This index is zero-based and has been decremented by one by JXPath before sending to this method. We know that the parent would create a meals collection as part of its constructor, so we don’t check for the existence of one. If we knew for sure that the parent constructor wouldn’t create a collection, we would have to check for its existence and create one if one wasn’t present. Finally, we add a new Meal at the specified index to the meals collection. To use this factory, we first set it in the context by calling context.setFactory(new MealFactory()), and then we use context.createPath("meals[1]") to create it or use the context.createPathAndSetValue method to create and set the properties of the newly created meal. The value in the square brackets is the index that is passed to the createObject method, after decrementing it by 1; therefore, if there is no meal at collection index 0, for meals[1], one is created. Before we move on to consider the other cool things that you can do with JXPath, we need to discuss two important concepts. The first has to do with variables and the second with pointers. 5.1.6 Understanding variables and pointers Variables represent a way that you can define parameters to be passed to JXPath. Variables are just like instance variables in Java: Once registered, these variables can be accessed from the context they were registered with at any time. Variables can be modified and removed as well. To create and register a variable, you use the method declareVariable(String varName, Object value). However you do this not on the context but on the interface org.apache.commons.jxpath.Variables. JXPathContext defines a getVariables() method that returns an implementation of this interface. The default implementation, which uses a HashMap internally, is called BasicVariables and can be overridden by using the setVariables(Variables vars) method on the JXPathContext. Once registered, variables can be treated like nodes. You can change their value by using the setValue method on the context. Variables are differentiated from the rest of the XPath syntax by putting a $ character in front of the variable name. At compile time (compilation of expressions, that is, not Java compilation), JXPath removes the variable and replaces it with the actual variable value. The following lines of code show how you can a variable to reference the mealDay parameter in listing 5.7: Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF context.getVariables().declareVariable("mealDay", new Integer(mealDay)); Iterator itr = context.iterate("meals[weekday=$mealDay]/ingredients"); You can reset the value of variables the same way you would set the value of properties using JXPathContext. For example, setValue("$mealDay", new Integer(2)) changes the value of the mealDay variable. Note that variables can be created on any types, not just the primitive types. This allows you to create variables on Collection, Map, and Array as well. Finally, variables can be deregistered by calling the undeclareVariable(varName) method of the Variables interface. Pointers! I can almost hear you scream and shout and say that JXPath, being Java-based, should have nothing to do with pointers. Well, hold on. These pointers are simple location markers (pointers) to nodes within an XPath and have nothing to do with memory management (but, like real pointers, they’re a reference to something—hence the name pointer, because they point to a node). Think of pointers as inherent nodes, without the value they may represent. Representing nodes (properties) as pointers can be useful. Consider the case where you need to repeatedly use the value represented by the same XPath expression. JXPath is optimized to handle XPaths that are represented as pointers better than paths that aren’t. Thus using pointers in such situations is a performance enhancement. Since pointers are dependent on the XPath they represent, they’re context-insensitive and therefore can be interchanged between contexts. Pointers are specified by the interface org.apache.commons.jxpath.Pointer. The org.apache.commons.jxpath.ri.model.NodePointer superclass implements this interface. This class is called the superclass because it’s the parent class to several kinds of pointer classes. Since a pointer maps to actual paths within an XPath, several pointer classes are needed to represent the possible paths. The org.apache.commons.jxpath.ri.model.VariablePointer class represents a pointer to a context variable, and the org.apache.commons.jxpath.ri.model.beans. CollectionPointer class is used to point to a collection. Similarly, the org.apache. commons.jxpath.ri.model. beans.PropertyPointer class represents a pointer to the property of a bean and it itself is subclassed to represent several variations on bean properties. You access the pointer to a particular XPath by using the method getPointer(String xpath) on the JXPathContext. You can get an iteration of all possible pointers to paths that represent variable results by using the iteratePointers(String xpath) method. On the Pointer object that is returned by these methods, you can use the getValue() and setValue(String value) methods to get and set the object, property, or collection that the pointer represents, respectively. Now that you understand what variables and pointers are in relation to JXPath, let’s expand this knowledge by looking at how they’re useful. Using variables and pointers Combining variables and pointers allows you to access concrete locations of your XPaths. For example, suppose you use a variable as we discussed in the previous section: context.getVariables().declareVariable("mealDay", new Integer(mealDay)); Iterator itr = context.iterate("meals[weekday=$mealDay]/ingredients"); This code doesn’t tell us the exact locations of all the meal ingredients that were accessed. To access this information, we use pointers as shown here: Iterator itr = context.iteratePointers("meals[weekday=$mealDay]/ingredients"); while(itr.hasNext()) { System.err.println(((Pointer)itr.next())); } Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 19 Note that you don’t need to have separate code for iterating through the values and path information. If you’re interested in the value of a path as well as exact path information, then use the pointer code, and use the getValue method of the Pointer interface to access the value: This is illustrated here: while(itr.hasNext()) { Pointer pointer = (Pointer)itr.next(); System.err.println(pointer + ": " + pointer.getValue()); } Pointers are also useful in the area of extension functions. We haven’t covered these functions before, and it’s time to look at them. We said earlier that since JXPath is based on XPath, it supports the functions provided by the XPath specification. So, for example, if we wanted to find out the number of meals in the context bean, we could use the standard XPath function count() as shown here: System.err.println("Number of meals: " + context.getValue("count(meals)")); In addition to calling built-in XPath functions, you can also call existing methods and constructors of classes or create your own functions. The former are called standard extension functions, and the latter are called custom extension functions. Using standard extension functions is as simple as using fully qualified names of the classes that are being used. For example, we can create a new MealPlan class: context.getVariables().declareVariable("startDate", startDate); context.getVariables().declareVariable("endDate", endDate); System.err.println(context.getValue( "com.manning.commons.chapter05.MealPlan.new($startDate, $endDate)")); As you can see, we declare the parameters to be passed to the constructor as variables before we use them. Declaring them before allows us to register any types of parameters; in this case, these parameters are of type Date. The actual object is created by using the new method and by passing in the parameters (variables). An empty constructor can be called without passing anything in the new method. You can also call normal methods of a class by using the functions. To do so, you have to specify the variable on which the method is to be invoked as the first parameter of the method. Therefore, to get the startDate in seconds, you can use the syntax System.err.println(context.getValue("getTime($startDate)")); where the startDate variable has been defined and registered in the JXPathContext. Note that we’re invoking the getTime method of the Date class, but you can invoke any method on any class. Although calling preexisting methods is useful because it gives you flexibility in dealing with object graphs, it doesn’t allow the same level of functionality as the standard XPath functions. Let’s see how we can use these functions. Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF Using custom extension functions JXPath allows you to create and register custom extension functions, which provide the same level of flexibility and conciseness that is provided by the XPath functions. Custom extension functions are specified using the org.apache.commons.jxpath.Functions interface and implemented using the following three classes in the same package: ƒ ClassFunctions registers a single set of functions (defined in the same class). ƒ FunctionLibrary groups several functions. Because JXPath can register only one function at a time, you need to use this class to aggregate all your functions and then register this class. ƒ PackageFunctions registers functions grouped together in a package. Thus registering the package java.lang will register all classes in this package. To illustrate the use of custom extension functions, let’s add a new class in our Meal Plan application that will be used to register these functions. At the moment, this class contains a single method to find out the difference between the start and end dates. This is shown in listing 5.11. Listing 5.11 Adding custom extension functions: MealPlanFunctions utility functions package com.manning.commons.chapter05; import java.util.Date; import java.util.Calendar; import java.util.GregorianCalendar; public class MealPlanFunctions { public int getDateDiffInDays(Date startDate, Date endDate) { Calendar calendar = new GregorianCalendar(); calendar.setTime(startDate); int firstWeek = calendar.get(Calendar.DAY_OF_YEAR); calendar.setTime(endDate); int secondWeek = calendar.get(Calendar.DAY_OF_YEAR); return (secondWeek - firstWeek); } } As you can see, this code is a simple Java class and has no interface as such with JXPath. It becomes interesting and usable when we register it with JXPath, as shown in the partial code in listing 5.12. Listing 5.12 Partial listing that uses a custom extension function class context.getVariables().declareVariable("startDate", startDate); context.getVariables().declareVariable("endDate", endDate); context.getVariables().declareVariable("mpf", new MealPlanFunctions()); context.setFunctions( new ClassFunctions(MealPlanFunctions.class, "mpfunc")); Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 21 System.err.println("Date Difference: " + context.getValue( "mpfunc:getDateDiffInDays($mpf, $startDate, $endDate)")); First we declare the variables that will be used later in this context. Notice that MealPlanFunctions is also declared as a variable. The MealPlanFunctions class is registered as a ClassFunction with the JXPathContext and given the namespace mpfunc. The function is used within the JXPath context by prefixing it with mpfunc:. As you can see, custom extension functions are registered using the setFunctions method on the JXPathContext. You can install only one set of functions; if you want to install several classes of functions, use the FunctionLibrary class, register all the functions with this class, and register the library instead. One important thing to notice from this code listing is that we had to declare the MealPlanFunctions class as a variable, and we used it later while retrieving the value as the first variable in the function (mpfunc:getDateDiffInDays($mpf, $startDate, $endDate)). This behavior is similar to the standard extension functions, discussed earlier. However, if the method in the function class that is being called is static, we can bypass this variable and only pass the function’s actual parameters. JXPath uses the first variable of a function as the object on which the function is being called, except for static methods. To get a handle on the current node’s context in your custom function methods, you can add an extra parameter to your methods, of the type org.apache.commons.jxpath.ExpressionContext. JXPath recognizes that your method requires the context and passes the current one automatically. This can be used to recognize the properties of the current context. Note that it only works for class and package functions. Listings 5.11 and 5.12 are good for use when the variables that are passed to the custom function are singular in nature. But what happens when we need to pass multiple values, like a collection or an array? Further, what happens when the return type is also a multiple value? The answer is, nothing special. Passing a collection is the same as passing a regular variable. You can alternatively use an org.apache.commons.jxpath.NodeSet implementation, which is a list of the nodes you want to pass. You can use the concrete implementation BasicNodeSet when passing values back from a custom extension function as a collection of pointers. This is useful when you want to send values back from a function as a collection of node pointers rather than the as values themselves. This section completes our discussion of the JXPath component. Although JXPath is a useful tool on its own, combining it with Betwixt provides a comprehensive solution to your XML processing needs. Let’s look at Betwixt next. 5.2 The Betwixt story Betwixt is a Commons tool that lies in the category commonly classed as data-binding tools. Before we delve into Betwixt proper, let’s take a brief look at data-binding concepts. 5.2.1 Understanding data binding Data binding is the process by which data is transferred from one form to another. Although it’s primarily used in conjunction with XML, data binding is a more generic concept and can be used to describe data in any format. The transformation from one form to another is inherently considered to be bidirectional. Why is it called data binding? Well, one explanation is, because the two forms of data are considered bound to each other. Thus, technically, a change in the data in one form should automatically be reflected in the second form. Another explanation is that in business processes, the form of the data isn’t important, what Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF is important is the data itself, and data binding provides a mechanism for different forms of data to be seamlessly accessible. Whatever the reasons for the name, the concept is straightforward. Data binding is the process of two- way transformations between data in different formats, with the emphasis on the actual data without regard to the structure of the data itself. Since we’re concerned here with Java- and XML-based objects, we can further refine this concept. For our purposes, data binding is the two-way transformation between Java objects (JavaBeans) and XML data. The process by which Java objects are converted to corresponding XML representation is called marshalling, and the reverse conversion from XML to Java objects is called unmarshalling. Finally, as we said before, although we’re only concerned with Java and XML data binding, the concept can be extended to other formats as well. Thus, bidirectional transformations between XML data and a relational database tables/columns is also called data binding. 5.2.2 Betwixt structure Betwixt closely follows the JavaBean structure for maintaining, accessing, and modifying information about a JavaBean and translating this information back and forth between its XML representations. The classes in the Sun package java.beans provide interfaces and concrete classes that give you information about a bean. Betwixt uses a similar mechanism: Specifically, it uses classes called XMLIntrospector and XMLBeanInfo that mirror the work done by the Introspector and BeanInfo classes from the java.beans package. Because these classes are so similar, let’s look at those from the java.beans package first. This will help you better understand the process behind the Betwixt classes. Java’s BeanInfo and Introspector classes Introspector is a concrete class in the java.beans package. It provides a standard way to learn more about a class, be it a JavaBean or not. If you call the static method getBeanInfo(Class targetClass), passing it the class that you want to learn more about, it returns a BeanInfo object that contains information about the target class. Introspector works by first looking for an explicit BeanInfo class for your target class. Thus, as a conscientious developer, you may decide to provide a BeanInfo class for your target class that tells anyone who cares details about your target class over and above what the introspector can gather by itself using low- level reflection. If the introspector can locate such an explicit BeanInfo class about your target class, it merges this information with its own implicit reflection analysis on your target class and returns concrete BeanInfo information combining the two. So, BeanInfo is information about your JavaBean packaged in a convenient class. However, BeanInfo is only an interface; if you decide to provide a BeanInfo class for your target class, you’ll either need to implement this interface or use the SimpleBeanInfo class, extend it, and override the methods you want. The information about your target class should be provided using the MethodDescriptor, PropertyDescriptor, EventDescriptor, or BeanDescriptor class. Each of these class names should be self-describing. We recommend that you look up the java.beans package in the Java API for more information about these classes. Figure 5.3 shows the relationship between the classes we just discussed. Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 23 Figure 5.3 Relationships between important classes of the java.beans package Now that you know more about these base Java classes, let’s look at how Betwixt uses the same principle in its XMLBeanInfo and XMLIntrospector classes to inspect information about Java objects. Reading objects using XMLBeanInfo and XMLIntrospector If we were to break down the whole bean Æ XML Betwixt process into concrete steps, the first step would involve the XMLIntrospector class gathering information about the target class by searching for any java.beans.BeanInfo information, the second step would use this information to create an XMLBeanInfo object, and the final step would write this information out as XML using the BeanWriter and AbstractBeanWriter classes. These steps are shown in figure 5.4. Figure 5.4 Steps in the bean Æ XML process using Betwixt In this section, we’ll discuss the first two steps. The next section will look at the classes of step 3. The XMLIntrospector class uses the java.beans.Introspector class to analyze the target class before it’s written out as XML. The default introspection using java.beans.Introspector can be overridden by providing a .betwixt file, which contains information about the target class. This file must be named the same name as the target class (and an extension of .betwixt) and must be available in the CLASSPATH. The XMLBeanInfo class uses org.apache.commons.betwixt.NodeDescriptor-based classes to encapsulate information. The org.apache.commons.betwixt.ElementDescriptor class defines information about the elements to be present in the XML, and org.apache.commons.betwixt. AttributeDescriptor defines information about the attributes. Both these classes derive from the NodeDescriptor class and are similar to the way MethodDescriptor and other classes define information about a JavaBean. Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF There is an internal process for caching preprocessed classes. Before a target class is introspected (a potentially time-consuming process), an internal registry is consulted. If this registry, defined by the interface org.apache.commons.betwixt.registry.XMLBeanInfoRegistry and implemented using org.apache.commons.betwixt.registry.DefaultXMLBeanInfoRegistry, doesn’t contain XMLBeanInfo information about the target class, only then is the introspection continued. After the introspection is performed, this information is stored in the registry using the target class as the key. You can turn off this caching by using org.apache.commons.betwixt.registry. NoCacheRegistry instead of the default registry. This registry—indeed, any registry you create—can be set on the XMLIntrospector by calling setRegistry(XMLBeanInfoRegistry registry) method. Even before the target class is looked up in the cache, the target class is normalized. Normalization is the process by which you can make changes to the target class before it’s introspected. These changes can range from making changes to the superclass of your target class that should be introspected to supplying interface information that a class implements. This process allows you to make changes in the target class, thus restricting or enabling the class information before it’s written out. The default normalization, implemented by the class org.apache.commons.betwixt.strategy. ClassNormalizer, makes no changes to your class and returns it as it is. You can extend this class and enable your own normalization procedure by setting this class as the normalizer Betwixt should use by calling setClassNormalizer(ClassNormalizer classNormalizer) on the XMLIntrospector. Betwixt comes with another normalizer called ListedClassNormalizer, which allows you to substitute a target class for another class by calling the method addSubstitute(Class targetClass, Class substituteClass). This normalizer is useful for cases when you may want to restrict introspection of your classes to a standard set. Betwixt writes out XML representations by using the classes AbstractBeanWriter and BeanWriter, which appear in the third step of the Betwixt bean Æ XML process shown in figure 5.4. The next section looks at these classes. Writing objects using AbstractBeanWriter and BeanWriter AbstractBeanWriter is the superclass of BeanWriter. It contains concrete code to process the target class, whereas BeanWriter contains code to write the output as an XML stream. These classes, and the corresponding reader class called BeanReader, are in the org.apache.commons.betwixt.io package. When Betwixt is writing your target bean as XML, it uses the org.apache.commons. betwixt.BindingConfiguration class to resolve three runtime issues. This class is a placeholder for resolving these issues and contains methods to set and get these issues as properties. This class can be set with the correct properties and attached to the AbstractBeanWriter (or the BeanReader, because the same configuration can be used for both reading and writing) by calling the method setBindingConfiguration(BindingConfiguration config). The three issues in question are as follows: ƒ Matching cyclic elements ƒ Overriding the bean type ƒ Object to String conversions The first issue involves using an ID for matching cyclic elements. When an XML file is being written out, elements that reference each other, leading to a cyclic check, can be referenced using ID and IDREF attributes. This is the default behavior of Betwixt. However, you can change it by setting setMapIDs(boolean mapIDs) to false in the BindingConfiguration class. As a side note, the Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 25 classes in the package org.apache.commons.betwixt.io.id can be used to customize the value of the ID and IDREF attributes. By default, Betwixt uses a sequential ID generator, SequentialIDGenerator, which starts at zero. An ID generator must implement the org.apache.commons.betwixt.io. IDGenerator interface or extend the org.apache.commons.betwixt.io.id. AbstractIDGenerator class. Another ID generator, besides the sequential one, is RandomIDGenerator. The second issue deals with the class name attribute used to override the type of a bean. By default, Betwixt uses className. You can set it to your actual class name, or to its superclass name, or to anything you desire. Note that you must use the same name (or the same BindingConfiguration class) when reading the XML back in, to ensure that your object is created exactly as it was originally. The final issue that BindingConfiguration allows you to handle is the class that will be used for Object to String conversions. Such conversions, as may be obvious, are used to represent String versions of Objects. However, if you’re scratching your head wondering how it’s different from a toString() operation, hold on to your hats: We’re talking about a round-trip conversion during which Objects are converted to String representations and String representations are converted to their original Object types. Betwixt’s default mechanism for doing this uses the ConvertUtils package of the BeanUtils component, which we’ll discuss in module 7. So far, we’ve concentrated on writing beans as XML. In the next section, we’ll cover the XML Æ bean process, which is based on the Digester component. Converting XML to beans On the other end of the bean Æ XML spectrum is the XML Æ bean process. It relies on the org.apache.commons.betwixt.io.BeanReader class, which is the primary class for reading an XML document and converting it to a tree of beans. BeanReader uses a subset of the Digester component to perform its functionality, and the BeanReader class extends the org.apache.commons. digester.Digester class. It makes sense, because Digester, which we discussed in the previous module, provides functionality to create beans read from an XML file. The Digester subset classes are placed in the package org.apache.commons.betwixt.digester. You need to register the top-level element in the XML document as a JavaBean class with the BeanReader before you can parse a document. You can do this either by calling registerBeanClass(Class clazz) or by calling registerBeanClass(String path, Class clazz). The second method allows you to match the top-level element in case it differs from the name of the actual class, or if the top-level element isn’t the element that you want created. In this case, the path is an XPath traversal to the actual element within the document. This rounds out our coverage of the Betwixt structure. It’s time to look at some code and see Betwixt at work. 5.2.3 Betwixt in action We’ll start working with Betwixt by looking at a simple round-trip operation of converting a JavaBean to an XML representation and then back from that XML representation to the original JavaBean. Note: Before you try any of these examples, make sure you have the commons-collections, commons- logging, commons-digester, and commons-beanutils libraries in your CLASSPATH, in addition to the Betwixt library. Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF Simple round trip This example of using Betwixt will show how you can convert a bean Æ XML and then use the same Betwixt-created XML representation to arrive at the original bean. Listing 5.13 uses the Ingredient class from listing 5.3 to show this round-trip conversion (note that this code will not work if the Ingredient class isn’t changed to include an empty constructor). Listing 5.13 Simple round-trip conversion example package com.manning.commons.chapter05; import java.io.File; import java.io.FileWriter; import org.apache.commons.betwixt.io.BeanReader; import org.apache.commons.betwixt.io.BeanWriter; public class SimpleRoundTrip { public static void main(String args[]) throws Exception { FileWriter fWriter = new FileWriter("output.xml"); BeanWriter bWriter = new BeanWriter(fWriter); Ingredient ingredient = new Ingredient("POT", "Potato", "None"); bWriter.write("ingredient", ingredient); bWriter.flush(); BeanReader reader = new BeanReader(); reader.registerBeanClass("ingredient", Ingredient.class); Ingredient ingReadFromFile = (Ingredient)reader.parse(new File("output.xml")); System.err.println(ingReadFromFile); } } If you run this example, you’ll notice that although bean Æ XML works perfectly (you get a file called output.xml), the reverse process doesn’t run: You get an error that states Could not create instance of type: Ingredient. The reason should be obvious: The Ingredient class, as it stands in listing 5.3, doesn’t contain an empty constructor. Betwixt requires that all classes you want created by reading from an XML file should contain an empty constructor that can be invoked, and therefore it fails when trying to use the Ingredient class from listing 5.3. To be able to completely run the code in listing 5.13, you should change the Ingredient class to contain an empty constructor. c To convert a JavaBean to XML, you need an instance of the BeanWriter class. This class contains four separate constructors, allowing you to use an OutputStream (or one with a different encoding than the platform default), a Writer, or a default empty constructor that defaults to using System.out for its output. In this case, we’ve used a FileWriter, which outputs the results of the conversion to a file called output.xml. c Writers for output d Dummy Ingredient class e Reader for input f Parse original output Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 27 d The Ingredient class is created here with dummy data. Note that contrary to what we mentioned earlier, JavaBean-to-XML conversion doesn’t require the presence of an empty constructor. Once we’re ready with the bean we want to write, we write it out using the write method to the underlying stream. The first argument is the top-level element name, and the second argument is the bean to write. Finally, although Betwixt is configured to flush automatically for OutputStreams, it’s a good idea to do so yourself in all cases. e To start the reverse process, we use the BeanReader class, which is really a wrapper around the Digester component (see module 4). Before the XML document can be read in, Betwixt needs to know the top- level element and the class it maps to. This is done using the registerBeanClass method. By registering the class, Betwixt creates an ObjectCreateRule behind the scenes for the internal Digester instance. f All that’s left is for the BeanReader class to read in the XML representation. This is again delegated internally to the Digester instance. The resulting object that is created is an exact representation of the original Ingredient object. With this example behind us, let’s look at how you can read information about a target bean from an external file. Using a .betwixt file A .betwixt file is used to provide information about a target class. If Betwixt finds a .betwixt file in the CLASSPATH that matches the target class name, it doesn’t use the default java.beans. Introspector and instead relies on the information about the bean in the XML file. Note that this file can be used in both bean Æ XML and XML Æ bean transformation. Any properties that you set in the code are overwritten if matching properties are found in this file. This implies that the .betwixt file has the final say in how Betwixt’s transformation for a target class is configured. In this section, you’ll see how to use this file. The .betwixt file should be described in XML. This means that there must either be a schema or a Document Type Definition (DTD) for the valid structure of this file. Listing 5.14 shows the relevant DTD, which you can find in the resources section of the Betwixt source code. Listing 5.14 DTD for .betwixt file 28 JAKARTA COMMONS ONLINE BOOKSHELF type CDATA #IMPLIED uri CDATA #IMPLIED value CDATA #IMPLIED property CDATA #IMPLIED updater CDATA #IMPLIED class CDATA #IMPLIED > The root element of the .betwixt file, even though it may not be obvious from the DTD, should be the info element; it has only one attribute, primitiveTypes. This attribute specifies whether primitive data types like String and int in the target class should be written out as elements or attributes. Potential bug: Even though the DTD specifies that the default value for the primitiveTypes attribute is attribute, in reality the default seems to be element. By the time this book goes to print, this bug may have been resolved. Let’s start by creating a .betwixt file for our Ingredient class. We’ll use the same code as in the previous section, and we’ll drop this ingredient.betwixt file in the CLASSPATH. This will ensure that it’s read by Betwixt and used for writing output.xml from listing 5.13 (and for reading and creating an Ingredient object). Listing 5.15 shows the first draft of this file. Listing 5.15 First draft of ingredient.betwixt The root element is info; it contains a single attribute called primitiveTypes. The root element in the output.xml file must be named ingredient. And, Betwixt must use java.beans.Introspector to do the rest of the work of looking at the Ingredient bean. Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 29 If you place this file in the CLASSPATH and rerun the code in listing 5.13, your output.xml file will look like this: The output is a single line with attributes for all the properties and the root element is ingredient. To stop the printing of an element/attribute, use the element hide in the .betwixt file and specify the property that you want to suppress. This is shown in the second draft, in listing 5.16. Listing 5.16 Stopping the output of an element in ingredient.betwixt using the hide element You can use the hide element to suppress the printing of undesired properties. Note that the list of hide elements should appear before the element element. The output of using this file now looks like this: This effectively suppresses the alternate property. The result would be similar if primitiveTypes was set to element instead of attribute: Both key and name would be printed as elements, and alternate would be suppressed. If you don’t want default matching for your properties and elements/attributes, you can specify your own. You can specify as many properties you want to match and leave the rest to the default introspector by using the addDefaults element. Until now, we’ve left the matching of all the elements to the default introspector. Let’s change this: listing 5.17 shows how. Listing 5.17 Overriding the default matching behavior As you can see, the elements in the output.xml file are now explicitly matched to their respective properties in the Ingredient class. You could change this behavior to match these properties to any element that you wanted; this works in both directions of a transformation. You can also specify the method to use for the setting of these properties. Thus, instead of the default setXXX method, you can use any method that you wish to call, by using the updater attribute. For example, suppose you have a method called Licensed to Tricia Fu 30 JAKARTA COMMONS ONLINE BOOKSHELF setDifferentName instead of setName, and you want to use this method to set the name property. You change this in listing 5.17 as shown here: Now, each time Betwixt encounters the name element in the XML file, it will map that to the name property of the Ingredient object using the setDifferentName method. You can take this behavior one step further by using the attribute class, which allows you to use a different class than the primitive class for the instantiation of your property. Thus, if you want your String properties to be instantiated using a modified version of the String class called ModifiedString, you can use the following: This is especially useful in cases where the type of a property is specified using an interface or an abstract class, and you want to use a specific implementation to create your object. Working with multiple values To show how Betwixt works with collections and maps, consider listing 5.18. It shows the code for converting a Meal object to XML and vice versa. Recall from listing 5.2 that the Meal class contains the ingredients as a HashMap. Listing 5.18 Converting a Meal object to XML and back package com.manning.commons.chapter05; import java.io.File; import java.io.FileWriter; import org.apache.commons.betwixt.io.BeanReader; import org.apache.commons.betwixt.io.BeanWriter; public class MultipleValues { public static void main(String args[]) throws Exception { FileWriter fWriter = new FileWriter("output.xml"); BeanWriter bWriter = new BeanWriter(fWriter); Meal meal = new Meal(1, 1, "Potato Bake"); Ingredient ingredient1 = new Ingredient("POT", "Potato", "None"); Ingredient ingredient2 = new Ingredient("CRM", "Cream", "None"); meal.addIngredient(ingredient1); meal.addIngredient(ingredient2); bWriter.enablePrettyPrint(); bWriter.write("meal", meal); bWriter.flush(); BeanReader reader = new BeanReader(); reader.registerBeanClass("meal", Meal.class); Create new meal object and add ingredients Make output XML more readable with indentation Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 31 Meal mealReadFromFile = (Meal)reader.parse(new File("output.xml")); System.err.println(mealReadFromFile); } } Most of this code doesn’t change from listing 5.13. As with the Ingredient class, a default empty constructor needs to be added to the Meal class to run listing 5.18.The output.xml file created by running this code is shown in listing 5.19. Listing 5.19 output.xml for multiple values represented as a HashMap POT POT Potato None CRM CRM Cream None Potato Bake 1 1 As expected, the meal root element contains an ingredients element that represents the map. Each value in the map is entered in the XML as a separate entry element. Each entry element contains two elements: the key element, representing the key; and the value element, representing the value associated with that key (in this case, an Ingredient object with its own properties marked as elements). The rest of the meal properties are listed at the end. If, instead of a Map, we were dealing with a collection-based object, the output would be similar. But instead of the entry element, we’d have the actual name of the object (); and, of course, there would be no key element. The reverse process relies on the presence of an addXXX method in the original object. Thus, the Meal object is faithfully re-created with the right ingredients, because Betwixt recognizes the presence of an addIngredient method in the Meal class. You can plug in your own method by specifying the update attribute in a .betwixt file, for the matching element. For example: Licensed to Tricia Fu 32 JAKARTA COMMONS ONLINE BOOKSHELF You now know how to read and write multiple values using Betwixt. In the next section, we’ll consider the custom conversion process, which lest you specify specially built patterns for String Æ Object and Object Æ String conversions. Custom conversions Betwixt uses the ConvertUtils component of Commons BeanUtils for converting Objects to Strings and vice versa. However, at times, this default behavior may not be what you want. For example, you may want the dates in your output to be printed in a particular format (the default format prints the date as EEE MMM dd HH:mm:ss zzz yyyy, using the UK as its locale). You can do this by using a custom implementation of the ObjectStringConverter class and installing it as the default on the BindingConfiguration class. Listing 5.20 shows an implementation of the ObjectStringConverter class for converting dates in a format that is different from the default implementation. Note that this class only extends the objectToString method, which enables bean Æ XML conversion, and not the stringToObject method, which you would have to override for XML Æ bean conversion. Listing 5.20 Converting dates in a special format package com.manning.commons.chapter05; import java.text.SimpleDateFormat; import org.apache.commons.betwixt.expression.Context; import org.apache.commons.betwixt.strategy.ObjectStringConverter; public class DateConvertor extends ObjectStringConverter { private static final SimpleDateFormat formatter = new SimpleDateFormat("dd-MMM-yyyy"); public String objectToString( Object object, Class type, String flavor, Context context) { if(object != null) { if(object instanceof java.util.Date) { return formatter.format( (java.util.Date) object ); } } return super.objectToString(object, type, flavor, context); } } To use this special converter, we need to install it on Betwixt using the BindingConfiguration class, as shown in listing 5.21. Desired format for date output Override method to specify custom conversions Convert Date objects in special format Let superclass handle remaining conversions Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 33 Listing 5.21 Installing and using the custom converter package com.manning.commons.chapter05; import java.io.File; import java.io.FileWriter; import org.apache.commons.betwixt.io.BeanReader; import org.apache.commons.betwixt.io.BeanWriter; import org.apache.commons.betwixt.BindingConfiguration; public class CustomConversion { public static void main(String args[]) throws Exception { FileWriter fWriter = new FileWriter("output.xml"); BeanWriter bWriter = new BeanWriter(fWriter); MealPlan plan = new MealPlan(); Meal meal = new Meal(1, 1, "Potato Bake"); Ingredient ingredient1 = new Ingredient("POT", "Potato", "None"); Ingredient ingredient2 = new Ingredient("CRM", "Cream", "None"); meal.addIngredient(ingredient1); meal.addIngredient(ingredient2); plan.addMeal(meal); bWriter.enablePrettyPrint(); BindingConfiguration bConfig = bWriter.getBindingConfiguration(); bConfig.setObjectStringConverter(new DateConvertor()); bWriter.setBindingConfiguration(bConfig); bWriter.write("mealplan", plan); bWriter.flush(); } } The output of running the code in listing 5.21 is as shown in listing 5.22. Notice that the dates are written in the special format dd-MMM-yyyy. If you run the code in listing 5.21 without installing the special converter, the dates will be printed in the default format EEE MMM dd HH:mm:ss zzz yyyy. Listing 5.22 output.xml, using a special converter on the MealPlan object 23-Apr-2004 POT Use whole MealPlan object from listing 5.1 Built-in BindingConfiguration with default values Set ObjectStringConverter to special DateConverter Update Betwixt instance with new configuration detail Licensed to Tricia Fu 34 JAKARTA COMMONS ONLINE BOOKSHELF POT Potato None CRM CRM Cream None Potato Bake 1 1 16-Apr-2004 While reading this XML file, we can’t use the default converter: It expects the date in the default format, but the date here is in the special format. To make a complete round trip, the stringToObject method also needs to be overwritten in listing 5.20. We’ll leave that as an exercise for you! 5.3 Summary In this module, we covered the basics of two components and illustrated their usage through a variety of examples. We explored the underlying technology in both cases: XPath in the case of JXPath, and data binding in the case of Betwixt. Each component’s structure was examined as well, to make sure you understand how it operates. JXPath is an extensible and flexible path-traversal tool. You can enhance it by adding custom functions, and it can access standard XPath functions as well. Betwixt is a utility tool for converting JavaBeans to XML and vice versa. It’s a flexible tool that uses Digester rules to create objects from XML files. It also converts JavaBeans to XML in a manner that can be customized to a high degree. In the next module, we’ll look at the Validator component, which provides a data-validation framework. Licensed to Tricia Fu MODULE 5: JXPATH AND BETWIXT: WORKING WITH XML 35 Index BeanInfo, 22 Betwixt .betwixt DTD, 27 BindingConfiguration, 24 caching pre-processed classes, 24 custom conversions, 32 data binding, 22 internal registry, 24 java.beans package classes, 22 matching cyclic elements, 25 normalization, 24 object to string conversions, 25 overriding bean type, 25 overriding default introspection, 23 overriding default matching behavior, 29 resolving runtime issues, 24 roundtrip example, 26 steps in bean processing, 23 structure, 22 suppressing element values, 29 understanding the Betwixt process, 23 using a .betwixt file, 27 using an external file for class information, 27 working with maps and collections, 30 writing objects, 24 XML-to-bean process, 25 BindingConfiguration, 24 classes getting information about. See Introspector containers, 14 data binding. See Betwixt Introspector, 22 java.beans package, 22 JXPath combining variables and pointers, 18 Compiler, 9 containers, 14 Context, 9 behind the scenes, 9 creating, 9 creating collection based factories, 16 creating objects, 15 creating objects and setting properties shorthand, 16 custom extension functions, 19 extension functions, 19 modifying simple properties, 15 object traversal problem scenario, 1 overview, 6 pointers, 18 accessing pointers, 18 creating pointers, 18 reading data from a file, 14 reading data in maps and collections, 12 reading data of a simple object, 10 reading data of map-based JavaBean, 13 reading lenient values, 11 reading multiple values, 13 reading nested objects data, 11 setting factory for object creation, 16 standard extension functions, 19 using custom extension functions, 20 using iterate to read multiple values, 14 using pre-compilation, 10 variables, 17 creating variables, 17 Variables resetting variable value, 18 marshalling. See data binding naming parts of an XML document. See XPath object traversal, 1 pointers, 18 SimpleBeanInfo, 22 unmarshalling. See data binding XML documents naming parts. See XPath XML Path language. See XPath XMLBeanInfo, 23 XMLIntrospector, 23 XPath, 7 built-in functions, 8 example, 7 root element, 8 selecting actual elements, 8 selecting attributes, 8 standardized syntax, 8 using wildcards, 8 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Validating data with Validator Vikram Goyal MODULE MANNING 6 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 6 Validating data with Validator 6.1 The data validation process....................................................................................................................................... 1 6.2 The Validator component ......................................................................................................................................... 3 6.3 Validator in action .................................................................................................................................................... 6 6.4 Summary................................................................................................................................................................. 25 Index ............................................................................................................................................................................. 26 In the world of programming, once in a while a nifty idea comes along—a valuable tool or a spark of imagination that makes you wonder how you ever programmed without it. Commons Validator is one such tool; once you start using it, it’s difficult to program without it. Validator provides a guideline and API for validating user data. It forces the developer to centralize and streamline data validation and lays down the best possible way to manage this validation. At the same time, it provides an API that brings order to data validation. It really is the tool that you can’t live without, if your application regularly accepts user data. In this module, we’ll start with a little background about data validation—why you should do it, how to do it, and what should be done. We’ll then move on to discuss the Validator component, giving an overview and discussing its API. This will prepare you for the Validator examples that follow. You’ll see how to use Validator in several situations, including web-based and normal applications. In the web applications, we’ll consider Struts and non-Struts applications, to cover Validator usage in almost all situations you’re likely to encounter. 6.1 The data validation process Computers are dumb. Yes, that’s right. Computers are dumb machines that do what you and I tell them to do. So why is it so difficult for us to make them work the way we want them to? Is the problem something we do, or something we don’t do? The answer isn’t easy to figure out. How many of us have spent hours of frustrating detective work trying to figure out what the computer is doing, only to realize that it was after all, a stupid mistake on our part that made it behave that way? Yes, it’s not computers that are dumb: It’s us. With that frightening realization out of the way, let’s see how we can minimize the time and effort required to figure out problems in our code. Since every application depends on the data it receives, the first salvo to fire is targeted toward making sure this data is validated to conform to set guidelines. This is why data validation processes are the most important facets of any application. Let’s start our discussion of the process with an understanding of why data validation should be performed. 6.1.2 Why bother? In simple terms, data should be validated because users are unpredictable. You can’t rely on users to consistently provide you with valid data that’s applicable for your application. Even the most fastidious users will tire and input invalid numbers in a salary field, for example. However, data should be validated for a number of reasons to cover a variety of issues that you’re likely to encounter: Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF ƒ To cater to unpredictable users. For example, a user may enter a String value in a Number-only field, due either to ignorance or a genuine mistake. ƒ To constantly protect your application from rogue and mischievous users. ƒ To ensure the integrity of data already in your system. There is nothing more frustrating than sifting through your application’s data store for invalid data that has the potential to render your data systems invalid. ƒ To ensure conformity and compliance with existing systems. ƒ To reduce application-processing time. In most systems, frameworks exist to separate the task of business processing and front-end interactions into separate modules. If data validation at the front end can sift through the user input and invalidate entries that don’t conform to set guidelines, it reduces the processing that business units have to do. This is by no means an exhaustive list of the reasons for data validation, but it covers the most prominent ones. In the next section, we’ll discuss how to perform data validation. 6.1.3 The how-to of data validation We all realize the need to perform data validation and agree that’s a required part of the application development process. But how do we perform validation? Following are some basic guidelines for this process: ƒ The first step in data validation should be performed at the data-entry point. There is no sense in holding data and performing validation at a later stage, because any subsequent processing can be avoided if the supplied data fails. The only exception to this rule arises when the validity of data depends on subsequent data collection or business process. ƒ You should take into account existing criteria for data-set validation. This means that validation can only be performed against rules governing a collection of previous data. These rules must be explicit and precise and should be made available to the validation routines before data is gathered from the user. ƒ The data validation rules must also be separated from the source code so you can modify the validation rules without affecting applications that are in production mode. Therefore, these rules must be supplied to the validation routines as an external process that’s only looked up at the precise moment the validation is to be performed. ƒ The data validation process should also make sure the user or system supplying the data can be made aware if any data validation checks and consequently can be allowed to resubmit with correct data or gracefully withdraw from continuing with the application. ƒ The data validation routines must be independent of the surrounding process. This means that the same validation routines should work identically in all applications, regardless of the way input is supplied to the routines. Let’s now discuss what to look for when you’re validating data. Licensed to Tricia Fu 3MODULE 6: VALIDATING DATA WITH VALIDATOR 3 6.1.4 What to validate A good validation routine validates all data that’s processed by your application. There is a trade-off between validation time and the strictness of validation rules; but in all fairness, it’s better to take more time to validate than it is to take extra time to fix problems due to flexibility of the validation rules. But what should a validation routine check for? Although a lot depends on the structure of an application, it’s easy to arrive at some general rules. A validation rule should do the following: ƒ Check for the presence or absence of a data value. If a data value is expected and isn’t given, this should be noted. If data value isn’t expected and is given, the routine should ignore the data. ƒ Require that a data value be within a given range. For example, if a Number field is restricted to a range of 0–10, a value of 11 should fail validation. The boundary conditions of the range should be clearly defined (whether the range boundaries are included or excluded from the range). ƒ Check the data values to be sure they conform to expected datatypes. For example, numeric datatypes should only contain numeric values, and alphanumeric datatypes can contain both numeric and alphabetical values. ƒ Require that the data value follow a particular pattern. For example, if you’re expecting a phone number for the value of your data, you may force the user to only input the number in the format XXX-XXXX- XXX. ƒ Require that the data value be at least, or at most, a specific length. This allows validations on the length of fields—for example, when validating passwords. Again, this list isn’t exhaustive, but it lists the basic validation checks that must be performed. In the next section, we’ll start the review of the Validator component with a look at its structure and API. 6.2 The Validator component The Validator component gained widespread usage and acceptance along with the Struts component (http://jakarta.apache.org/struts). Although Validator was created as a standalone project to validate user data in a neutral environment, it found most use as a plug-in in Struts-based web applications to validate data at both the client level and the server level. At a glance, the key features of the Validator component are as follows: ƒ Defines interfaces for validation routines and validation rules ƒ Provides internationalized error and system messages ƒ Provides support for both front end and back end validations ƒ Provides several ready-to-use validation routines ƒ Supports external and internal configuration ƒ Can be extended with custom routines We’ll explore the usage of these features when we look at the Validator component in action, later in this module. Using the Validator component is a fairly easy task: It involves three steps, each of which is centered on the class Validator. But before we look at these steps, let’s discuss the Validator API. Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF 6.2.1 The Validator API The centerpiece of the Validator API is the Validator class, which is in the main package org.apache.commons.validator. Actually, the main package is the centerpiece of the Validator component. There is only one other package, org.apache.commons.validator.util, which, as you can guess, contains a bunch of utility classes; the most important is ValidatorUtils. The Validator API is small and compact and serves its purpose well. The Validator class works with two other classes from the same package. The first, ValidatorResources, is used to define the rules associated with a particular set of input data. The individual fields in the input data are called fields, the set of input data is collectively called a form, and a collection of forms is called a form-set. The relationship between the elements is shown here: Fields Æ Forms Æ FormSet Thus, the ValidatorResources class is responsible for making the Validator class aware of the rules to be applied to a form-set. Note that this information is sensitive to the current user’s locale. In the Validator API, each element of the user data is represented using a separate class. Form represents user forms, Field represents individual field elements of a form, and FormSet represents the collection of forms. The other class that Validator works with is, in reality, not one class but a group of classes. These classes are a set of validation routines that implement the validation rules; they’re listed in table 6.1, along with the validations they perform. However, you don’t use these classes directly. Standalone applications need to build a wrapper around these rules to be able to use them. In this module, we’ll create a wrapper class that will encapsulate most of these rules. Table 6.1 Prebuilt validation rules Class Validation performed DateValidator Performs date validations based on a supplied locale and format EmailValidator Performs validations on email addresses to accept or reject a String as a valid email CreditCardValidator Does a Luhn algorithm (http://www.webopedia.com/TERM/L/Luhn_formula.html) to check for a valid credit card UrlValidator Validates a String to accept or reject it as a valid URL GenericValidator Performs validations for most generic types: whether a number is in a range, whether a data value is alphanumeric, whether a data value doesn’t exceed its allowed maximum value, and so on ISBNValidator Performs validation to accept or reject a correctly formed ISBN number, based on the ISBN algorithm defined at http://www.isbn.org/standards/home/isbn/international/html/usm4.htm Each rule can be modified to set the various options that are possible for it, which affects the way the rule behaves. You can also extend these rules using by custom validation rules that you can plug in quite easily. You’ll see how to do both with a variety of examples shortly. The results of validation are stored in the ValidatorResults class, which contains individual ValidatorResult class instances corresponding to each field in which the validations were performed. In the next section, you’ll see how to put together a validation process using the Validator API. 6.2.2 The validation process One final thing before we move to the examples: In this section, let’s lay down the basic validation process, with the help of a figure and a list of tasks. Figure 6.1 shows the basic validation process applicable to the Validator component. Licensed to Tricia Fu 5MODULE 6: VALIDATING DATA WITH VALIDATOR 5 Figure 6.1 The validation process using the Validator component Notice that as far as the Validator component is concerned, it only needs to validate a Java class that conforms to the JavaBean syntax. You can create any process around this to suit the needs of your application. As shown in the figure, these are the steps of the validation process: 1. Create or modify a wrapper class around validation rules. 2. Define the validation rules for a targeted data set. 3. Convert or accept user input data as a JavaBean. 4. Create an instance of Validator. 5. Pass in the rules and user data to this instance. 6. Call the validate method of the Validator instance, and accept results in the ValidatorResults class instance. 7. Process the results. Depending on your application’s needs, some of these steps may not be required; and in some cases, the steps may need to be shuffled. For example, once you create the wrapper class in step 1, you probably won’t need to create it again and will modify it only if new validation rules are added. Moreover, some of these tasks may be done in parallel. However, most validation work follows the seven steps outlined in order, and we’ll follow them in the examples in the next section. Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF 6.3 Validator in action The Validator API is dependent on several other Commons components, listed in table 6.2. These components, as well as commons-validator.jar, must be in your CLASSPATH before you run the examples in this section. Table 6.2 Validator dependencies Component Covered in… Web site Commons-Logging Module 13 http://jakarta.apache.org/commons/logging Commons-Beanutils Module 7 http://jakarta.apache.org/commons/beanutils Commons-Collections Module 7 http://jakarta.apache.org/commons/collections Commons-Digester Module 4 http://jakarta.apache.org/commons/digester ORO (text-processing Java classes) Not covered http://jakarta.apache.org/oro/ Let’s begin this section with the first draft of the wrapper class. Recall that a wrapper class is a class built around the validation rules that exercises these rules for your application. 6.3.1 Building a wrapper class In this section, we’ll create a simple wrapper class. This wrapper class contains a single method that enforces a required condition on user data. We’ll build on this class as we go along. Listing 6.1 shows the first draft of the wrapper class. Listing 6.1 First draft of the wrapper class package com.manning.commons.chapter06; import org.apache.commons.validator.Field; import org.apache.commons.validator.GenericValidator; import org.apache.commons.validator.util.ValidatorUtils; public class ValidatorWrapper { public static boolean doRequired(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); return !GenericValidator.isBlankOrNull(value); } } The first method that we have added in the wrapper class deals with checking whether a field in the user form has an empty value or isn’t present. This is equivalent to confirming whether a required field is missing from the user data. The method is given the signature of doRequired(Object bean, Field field) and returns a boolean true or false. The bean parameter is the user data represented as a JavaBean, and the field parameter is the Validator component’s representation of the field in the user data that is to be validated. ValidatorUtils getValueAsString is a utility method that uses reflection to get the value of the field as String. This value is extracted from the user data. This step is important because the Validator API only accepts String values for its built-in validators. Once the value has been validated as a String, we use the GenericValidator class. This class contains a method called isBlankOrNull, which examines the String passed in and returns true if the String value is null or is of zero length. Because we want the doRequired method to force the presence of Licensed to Tricia Fu 7MODULE 6: VALIDATING DATA WITH VALIDATOR 7 a non-null or non-empty String, we negate the result from GenericValidator and return it. This completes step 1 of the validation process. We’ll build on it in the next section. 6.3.2 Validating a simple JavaBean As far as Validator is concerned, all user data is in essence based on JavaBean-style get and set properties. This way, it’s easy to retrieve the value of a property or field in the user data and validate it according to Validator’s rules. So, in this section, we’ll demonstrate how to validate a simple JavaBean. This will help you understand all the validation steps, the first of which we covered in the previous section. In listing 6.2, we create a simple JavaBean that contains two properties, firstName and lastName, along with their respective get and set methods. Listing 6.2 A simple JavaBean package com.manning.commons.chapter06; public class SimpleJavaBeanV1 { private String firstName; private String lastName; public String getFirstName() { return this.firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public String getLastName() { return this.lastName; } public void setLastName(String lastName) { this.lastName = lastName; } } Now that we have a basic JavaBean to validate and act as our user data, we need to create the targeted set of validation rules that will be applied to this JavaBean. This is step 2 of the validation process, and it’s defined in an XML file. Creating targeted validation rules Creating targeted validation rules means that Validator needs to be told which method of which class to call to find out whether a particular field passes a particular validation. To make this step of the process independent enough that it can be performed during runtime, Validator allows you to define these rules in an XML file and not in source code (although it can be done in source code as well). Listing 6.3 continues our example and creates a set of rules using the JavaBean defined in listing 6.2 and the wrapper class created in listing 6.1. Listing 6.3 Creating a targeted set of validation rules in the file validatorsimple.xml 8 JAKARTA COMMONS ONLINE BOOKSHELF classname="com.manning.commons.chapter06.ValidatorWrapper" method="doRequired" methodParams="java.lang.Object, org.apache.commons.validator.Field" msg=""/>
Several things about this XML file must be noted. First, Validator provides a DTD that defines a list of allowed elements for this file. Second, Validator uses the Digester component (module 4) from Commons to parse this file. Why is that important to know? Because if you’re familiar with Digester, you know that parsing XML files with Digester requires a set of rules for Digester defined in its own XML file. Fortunately, you don’t have to worry about creating this file, because Validator provides such a file in its own jar file. As you can see, the root element of a validation rules file is the element. Only two elements are allowed within this root element— and —although they can appear multiple times. These elements interplay against each other to form the set of targeted rules for a JavaBean. Only one targeted Validator is defined in listing 6.3: required. Its class is defined as com.manning.commons.chapter06.ValidatorWrapper, and the method to call is doRequired, with the given parameters. This is the wrapper class, and its sole method was created in listing 6.1. By putting these values in this XML file, we’re telling Validator that in the form-set element(s) to follow, if a field needs to be validated to ensure that it’s present, it can call this required Validator. Targeting isn’t complete until the field elements to be targeted are told which rules to follow. Thus both the field elements, identified by the properties firstName and lastName respectively, are told to follow the required rule, defined earlier. This is done by putting the required value in the depends attribute. This attribute can contain several rules, separated by commas; the order in which the rules are listed is the order in which they’re exercised. This completes step 2 of the validation process. Now you need to know how to perform the validation. Validating the JavaBean instance We have the wrapper class that contains the methods to be called for validation. We have the JavaBean structure, an instance of which needs to be validated. We have the targeted set of rules that creates a mapping between the JavaBean properties/fields and the wrapper class methods. All we need to do now is create a Validator instance, accept user data, validate it, and process the results. These are steps 3–7 of the validation process, and they’re combined in listing 6.4. Define collection of forms Define collection of fields Define individual fields and their validations Licensed to Tricia Fu 9MODULE 6: VALIDATING DATA WITH VALIDATOR 9 Listing 6.4 Performing validation for a simple JavaBean instance package com.manning.commons.chapter06; import org.apache.commons.validator.Validator; import org.apache.commons.validator.ValidatorResult; import org.apache.commons.validator.ValidatorResults; import org.apache.commons.validator.ValidatorResources; import java.io.FileInputStream; public class ValidatorSimpleV1 { public static void main(String args[]) throws Exception { ValidatorResources resources = new ValidatorResources( new FileInputStream("validatorsimple.xml")); Object userData = simulateUserData(); Validator validator = new Validator(resources, "simpleform"); validator.setParameter(Validator.BEAN_PARAM, userData); ValidatorResults results = validator.validate(); ValidatorResult result = results.getValidatorResult("firstName"); ValidatorResult result1 = results.getValidatorResult("lastName"); System.err.println("First Name: " + result.isValid("required")); System.err.println("Last Name: " + result1.isValid("required")); } private static Object simulateUserData() { SimpleJavaBeanV1 simpleBean = new SimpleJavaBeanV1(); simpleBean.setLastName("Goyal"); return simpleBean; } } Before we analyze the listing, it might be a good idea if you run it and see the result on the command line. In creating the simulated user data we set only the lastName field; but we set both the lastName and firstName fields to required in the XML file, so validation of the user data will fail overall. c The ValidatorResources class loads the targeted rules that were created in the XML file from listing 6.3. This class loads the rules and uses the Digester component to parse them. d The user data is simulated to accept an instance of the SimpleJavaBeanV1 class. Only one of the required parameters of this class is set: lastName. c Load targeted rules d Accept user data e Create and initialize new Validator instance f Perform validation g Return results for each field h Print results for each field’s validation Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF e The Validator instance is now created. It accepts two arguments. The first, as expected, is the ValidatorResources instance. This instance initializes Validator to process the targeted rules defined in the XML file. The second argument, in this case simpleform, is the name of the
element in the XML file to be processed. Recall that the element can occur multiple times within the XML file. This means that the same XML file can be used to define the processing of several forms. By specifying which element to process by name, the Validator instance is able to pick the correct one in this situation. Note: The form name must be defined before validation can take place. What happens if you have multiple elements in the same XML file, and after processing one element, you want to process another one? You use the method setFormName(String formName) to reset the form to be processed and call the validate method again. In such cases, it may be better to create the Validator instance with the form-neutral constructor Validator validator = new Validator(resources) and set individual forms using the setForm method before doing any validation. The Validator instance is now told about the user data that is to be validated. This user data was created (simulated) in d. Note the way in which Validator is told about the user data: It’s specially set using the setParameter method, and the constant Validator.BEAN_PARAM identifies the JavaBean on which validations are to be performed. Needless to say, if you’re doing multiple validations, and each piece of user data is represented by a different JavaBean instance, you’ll need to reset this parameter. fg Finally, we can perform the actual validation. The validate method returns the result of the validation in the ValidatorResults class. This class contains a HashMap of individual ValidatorResult objects, corresponding to each field that was processed where the field name is used as the key for each result. Thus, to get the validation result for the lastName property, we use getValidationResult("lastName"). h Each ValidatorResult object can now be queried to find out whether it passed validation for each validator registered for it. Recall that this was done in the depends attribute of the XML file. In this case, both firstName and lastName fields had only one validator registered in this attribute: the required validator. Therefore, when we call the isValid method, it needs to be qualified for the validator for which the result is required. The isValid method returns a simple true or false, based on the validity of the user data. In this case, the lastName field was valid, because it had been set and it was required. The firstName was also required, but it wasn’t set; therefore isValid("required") for it returns false. Figure 6.2 shows the relationship between ValidatorResults and ValidatorResult using this example as the basis. Note that it assumes that there may be more than one Validator registered per field (and not just the required Validator). We have completed all the steps required to validate a simple JavaBean using the Validator component. Although this may seem like a lot of work for a small gain, it seems worthwhile when you start dealing with complex objects and applications, where the task of validation can spiral out of control. In the next section, we’ll expand on our wrapper class and use it to good effect in a variety of application scenarios. Licensed to Tricia Fu 11MODULE 6: VALIDATING DATA WITH VALIDATOR 11 Figure 6.2 Relationship between ValidatorResults and ValidatorResult 6.3.3 Using Validator in a web application As you may know, the Validator component grew out of the Struts web application framework to validate user data coming via form submissions over HTTP. Although it was designed out of a need for web applications, the Validator component is useful across any situation where user data needs to be validated (as you saw in the previous section, where we validated a generic JavaBean). The Validator component isn’t tied to any framework, and wrapper classes easily let you build validation frameworks around it. Let’s now explore how to perform validation for web applications, first in a Model 1 web application, and then, in a Model 2 (Struts) web application. But before we do that, we need to expand on our wrapper class. Revisiting the wrapper class The wrapper class that we built in section 6.3.1 is useful, but it contains only one method that validates for the presence or absence of a field (the required validator). Let’s add some more methods to this class. Following are the validators we may want to add: ƒ Numeric—Validates a value to make sure it contains numbers only (positive integers). ƒ RangeCheck—Validates that a user data field is within a specified number range. The range boundaries must be supplied. ƒ Date—Validates a user data field to be sure it represents a date. The list of allowed patterns must be supplied. ƒ Maximum, Minimum—Validate a numeric value to make sure it isn’t more than or less than a supplied value, respectively. ƒ RegExp—Validates a value to make sure it conforms to a supplied regular expression. Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF ƒ Email, URL—Validates a value to ensure it represents a valid email/URL. ƒ CreditCard—Validates a value to ensure it represents a valid credit card number. ƒ ISBN—Validates a value to ensure it represents an ISBN book number. This list isn’t exhaustive, but it represents some of the most common validators you’re likely to require, especially for web applications. Listing 6.5 implements the methods required for these validators. Note that this listing extends the previous wrapper class code from listing 6.1, and therefore the doRequired method remains in it. Listing 6.5 Extending the wrapper class by adding more Validator methods package com.manning.commons.chapter06; import org.apache.commons.validator.Field; import org.apache.commons.validator.ISBNValidator; import org.apache.commons.validator.GenericValidator; import org.apache.commons.validator.GenericTypeValidator; import org.apache.commons.validator.util.ValidatorUtils; public class ValidatorWrapper { public static boolean doRequired(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); return !GenericValidator.isBlankOrNull(value); } public static boolean doNumeric(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); Integer valInt = GenericTypeValidator.formatInt(value); if(valInt == null) return false; return valInt.intValue() > 0; } public static boolean doDate(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); String datePattern = field.getVarValue("datePattern"); return (GenericTypeValidator.formatDate( value, datePattern, false) == null) ? false : true; } public static boolean doRange(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); String min = field.getVarValue("min"); String max = field.getVarValue("max"); c Check whether value is positive numeric e Check whether value is a date f Retrieve variable associated with Field (datePattern here) g Check whether value is within given range h Get user-defined minimum and maximum values for range d GenericTypeValidator converts values to respective types Licensed to Tricia Fu 13MODULE 6: VALIDATING DATA WITH VALIDATOR 13 try { int valueInt = Integer.parseInt(value); int minInt = Integer.parseInt(min); int maxInt = Integer.parseInt(max); return GenericValidator.isInRange(valueInt, minInt, maxInt); } catch (Exception e) { return false; } } public static boolean doEmail(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); return GenericValidator.isEmail(value); } public static boolean doURL(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); return GenericValidator.isUrl(value); } public static boolean doRegExp(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); String expression = field.getVarValue("expression"); return GenericValidator.matchRegexp(value, expression); } public static boolean doCreditCard(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); return GenericValidator.isCreditCard(value); } public static boolean doISBN(Object bean, Field field) { String value = ValidatorUtils.getValueAsString(bean, field.getProperty()); return new ISBNValidator().isValid(value); } } The wrapper class is now more useful, containing implementations for all the validators listed earlier. As you can see, the methods are implemented by extending the existing implementations in the Validator component. Some of the validators require configuration options set before they can be used. These options are provided in the XML file by using the element for each element. You’ll see how to do this in the next section, but before that, note how these options are accessed in the wrapper class. The method getVarValue(String varName) is used in the doDate, doRange, and doRegExp methods. For example, the doRange method requires that the options minimum and maximum be set for it as the i Return false if parsing of integers fails Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF range boundaries before it can decide whether a supplied value falls within the range. By using getVarValue("min") and getVarValue("max"), it’s easy to get these options from the XML file. Let’s now see how to use this wrapper class in a Model 1 web application. Using Validator in a Model 1 web application A Model 1 web application uses a decentralized mechanism for accepting user requests, processing these requests, and dispatching the response back to the user. All this is typically done in a JSP page with the help of JavaBeans. Note: For more information about Model 1 and Model 2 web applications as well as the general structure of web applications, please refer to the Manning book Struts in Action (http://www.manning.com/husted), or look online at http://www.javaworld.com/javaworld/jw-12- 1999/jw-12-ssj-jspmvc.html. In this section, we’ll use the wrapper class built in the previous section to create a validating web application. We need several components to do this: ƒ ValidatorWrapper.java—Contains implementations of various validators ƒ validatorweb.xml—XML file containing targeted validation rules for the web application ƒ validatorform.jsp—JSP page containing the form that needs to be validated ƒ checkform.jsp—JSP page that accepts input from validatorform.jsp and uses validator to validate the form data ƒ WebJavaBeanV1.java—JavaBean that encapsulates the user data of the validatorform.jsp form We created ValidatorWrapper.java in the previous section. Figure 6.3 shows how validatorform.jsp looks in an Internet Browser. It’s a simple JSP page; we won’t go into the construction details, but you can download this page and the rest of the code from the book’s web site. As you can see, the form contains fields that require all the validations we created in the ValidatorWrapper class. Moreover, the last field, Multi Field, requires two validations: it’s a required field and it should also be an email. Let’s now create validatorweb.xml, the XML file that contains the list of targeted validation rules. Listing 6.6 shows this file. The XML file isn’t too dissimilar from the previous XML file we created for the JavaBean validation in listing 6.3. It contains many more validators—in fact, it contains all the validators that are now implemented by the wrapper class. Next, it contains the field definitions for each field from the validatorform.jsp and specifies the validation to be performed on each field. Multiple validations are separated by commas. Licensed to Tricia Fu 15MODULE 6: VALIDATING DATA WITH VALIDATOR 15 Figure 6.3 validatorform.jsp in a browser Listing 6.6 validatorweb.xml, which contains targeted validation rules for the web application Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF Licensed to Tricia Fu 17MODULE 6: VALIDATING DATA WITH VALIDATOR 17 datePattern MM/DD/yyyy min 20 max 40 expression ^.*\b(master|commander)\b.*$ It might be obvious from looking at this XML file that the validator definitions contained in the element aren’t likely to change between applications. After all, if you require the range check validator in one application or form, you may require it in others as well, with the same validation being applied. It therefore makes sense to separate the element into its own XML file and the element into its own individual and application/form-specific file and merge them before passing them the ValidatorResources class. This way, validators are uniform across applications, and you don’t need to specify them for each application. (This is what Struts does: It separates these files into validator-rules.xml and validator.xml respectively, as you’ll see in an example in the next section.) We’re now ready to create the JavaBean that will encapsulate the form data from validatorform.jsp and that will be passed to the Validator component. This is shown in listing 6.7. Listing 6.7 WebJavaBeanV1: JavaBean encapsulating form data package com.manning.commons.chapter06; public class WebJavaBeanV1 { private String required; private String numeric; private String date; private String range; private String email; Supply datePattern variable value required for date Validator Supply min and max variable values required for range Validator multi field requires multiple validations Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF private String url; private String regExp; private String cc; private String isbn; private String multi; public String getRequired() { return this.required; } public void setRequired(String required) { this.required = required; } public String getNumeric() { return this.numeric; } public void setNumeric(String numeric) { this.numeric = numeric; } public String getDate() { return this.date; } public void setDate(String date) { this.date = date; } public String getRange() { return this.range; } public void setRange(String range) { this.range = range; } public String getEmail() { return this.email; } public void setEmail(String email) { this.email = email; } public String getUrl() { return this.url; } public void setUrl(String url) { this.url = url; } public String getRegExp() { return this.regExp; } public void setRegExp(String regExp) { this.regExp = regExp; } public String getCc() { return this.cc; } public void setCc(String cc) { this.cc = cc; } public String getIsbn() { return this.isbn; } public void setIsbn(String isbn) { this.isbn = isbn; } public String getMulti() { return this.multi; } public void setMulti(String multi) { this.multi = multi; } } Notice that the property names in this JavaBean must correspond exactly to the field names defined in the XML file. If the name of the property is cc, its corresponding set and get methods must be setCc and getCc (with the first character of the property capitalized). This is in accordance with the JavaBean specification, and if this requirement isn’t followed, Validator will have problems locating the correct get methods for your fields. Finally, listing 6.8 shows checkform.jsp. This JSP has multiple functions: It accepts user data, converts it into WebJavaBeanV1, creates a Validator instance, validates the JavaBean, and sends the results back to the originating form. Listing 6.8 checkform.jsp: validates user data submitted over the Web <%@ page import= "java.util.*, java.io.FileInputStream, org.apache.commons.validator.*, Get and set method names must correspond to field names from XML file Wrap imports to fit; must be on one line in actual code! Licensed to Tricia Fu 19MODULE 6: VALIDATING DATA WITH VALIDATOR 19 org.apache.commons.validator.util.*, com.manning.commons.chapter06.WebJavaBeanV1" %> <% String required = request.getParameter("requiredField"); String numeric = request.getParameter("numericField"); String date = request.getParameter("dateField"); String range = request.getParameter("rangeField"); String email = request.getParameter("emailField"); String url = request.getParameter("urlField"); String regExp = request.getParameter("regExpField"); String cc = request.getParameter("ccField"); String isbn = request.getParameter("isbnField"); String multi = request.getParameter("multiField"); WebJavaBeanV1 bean = new WebJavaBeanV1(); bean.setRequired(required); bean.setNumeric(numeric); bean.setDate(date); bean.setRange(range); bean.setEmail(email); bean.setUrl(url); bean.setRegExp(regExp); bean.setCc(cc); bean.setIsbn(isbn); bean.setMulti(multi); ValidatorResources resources = new ValidatorResources ( getClass().getResourceAsStream("/validatorweb.xml")); Validator validator = new Validator(resources, "webform"); validator.setParameter(Validator.BEAN_PARAM, bean); ValidatorResults results = validator.validate(); List fieldList = resources.getForm(Locale.getDefault(), "webform").getFields(); Iterator itr = fieldList.iterator(); while(itr.hasNext()) { Field field = (Field)itr.next(); ValidatorResult result = results.getValidatorResult(field.getKey()); String depends = field.getDepends(); StringTokenizer dependsTokens = new StringTokenizer(depends, ","); int countDepends = dependsTokens.countTokens(); for(int i = 0; i < countDepends ; i++) { String token = dependsTokens.nextToken().trim(); request.setAttribute( field.getKey(), result.isValid(token) ? "Yes" : "No"); } } c Accept user data from submitted form d Create JavaBean and sets its properties e Create ValidatorResources loading validatorweb.xml f Create new validator for webform g Set JavaBean to validate h Perform validation i Get list of all fields in webform j Iterate over list of fields k Return ValidatorResult and validators for each field l Set result of each validation Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF request.setAttribute("bean", bean); request.getRequestDispatcher("validatorform.jsp").forward( request, response); %> c Checkform.jsp begins by collecting the user data from the incoming request here. d A new JavaBean is created, and the user data is set on the properties of this bean. efgh An instance of ValidatorResources is created, using the validatorweb.xml that contains the targeted validation rules. These steps don’t differ much from the similar steps in listing 6.4, where we validated a simple JavaBean. i The ValidatorResources object is queried for a list of form fields found in the validatorweb.xml file. jkThis list of fields is iterated over, and the result for each field’s validation attempt is queried. The getDepends() method returns a comma-separated list of dependencies for each field. These dependencies are nothing but the validators that were registered for each field in the XML file. Except for the multi field, all other fields had only one validator registered. l The result for each field’s individual dependency is set in the request so it can be shown on the sending page. Once the processing of the results is done, the original data bean is reset in the request scope, and the user is thrown back to the originating page so the result can be shown. Figure 6.4 shows the result for one round trip, using the data from figure 6.3 as an example. Figure 6.4 Result of Model 1 web application validation Licensed to Tricia Fu 21MODULE 6: VALIDATING DATA WITH VALIDATOR 21 You now know how to validate form data using the Validator component in a Model 1 application. Let’s now explore using Validator in a Struts-based web application, which represents a Model 2 framework. Using Validator in a Model 2 web application (Struts-based) Because of the popularity of the Struts framework, you may have heard of the Validator component because of its association with Struts rather than using it on its own. This has in turn led to the use of Validator the same way Struts that uses it. As you’ve seen, Validator is a standalone component, and it can be used on its own to validate data in any environment. It’s worthwhile pointing out that the Validator component in Struts is based on the Commons Validator component and relies on it heavily to provide its validations. Before we show an example of using the Struts Validator, look at table 6.3, which shows which Validator component maps to which component in the Struts vocabulary. (For a detailed description of the Struts components, please refer to Struts in Action.) Table 6.3 Comparing Struts and Validator components Validator component Struts component JavaBean ActionForm Wrapper class FieldChecks.java Validating class ValidatorActionForm, DynaValidatorForm, and DynaValidatorActionForm XML files validator.xml and validator-rules.xml Using the Struts Validator component isn’t too different from using it in the Model 1 web application. You still have to create a targeted set of rules; but unlike in the Model 1 application, you don’t have to create the basic validation rule information, because Struts comes prebuilt with that information. If you create a custom validator, you still have to make an entry in this file for this validator. Listing 6.9 shows these rules for a form, which is similar to the form created in the previous section but with fewer fields. Listing 6.9 validator-rules.xml for a Struts-based web application
datePattern Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF MM/DD/yyyy min 20 max 40
Because Struts is a web-based application that relies heavily on user interaction, this XML file differs slightly from the previous XML file containing validation rules. Besides the obvious difference that this file only contains the formset definitions, the other differences arise from the fact that there is a heavy use of elements that let the user know about errors in the input, if there are any. Note that nothing stops you from doing the same for a normal, non-Struts web application, but Struts makes it so easy to integrate messaging and internationalization that it’s easier to showcase this functionality here. You’ve come across a msg attribute before. In the XML files listed before this section, all elements contain a msg attribute, albeit empty. The msg attribute there could have been used to list the message to be shown to the user in case validation failed for that particular validator. However, if a field tag is dependent on the same validator and contains a element, its message value will override the value of the msg attribute of the validator, if validation fails. The message to be shown to the user is derived from a resource file. Notice that the msg uses the attribute key to point to the message string to be picked from this resource file. This helps in localizing messages that are shown to the users. Listing 6.10 shows part of an example resource file. Listing 6.10 An example resource bundle file validator.required=The field {0} is required validator.date=Date is missing or not correct validator.email=Email is missing or not correct validator.range={0} is not in the range {1} - {2} validator.numeric=The number field does not contain a valid number As you can see, the keys referred to in listing 6.9 point to message strings in this file. You can further parameterize these messages by providing the value of parameters in the element in listing 6.9. Struts Licensed to Tricia Fu 23MODULE 6: VALIDATING DATA WITH VALIDATOR 23 replaces the values of the {i} word with the value of the arg elements, in the order in which they’re listed. Thus elements value corresponds to the {0} parameter, corresponds to {1}, and so on. To create a JavaBean and associate a JSP page with a form, we need to edit the struts-config file, which is found in the WEB-INF directory of a Struts application. Listing 6.11 shows an example configuration: A DynaValidatorForm is used to define a JavaBean, and the form is then associated with an action mapping. Listing 6.11 Configuring the Struts application The Struts configuration file is used to define, among other things, the mappings between actions (form submissions) and the classes that handle these actions. In between, it also allows you to define the JavaBeans Define dynamic form Define form properties Create action mapping Create action mapping Use dynamic form defined earlier Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF that will be used to capture the data from the form, whether it needs to be validated or not, and the scope (lifetime) of the form. In this configuration, we have defined one action (checkForm), mapped it to the com.manning.commons.chapter06.CheckFormAction action class, told it to use the webform dynamic form defined earlier, set the scope of the form to request, and set the JSP (strutsvalidatorform.jsp) from which the action will be called. You’ll notice that the form that’s declared is an instance of org.apache.commons.struts. validator.DynaValidatorForm. This is an especially useful form, because it lets you do two things that aren’t possible with normal JavaBeans. First, as the name suggests, this form is self-validating. This means that before this form is passed to the action class as a representation of the user data, it uses the Validator component to validate the data it’s been given. Second, it’s a dynamic form. We didn’t need to create a Java file to represent this class as a JavaBean, but defined its structure in the Struts configuration file. This reduces the lines of code that you have to write. Let’s see the JSP file strutsvalidatorform.jsp used to test this configuration. Figure 6.5 shows the result of running this file with sample data, and the Struts response. Figure 6.5 Result of submitting the strutsvalidatorform.jsp form Licensed to Tricia Fu 25MODULE 6: VALIDATING DATA WITH VALIDATOR 25 The strutsvalidatorform.jsp file isn’t very different from the file used in figure 6.3 (other than the fact that it has fewer fields). However, note that we used the Struts tag to print a list of validation errors instead of writing these errors manually. This brings us to the end of this section on using the Validator component in a Model 2 situation. As you may have realized by now, using the component in Model 1 and Model 2 web applications is similar, because the Validator component is designed to be independent of the surrounding framework. 6.4 Summary This module covered one of the most important tools in the struggle for better applications: the Validator component. This component nips user data errors in the bud before they can become a menace to your applications. The module began with an explanation of the need for validation and gave some brief insight into its mechanics. We introduced the Validator component and explained its structure and API. We then looked at examples of using this component. We started by using the Validator component to validate simple data in the form of a JavaBean and made clear that all data, if it can be transformed to a JavaBean, can be validated using this component. Then we looked at web applications, both Model 1 and Model 2, and used this component to process and validate data in them. Along the way, we introduced the features of this component. In the next module, we’ll start our coverage of the Commons components that enhance the usage of core Java libraries. This discussion is spread over two modules: Module 7 covers Collections, and module 8 covers BeanUtils and Lang. Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF Index CreditCardValidator, 4 data validation process, 1 how to perform validation, 2 reasons for data validation, 1 what to validate, 3 DateValidator, 4 DynaValidatorForm, 21, 23, 24 EmailValidator, 4 Field class, 4 Form class, 4 FormSet class, 4 GenericValidator, 4, 6, 12 ISBNValidator, 4 org.apache.commons.validator, 4 org.apache.commons.validator.util, 4 UrlValidator, 4 Validator API, 4 comparing Struts and Validator components, 21 creating a wrapper class, 6 creating targeted validation rules, 7 creating Validator instance, 8 editing struts-config file, 23 extending the wrapper class, 12 key features, 3 list of dependencies, 6 performing validation, 8 prebuilt rules, 4 processing results, 10 steps to validation, 5 understanding the validation process, 4 using in a web application, 11 using resource bundle file for Validator messages, 22 using Validator in a Model 1 web application, 14 using Validator in a Model 2 web application, 21 using Validator in Struts, 21 validating a simple bean, 7 Validator XML file, 8 XML file root element, 8 Validator class, 4 Validator XML file, 8 ValidatorResources, 4, 9, 10, 17, 20 ValidatorResult, 4 ValidatorUtils, 4 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Enhancing Java core libraries with Collections Vikram Goyal MODULE MANNING 7 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 7 Enhancing Java core libraries with Collections 7.1 A taste of things to come .......................................................................................................................................... 1 7.2 Collections and more ................................................................................................................................................ 3 7.3 Summary................................................................................................................................................................. 31 Index ............................................................................................................................................................................. 32 The Commons components we’ll discuss in this module and the next have one thing in common: They cover the deficiencies of the core Java classes by building wrappers around them or extending them to provide sets of useful libraries. These components are BeanUtils, which provides wrappers around the Java Reflection and Introspection API; Collections, which enhances the Java Collection API; and Lang, which provides helper utilities for manipulation of the core Lang package. In this module, we’ll study the Collections component; we’ll leave the discussion of the other two components to the next module. All three of these components complement the Java core API. These are very hands-on modules and, in that sense, are different from the rest of this book, because we don’t need to cover any background material before showcasing the examples. We’ll start with a simple example that uses part of each component to break the ice, and then we’ll move to the Collections component. 7.1 A taste of things to come As we said before, this module and the next differ from the other modules in this book because they deal with components that aren’t application-oriented. This means the components of these two modules don’t lend themselves to complete application development like the Modeler (module 11) or JXPath (module 5) Commons components. The Collections component classes and API can be used as utilities and helper routines to build applications and extend the core Java API in everyday functions. We’ll start with a small example that brings all of these components together, as shown in listing 7.1. Listing 7.1 Using the Collections, BeanUtils, and Lang components together package com.manning.commons.chapter07; import java.util.Map; import java.util.HashMap; import java.util.Iterator; import java.lang.reflect.Method; import org.apache.commons.collections.Bag; import org.apache.commons.collections.bag.HashBag; import org.apache.commons.beanutils.BeanUtils; import org.apache.commons.beanutils.PropertyUtils; Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF import org.apache.commons.lang.StringUtils; public class TasteOfThingsV1 { private static Map testMap; private static TestBean testBean; public static void main(String args[]) throws Exception { prepareData(); HashBag myBag = new HashBag(testMap.values()); System.err.println("How many Boxes? " + myBag.getCount("Boxes")); myBag.add("Boxes", 5); System.err.println("How many Boxes now? " + myBag.getCount("Boxes")); Method method = testBean.getClass().getDeclaredMethod("getTestMap", new Class[0]); HashMap reflectionMap = (HashMap)method.invoke(testBean, new Object[0]); System.err.println("The value of the 'squ' key using reflection: " + reflectionMap.get("squ")); String squ = BeanUtils.getMappedProperty(testBean, "testMap", "squ"); squ = StringUtils.capitalize(squ); PropertyUtils.setMappedProperty(testBean, "testMap", "squ", squ); squ = BeanUtils.getMappedProperty(testBean, "testMap", "squ"); System.err.println("The value of the 'squ' key is: " + BeanUtils.getMappedProperty(testBean, "testMap", "squ")); String box = (String)testMap.get("box"); String caps = Character.toTitleCase(box.charAt(0)) + box.substring(1, box.length()); System.err.println("Capitalizing boxes by Java: " + caps); } private static void prepareData() { testMap = new HashMap(); testMap.put("box", "boxes"); testMap.put("squ", "squares"); testMap.put("rect", "rectangles"); testMap.put("cir", "circles"); testBean = new TestBean(); testBean.setTestMap(testMap); } } The code listing not only mixes the three components together, but also shows how equivalent actions would be done in native Java code if these components weren’t available. The code creates a TestBean JavaBean Create TestBean with a TestMap as a property Create Bag that counts its contents c Bag working d Get map value with Java reflection e Combine BeanUtils and Lang to change map value f Show capitalization with Java Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 3 class that contains a single property, which is of the type HashMap. This HashMap is populated with random data in the prepareData method. After the preparation, each component is used to do some random work with this data to showcase a simple case of working with them. This TestBean is shown here: package com.manning.commons.chapter07; import java.util.HashMap; import java.util.Map; public class TestBean { private Map testMap = new HashMap(); public void setTestMap(Map testMap) { this.testMap = testMap; } public Map getTestMap() { return testMap; } } The first component that is used is Collections. This component’s Bag interface keeps a count of the number of items in a HashMap by populating a HashBag instance with the TestMap values. Note that the Bag doesn’t modify the contents of the underlying HashMap; it simply keeps a count of the number of times an item has been added to the Bag. This count is easily retrieved at c. The Lang and BeanUtils components are used together at e to get a value out of the TestMap of the TestBean and reset its value to a new one. The Lang component is used to capitalize the first letter of this value, which seems simple enough; and contrasting it with f, where a value is capitalized without this component, shows how easy it makes this task. The usage of the BeanUtils component provides a detailed study of the power of these components. As you can see in e, this component makes the process of getting and setting the value of a mapped property of a JavaBean at runtime very easy. Contrast this with the way Java reflection is used at d to achieve the same result, where accessing the value of the property requires several lines of code, and using the BeanUtils.getMappedProperty() method seems wonderful. We’ll now move on to discuss each of these components separately. In the next section, we’ll start with the Collections API. 7.2 Collections and more The Commons Collections API is a smorgasbord of utilities, tricks, enhancements, and implementations that work either with the Java Collections API or over it. This API includes several packages, and in this section we’ll subdivide each package into its own section. Table 7.1 shows the Java Collections API classes. Some of the examples require the use of classes from other packages as well as the main package (org.apache.commons.collections), and we’ll point out these cases as we go. You should review this table periodically as you go through the next few sections, because it provides a handy reference to the core Java Collections API and lest you contrast the core API with the new and enhanced components of the Collections component. Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF Table 7.1 Java Collections API classes Collection/Interface Properties Implementations Collection Root interface. Defines the general Collection contract. List An object that stores elements in a sequence. ArrayList (nonsynchronized) LinkedList (nonsynchronized) Vector (synchronized) Map An object that maps keys to values. Keys can’t be duplicated, and one key can map to only one value. HashMap (nonsynchronized, permits nulls) Hashtable (synchronized, no nulls) TreeMap (nonsynchronized, no nulls, ordered) WeakHashMap (based on weak references, allows nulls) Set An object that contains no duplicate elements and at most one null object. HashSet (nonsynchronized) LinkedHashSet (ordered) TreeSet (ordered) SortedMap A map whose elements are sorted based on the natural ordering of its keys. TreeMap SortedSet A set whose elements are sorted when traversed by an iterator. TreeSet We’ll start with an introduction of the Decorator pattern, which is used heavily in the Collections API to provide behavioral modifications of the basic Collections types. Don’t worry if you don’t know what the Decorator pattern is or why it’s important in our discussion of the Collections component; the next section explains it in detail with examples. 7.2.1 Decorate your classes’ behavior with Decorators The Decorator pattern allows objects to modify their behavior dynamically by delegating tasks around a hierarchy of objects. This means that objects don’t need to inherit their functionality from superclasses but, as required, can forward the task to a class that understands how to do it. We’ll explain further with an example. Consider listing 7.2, which shows three classes: DecoratorTesterV1, HighSalaryDecorator, and Salary. In this listing, the Salary class is the decorated class, and the HighSalaryDecorator class is the decorator. Listing 7.2 Showcasing the Decorator pattern package com.manning.commons.chapter07; public class DecoratorTesterV1 { public static void main(String args[]) { HighSalaryDecorator hsd = new HighSalaryDecorator(new Salary()); System.err.println(hsd.getSalary()); } } class HighSalaryDecorator { Salary salary; public HighSalaryDecorator(Salary salary) { this.salary = salary; } public long getSalary() { Decorator Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 5 return (salary.getBaseSalary() + 20000); } } class Salary { public long getBaseSalary() { return 50000; } } In conceptual terms, the HighSalaryDecorator class decorates the Salary class to provide a higher salary for the fortunate few. This is what decorators do (no, they don’t provide higher salaries): They decorate an existing class’s behavior by modifying it. There is a hierarchy between the HighSalaryDecorator class and the Salary class, which is demonstrated by the former containing a reference to the latter. Does this method of working with a class hierarchy seem familiar? If so, it’s because most of you have encountered it when using classes from the java.io package that follow this pattern (for example, the FilterOutputStream class is a decorator for the OutputStream class). What does all this have to do with the Collections component? Well, as we said before, Collections relies heavily on this pattern. Almost all new collection types introduced in this component have several common decorators that provide added functionality. These decorators are listed in table 7.2. Table 7.2 Decorators in the Collections component Decorator Class Function Composite CompositeCollection Provides decoration for a collection of collections, resulting in a unified single view. Predicated PredicatedCollection Decorates another collection by validating the elements that are added and rejecting the ones that fail validation. Synchronized SynchronizedCollection Decorates another collection to provide synchronized behavior for a multithreaded environment. Transformed TransformedCollection Decorates another collection by transforming the elements before they’re added to the decorated collection. Typed TypedCollection Decorates another collection to make sure the elements of the decorated collection are of a certain (given) type. It throws an IllegalArgumentException if an element other than the given type is added. Unmodifiable UnmodifiableCollection Decorates another collection to make sure it can’t be modified. Unmodifiable (bounded) UnmodifiableBoundedCollection Provides decoration to another (bounded) Collection, to ensure that its elements can’t be modified. (A bounded Collection is one that is limited in size.) Note that all the decorator classes listed in the table are defined in the org.apache.commons. collections.collection package. Almost all collection types in the Collections package extend these classes for type-specific decoration, and the decorate method of each type is used to apply the decoration. You’ll see examples of these classes in the coming sections. It’s also easy to add your own decorators if you aren’t happy with the ones that are already provided: You can extend the class AbstractCollectionDecorator in this package and provide the required decoration. Although most of the decorations are self-explanatory, the PredicatedCollection deserves to be explained in greater detail. Decorated Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF 7.2.2 Using the Predicate functor The PredicatedCollection is based on the idea of providing a simple way for collections to make a decision about the validity of an element. To do this, it uses the Predicate interface class, which is in the org.apache.commons.collections package. However, what is a predicate? Simple: A predicate is a functor. Still confused? OK: A functor is an object that represents a function, and a predicate is an object that performs the special function of returning a boolean true or false based on its input. Simply put, a predicate is the object equivalent of an If statement. Several functors (think special objects as functions) are available in the Collections package. These functor interfaces are defined in the main org.apache.commons.collections package and implemented in the org.apache.commons.collections.functors package. The interfaces are called Predicate, Closure, Transformer, and Factory. In this section, we’ll only talk about the Predicate functor classes. A list of the Predicate functor implementations is shown here. Note that some of these implementations use an extended version of the Predicate interface, called the PredicateDecorator; as we discussed in the previous section, it modifies an existing Predicate’s behavior. These classes are marked specially: ƒ AllPredicate (PredicateDecorator)—Returns true if all decorated predicates return true ƒ AndPredicate (PredicateDecorator)—Returns true for both decorated predicates ƒ AnyPredicate (PredicateDecorator)—Returns true if any decorated predicate returns true ƒ EqualPredicate—Returns true if the input to it is the same as that used to create the predicate ƒ ExceptionPredicate—Always throws an exception, no matter what the input ƒ FalsePredicate—Always returns false ƒ IdentityPredicate—Returns true if the input object’s reference is the same as the object that was used to create this predicate ƒ InstanceOfPredicate—Returns true if the input object is an instance of the type stored in the predicate ƒ NonePredicate (PredicateDecorator)—Returns true if none of the decorated predicates returns true ƒ NotNullPredicate—Returns true if the input isn’t Null ƒ NotPredicate (PredicateDecorator)—Returns the opposite of the decorated predicate ƒ NullIsExceptionPredicate (PredicateDecorator)—Throws an exception if the input is Null; otherwise returns the result of the decorated predicate ƒ NullIsFalsePredicate (PredicateDecorator)—Returns false if the input is Null; otherwise returns the result of the decorated predicate ƒ NullIsTruePredicate (PredicateDecorator)—Returns true if the input is Null; otherwise returns the result of the decorated predicate ƒ NullPredicate—Returns true if the input is Null ƒ OnePredicate (PredicateDecorator)—Returns true if only one decorated predicate returns true Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 7 ƒ OrPredicate (PredicateDecorator)—Returns true if any one of the decorated predicates returns true ƒ TransformedPredicate (PredicateDecorator)—Transforms the given input before returning the result of the predicate on the transformation ƒ TransformerPredicate—Returns the result of the transformed input, which must be of type Boolean; otherwise an exception is thrown ƒ TruePredicate—Always returns true, no matter what the input is ƒ UniquePredicate—Returns true or false based on adding the input to an internal Set; therefore, returns true the first time an input is supplied This section would be incomplete if we didn’t show you an example of how to use the Predicate functor classes. You’ll probably never need to use the classes we just listed; you’ll generally use the PredicateUtils class in the main org.apache.commons.collections package, which makes it easy to get an instance of any of these classes by using a static method. Listing 7.3 uses the code from listing 7.2 and adds a method to set an employee’s salary. However, the catch is that no two employees can have the exact same salary, and this requirement is implemented by using the UniquePredicate. Listing 7.3 Making sure that no two employees have the same salary package com.manning.commons.chapter07; import org.apache.commons.collections.Predicate; import org.apache.commons.collections.PredicateUtils; public class DecoratorTesterV2 { Predicate uniquePredicate = PredicateUtils.uniquePredicate(); public static void main(String args[]) { HighSalaryDecorator hsd = new HighSalaryDecorator(new Salary()); System.err.println(hsd.getSalary()); } public void setEmployeeSalary(long salary) { if(uniquePredicate.evaluate(new Long(salary))) { // proceed to set the salary } else { System.err.println("Same salary " + salary + " amount already set"); } } } // NOTE: Other classes omitted As you can see, you use the evaluate method of the Predicate instance to find out whether the predicate returns true or false. In this case, the same salary won’t be set for two employees because the predicate will block it. Listing 7.4 shows how to use a predicate that uses a decorator to modify its behavior. In this case, we want to modify listing 7.3 to let us set the salary for employees who work voluntarily for $1. This means the salary can be set either if it hasn’t been set before or if it equals $1. Create UniquePredicate instance Method to set employee salary Predicate evaluates to false if salary isn’t unique Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF Listing 7.4 Allowing volunteer workers’ salaries to be set package com.manning.commons.chapter07; import org.apache.commons.collections.Predicate; import org.apache.commons.collections.PredicateUtils; public class DecoratorTesterV3 { Predicate uniquePredicate = PredicateUtils.uniquePredicate(); Predicate equalPredicate = PredicateUtils.equalPredicate(new Long(1)); Predicate anyPredicate = PredicateUtils.anyPredicate( new Predicate[] {uniquePredicate, equalPredicate}); public static void main(String args[]) { HighSalaryDecorator hsd = new HighSalaryDecorator(new Salary()); System.err.println(hsd.getSalary()); } public void setEmployeeSalary(long salary) { if(anyPredicate.evaluate(new Long(salary))) { // proceed to set the salary } else { System.err.println("Same salary " + salary + " amount already set"); } } } // NOTE: Other classes omitted Using a decorator predicate makes combining decisions easier. In this listing, an instance of AnyPredicate decorates an instance of EqualPredicate and UniquePredicate. AnyPredicate returns true if any of the decorated predicates returns true. Thus, if a volunteer with a $1 salary comes along in your organization, the EqualPredicate part of the AnyPredicate decorator will return true, and the salary will be set. The UniquePredicate part will still work for non-$1 salaries. You now know substantially more about decorators and predicates than before, and it’s time to look at some of the new types in the Collections API. We’ll start with the Bag interface. 7.2.3 How many cookies in your Bag? The Bag interface is a nifty little collection that keeps track of the number of times an item appears in it. When you think about it, no collection in the Java Collections API does the same thing. The closest collection to the Bag interface in the core API is the List interface, because it allows duplicate elements. However, unlike the List interface, the Bag interface doesn’t stipulate that multiple copies of the same element exist within the collection. If an item is added to an implementation of the Bag interface, and that item exists already, it isn’t added again, but the count for that item is incremented. Thus, each element exists only once, and the interface keeps a count of the number of times it’s added or removed. This raises a question: How does the Bag interface determine whether an element exists in the interface? It does so the same way the Set interface from the core API does—by performing an equality check using the equals method. So, two elements e1 and e2 are equal if e1.equals(e2) returns true. Internally, a Bag implementation maintains what is called a unique set of its elements. Thus, when you add a new element, if Create EqualPredicate with value of 1 Create decorator anyPredicate Evaluate any registered predicate Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 9 that element exists in this unique set, it isn’t added; only its count is incremented. You can access this unique set of elements by calling the method uniqueSet on any implementation of this interface. The following list describes the interface definitions and classes associated with the Bag interface. Except for the interface definitions Bag and SortedBag, all the other classes are in the org.apache. commons.collections.Bag package: ƒ Bag (interface)—Defines the basic Bag interface ƒ SortedBag (interface)—Extends the Bag interface, and makes sure the elements of the unique set are sorted according to their natural order of sorting ƒ HashBag—HashMap-based Bag implementation. ƒ TreeBag—TreeMap-based SortedBag implementation These aren’t the only classes in the org.apache.commons.collections.Bag package, however. This package contains several decorator implementations (as do all the collection types in this component), which can be used with their corresponding interface. For example, the TransformedBag decorator works with a Bag interface implementation (currently only HashBag), and the TransformedSortedBag decorator works with a SortedBag interface (currently only TreeBag). If you’re feeling a bit confused about this multitude of classes, refer to Figure 7.1, which shows the relationship between the TransformedBag class, the TransformedCollection class, the HashBag class, and the Bag interface. Figure 7.1 Relationships between the decorator, the decorated, and the interface Using the classes of the Bag interface in code is straightforward. We’ll illustrate with the example of two cookie Bags: One contains its cookies in a random order, and the other contains cookies that are sorted by type. Initially the Bags contain a cookie of each type (the standard cookie types are Bar, Drop, Brownies, Cut Out, Molded, Sliced, and No Bake). This initial setup is shown in listing 7.5. Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF Listing 7.5 Creating the cookie Bags package com.manning.commons.chapter07; import org.apache.commons.collections.Bag; import org.apache.commons.collections.bag.HashBag; import org.apache.commons.collections.bag.TreeBag; import java.util.Arrays; public class CookieBagV1 { private Bag cookieBag; private Bag sortedCookieBag; public static void main(String args[]) { CookieBagV1 app = new CookieBagV1(); app.prepareBags(); app.printBagContents(); } private void printBagContents() { System.err.println("Cookie Bag Contents: " + cookieBag); System.err.println("Sorted Cookie Bag Contents: " + sortedCookieBag); } private void prepareBags() { prepareCookieBag(); prepareSortedCookieBag(); } private void prepareCookieBag() { cookieBag = new HashBag(Arrays.asList(cookieJar)); } private void prepareSortedCookieBag() { sortedCookieBag = new TreeBag(Arrays.asList(cookieJar)); } private String[] cookieJar = {"Bar", "Drop", "Brownies", "Cut Out", "Molded", "Sliced", "No Bake"}; } Elements can be added to a Bag two ways: through an existing collection in the constructor or later by using either the add(Object element) method to add one or add(Object element, int nCopies) to add multiple. For example, the following code snippet adds multiple random cookies to a random Bag: private void addRandomCookies() { int count = (int)(Math.random() * 10); int pick = (int)(Math.random() * 10); pick = pick > 6 ? 6 : pick; if (count > 5) cookieBag.add(cookieJar[pick], count); Store unsorted cookies Store cookies sorted by type Create HashBag from cookieJar Create TreeBag from cookieJar Store types of cookies Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 11 else sortedCookieBag.add(cookieJar[pick], count); } Removing elements from the Bag differs slightly from adding them, because if you call remove(Object element) it removes all copies of the element, not just one. To remove a specified number, use remove(Object element, int nCopies). Now, let’s assume that you like only a specific type of cookie, and you want only those specific cookies in your Bag. It would be nice if somehow any cookie that you add to the Bag transformed into your favorite kind. Can you do this transformation with the classes of this package? Yes, of course, by using the TransformedBag class. TransformedBag is a decorator for the Bag interface. However, you don’t know that to use this class, you must use a Transformer. We haven’t studied this class yet, and it’s time to take another diversion into the world of functors. Magical transformations with the Transformer functor Like the Predicate functors we discussed in section 7.2.1, Transformer functors are special objects that do the work of a function. Predicate instances were used to perform validation functions; in the case of Transformer instances, this function is to transform an input object to an output object, leaving the input object unchanged. As with the Predicate functors, the main Transformer interface is in the org.apache.commons.collections package, whereas actual implementations are in the org.apache.commons.collections.functors package. Once again, though, you probably won’t use the classes in the latter package directly; you’ll use the TransformerUtils class instead. Following is a list of available Transformer functors. As you may notice, transformers not only transform input objects, but some of them can also be used to extract information from this object: ƒ ChainedTransformer—Chains several transformers together, and returns the result. ƒ CloneTransformer—Returns a clone of the input object. ƒ ClosureTransformer—Runs a closure using the input object, and returns the result. ƒ ConstantTransformer—Ignores the input, and always returns the value used to create the transformer initially. ƒ ExceptionTransformer—Always throws an exception, no matter what the input is. ƒ FactoryTransformer—Calls a specified factory, and returns the result. ƒ InstantiateTransformer—Creates a new object using reflection, and returns it. Both the class and the parameters for the class must be supplied. ƒ InvokerTransformer—Invokes a method on an instance with the supplied parameters, and returns the result. ƒ MapTransformer—Uses the input key to look up a value in an existing supplied map, and returns the value. ƒ NOPTransformer—Does nothing; returns the input as is. ƒ PredicateTransformer—Calls a Predicate functor with the input value, and returns the result. ƒ StringValueTransformer—Calls String.valueOf on the input, and returns the result. Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF ƒ SwitchTransformer—Uses a predicate-to-transformer map during construction. When a transform method is called, like a switch statement, it uses the input to run through all the predicates and returns the value of the transform matching the predicate that returns true. Transformation is performed by using the transform method, much like the evaluate method of the predicates is used to perform validations. This is illustrated in listing 7.6. Listing 7.6 Using InvokerTransformer package com.manning.commons.chapter07; import org.apache.commons.collections.Transformer; import org.apache.commons.collections.TransformerUtils; public class TransformerExampleV1 { public static void main(String args[]) { Transformer transformer = TransformerUtils.invokerTransformer( "append", new Class[] {String.class}, new Object[] {" a Transformer?"}); Object newObject = transformer.transform(new StringBuffer("Are you")); System.err.println(newObject); } } The output of running this listing is Are you a Transformer? The input StringBuffer’s append method is called when the transform is done. This append method is passed the " a Transformer?" parameter value defined earlier to get the desired output. Now that you know how to use the Transformer functors, it’s time to see if we can achieve the goal of transforming all cookies that are added to our cookie Bag to our favorite cookie. Using transformers with types Transformers are very useful in the Collections API because they’re flexible and provide a lot of choices in the way objects are retained within the collection types. Continuing with our cookie example, let’s see how the ConstantTransformer helps us have only the cookie of our choice in our Bag. Listing 7.7 shows a modified version of listing 7.5 that uses the ConstantTransformer to retain only one type of cookie. Listing 7.7 Using the ConstantTransformer with the Bag collection type package com.manning.commons.chapter07; import org.apache.commons.collections.Bag; import org.apache.commons.collections.bag.HashBag; import org.apache.commons.collections.bag.TreeBag; import org.apache.commons.collections.TransformerUtils; import org.apache.commons.collections.bag.TransformedBag; import java.util.Arrays; Type of method’s parameters Value of method’s parameters Method to invoke Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 13 public class CookieBagV2 { private Bag cookieBag; private Bag sortedCookieBag; public static void main(String args[]) { CookieBagV2 app = new CookieBagV2(); app.prepareBags(); app.printBagContents(); app.addRandomCookies(); app.printBagContents(); } private void printBagContents() { System.err.println("Cookie Bag Contents: " + cookieBag); System.err.println("Sorted Cookie Bag Contents: " + sortedCookieBag); } private void addRandomCookies() { int count = (int)(Math.random() * 10); int pick = (int)(Math.random() * 10); pick = pick > 6 ? 6 : pick; if (count > 5) cookieBag.add(cookieJar[pick], count); else sortedCookieBag.add(cookieJar[pick], count); } private void prepareBags() { prepareCookieBag(); prepareSortedCookieBag(); } private void prepareCookieBag() { cookieBag = TransformedBag.decorate( new HashBag(Arrays.asList(cookieJar)), TransformerUtils.constantTransformer(cookieJar[2])); } private void prepareSortedCookieBag() { sortedCookieBag = new TreeBag(Arrays.asList(cookieJar)); } private String[] cookieJar = {"Bar", "Drop", "Brownies", "Cut Out", "Molded", "Sliced", "No Bake"}; } Most of this listing remains the same as listing 7.5, with the addition of the addRandomCookies method. The main change however, is in the prepareCookieBag method, where the ConstantTransformer class is used to convert any cookie that is added to the Bag to the Brownies cookie type. The transformed cookie Bag is created, and the method calls the decorate method on it. The method then has the basic cookie Bag with the original cookies, after which it chooses Brownies as the constant. The important thing to remember about this listing is that once a transformation is applied, existing elements in the collection do not change. Only elements added after the transformation are affected. Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF Therefore, the cookie Bag will still contain the initial seven types of cookies; but any new cookies added by the random method will all be brownies, no matter which random cookie is picked. Of course, if the decoration of the transformation is applied before any cookies are added, the Bag will always contain brownies. In a nutshell, to apply a decoration on a collection type, you must call the decorate method of the decoration on a basic type. Doing so returns a decorated type, which follows the rules of the applied decoration. For example, in listing 7.7, TransformedBag’s decorate method is called on the basic HashBag type to return a decorated TransformedBag. The same principle applies to all decorations and all collection types. In the following sections, we’ll cover the last two functor types available in the Collections API. Closure functors What is a ClosureTransformer? Well, a closure is a functor. A ClosureTransformer is a transformer that calls a Closure functor using its input and returns what this functor returns. But this section is about the Closure functor, and you still don’t know what it is. Just as Predicate functors are for validation and Transform functors are for transformation, Closure functors are objects that represent a block of code inside a method or iteration. Think of for/while loops, and then think of Closure functors. If you want a block of code to run n times, you use ForClosure and provide the iteration count. The main method for a Closure is the execute method, which doesn’t return a value. The following list describes the available Closure functors. As before, you’ll probably only use the ClosureUtils class to retrieve instances of these classes: ƒ ChainedClosure—Executes a list of chained Closures. ƒ ExceptionClosure—Always throws an exception, no matter what the input is. ƒ ForClosure—Runs a supplied Closure n times with the given input. ƒ IfClosure—Based on a main predicate, runs either one of two other closures. ƒ NOPClosure—Closure that does nothing. ƒ SwitchClosure—Uses two arrays, one for predicates and another for transformers, which map one on one to each other during construction. When the execute method is called, like a switch statement, it uses the input to run through all the predicates and executes the closure that maps to the predicate that returns true. ƒ TransformClosure—Calls a transformer on the input, but then ignores the result. ƒ WhileClosure—Executes an input until a supplied predicate returns false. It’s much like a while loop. Listing 7.8 shows a simple example of using IfClosure. Listing 7.8 Using the IfClosure class package com.manning.commons.chapter07; import org.apache.commons.collections.Closure; import org.apache.commons.collections.ClosureUtils; import org.apache.commons.collections.PredicateUtils; public class ClosureExampleV1 { Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 15 public static void main(String args[]) { Closure ifClosure = ClosureUtils.ifClosure( PredicateUtils.equalPredicate(new Integer(20)), ClosureUtils.nopClosure(), ClosureUtils.exceptionClosure()); ifClosure.execute(new Integer(20)); ifClosure.execute(new Integer(30)); } } We first create an IfClosure. Then we use the ClosureUtils class and create an EqualPredicate. There’s a true closure and then a false closure. The true predicate does nothing, and the false predicate throws an exception. And then there was one: Factory The final functor available in the Collections API is Factory. As the name suggests, this functor is used to create instances of objects. However, no input is expected in these factories. The main method for all Factory functors is the create method, as opposed to execute for Closure, evaluate for Predicate, and transform for Transformer. There are only three implementations of the Factory functor in the org.apache.commons. collections.functor package. As with the rest of the functors, you use the FactoryUtils class to create the instances of these classes. Following is a list of these classes: ƒ ConstantFactory—Returns the same constant value, supplied during construction ƒ ExceptionFactory—Always throws an exception ƒ InstantiateFactory—Uses reflection to create an instance of a class Listing 7.9 uses InstantiateFactory to create an instance of the StringBuffer class with the initial text Do you like functors?. Listing 7.9 Using the InstantiateFactory class package com.manning.commons.chapter07; import org.apache.commons.collections.Factory; import org.apache.commons.collections.FactoryUtils; public class FactoryExampleV1 { public static void main(String args[]) { Factory bufferFactory = FactoryUtils.instantiateFactory( StringBuffer.class, new Class[] {String.class}, new Object[] {"Do you like Functors?"}); System.err.println(bufferFactory.create()); } } Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF In this code, we create a StringBuffer instance Factory. We use FactoryUtils and declare the class type of the factory, the type of the parameters, and the value of the parameters. Finally, the StringBuffer is created using this Factory. As you can probably see by now, all the functors require similar procedures. This is helpful, because it brings uniformity to the way collection types are modified or decorated. In the next section, we’ll restart our discussion of the new types in the Collections API, which we left after discussing the Bag interface initially. 7.2.4 Map(ping) your way through Collections The original Map interface defined in the Java core API’s collections is a source of inspiration for several new classes and interfaces in the Collections package. Some of these classes and interfaces take a very different direction from the original interface, and some are worthy enhancements. For example, the classes in the package org.apache.commons.collections.bidimap are enhancements because they provide map lookup in both directions (hence the name bidirectional, or bidi for short), which isn’t possible with the basic interface. In this section, we’ll divide the discussion of the Map interface–related classes into their respective packages: ƒ org.apache.commons.collections.keyvalue ƒ org.apache.commons.collections.map ƒ org.apache.commons.collections.bidimap Not sure of direction? Use bidimap Suppose you’re building an application that manages the world’s super spies. One of this application’s tasks is to assign a unique code to each spy so you can look them up based on the code. So, you decide to put the names of the spies in a HashMap, along with their codes as a key. Listing 7.10 shows this simple application. Listing 7.10 Managing super spies using a HashMap package com.manning.commons.chapter07; import java.util.HashMap; public class HashMapExampleV1 { public static void main(String args[]) { HashMap agentToCode = new HashMap(); agentToCode.put("007", "Bond"); agentToCode.put("006", "Trevelyan"); agentToCode.put("002", "Fairbanks"); System.err.println(agentToCode); } } Given a code for a spy, you can easily look up his name using this HashMap. However, suppose that you get a panicked call from a spy in the field who can only remember his name and not his code. How can you retrieve his code from this HashMap based on the name? The simple answer is, you can’t. You need to use a bidirectional map that allows lookups in both directions, as shown in listing 7.11. Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 17 Listing 7.11 Using a bidirectional map package com.manning.commons.chapter07; import org.apache.commons.collections.BidiMap; import org.apache.commons.collections.bidimap.DualHashBidiMap; public class BidiMapExampleV1 { public static void main(String args[]) { BidiMap agentToCode = new DualHashBidiMap(); agentToCode.put("007", "Bond"); agentToCode.put("006", "Trevelyan"); agentToCode.put("002", "Fairbanks"); System.err.println("Agent name from code: " + agentToCode.get("007")); System.err.println("Code from Agent name: " + agentToCode.getKey("Bond")); } } First the DualHashBidiMap is created; it’s a map that uses two HashMap instances internally to store the data resulting in duplicate storage. The tradeoff comes from the ability to look up the values based on their keys. If you’re unhappy with the double storage, you can instead use the TreeBidiMap class, which uses interconnected nodes to hold the data only once. However, again there is a tradeoff in terms of data access speeds, because using TreeBidiMap is slower than using DualHashBidiMap. An added advantage comes from using TreeBidiMap, however: Data in this map is sorted in ascending order for both the key and the value. One caveat of using a bidimap is that both keys and values added in this map must be unique. With normal map instances, this restriction applies only to keys—values can be duplicated. However, because a BidiMap instance provides a way to look up keys based on values, values must have a one-to-one relationship with keys, and vice versa. This package contains a single decorator named UnmodifiableBidiMap that restricts any modifications on the bidimap once it’s decorated. Listing 7.12 shows an example of using this class. Listing 7.12 Restricting modifications on a bidimap using a decorator package com.manning.commons.chapter07; import org.apache.commons.collections.BidiMap; import org.apache.commons.collections.bidimap.DualHashBidiMap; import org.apache.commons.collections.bidimap.UnmodifiableBidiMap; public class BidiMapExampleV2 { public static void main(String args[]) { BidiMap agentToCode = new DualHashBidiMap(); agentToCode.put("007", "Bond"); agentToCode.put("006", "Trevelyan"); Create bidimap backed by two HashMaps Populate map Look up value based on key Look up key based on value Create and populate bidimap Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF agentToCode = UnmodifiableBidiMap.decorate(agentToCode); agentToCode.put("002", "Fairbanks"); agentToCode.remove("007"); agentToCode.removeValue("Bond"); } } As you can see, any attempts to add new agents or get rid of Mr. Bond are thwarted easily once the decoration is set. You can still see the agents in the map, but you can’t change them. Finally, here’s a list of relevant classes in this package: ƒ AbstractDualBidiMap—Base class for bidimaps based on two maps ƒ DualHashBidiMap—Extends AbstractDualBidiMap using two HashMap instances as storage ƒ DualTreeBidiMap—Extends AbstractDualBidiMap using two TreeMap instances as storage ƒ TreeBidiMap—Ordered bidimap that holds its data with interconnected nodes ƒ UnmodifiableBidiMap—Decorates other bidimap instances to make sure they aren’t altered after decoration; extends AbstractBidiMapDecorator Next, we’ll look at a class from the org.apache.commons.collections.keyvalue package, which allows multiple keys to be merged together. Why use one key? Use MultiKey! The classes of the org.apache.commons.collections.keyvalue package provide specialized versions of key-value mappings that are immensely useful in situations where normal key value handling doesn’t suffice. One of the more interesting classes in this package is MultiKey. The package’s other classes are simple management classes and, as such, are easy to use; therefore, we won’t discuss them here. The MultiKey class is useful for keys for HashMap instances, where a single key may not suffice. Consider a situation where you’re building a multilingual application that can greet people based on a code. You want to store the greetings in a HashMap using the code as the key, but since the application is multilingual, you also need to store the greetings under a language code. How do you store your greetings under both their own code and a language code? Conventionally, by using what is known as a map of maps. This is illustrated in listing 7.13. Listing 7.13 Creating a map of maps for a multilingual application package com.manning.commons.chapter07; import java.util.HashMap; public class MultiKeyExampleV1 { public static void main(String args[]) { HashMap codeToText_en = new HashMap(); codeToText_en.put("GM", "Good Morning"); codeToText_en.put("GN", "Good Night"); codeToText_en.put("GE", "Good Evening"); Decorate bidimap with UnmodifiableBidiMap Throw UnsupportedOperationException Create English codes in greeting map Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 19 HashMap codeToText_de = new HashMap(); codeToText_de.put("GM", "Guten Morgen"); codeToText_de.put("GE", "Guten Abend"); codeToText_de.put("GN", "Guten Nacht"); HashMap langToMap = new HashMap(); langToMap.put("en", codeToText_en); langToMap.put("de", codeToText_de); System.err.println("Good Evening in English: " + ((HashMap)langToMap.get("en")).get("GE")); System.err.println("Good Night in German: " + ((HashMap)langToMap.get("de")).get("GN")); } } The map of maps works by storing individual greetings and their codes in language-specific Map instances. These instances are then stored in one big map that uses the language code as the key to each greeting map. When a greeting is required, it’s accessed by first getting the map for a particular language and then using the greeting code on the resulting map. Is there a better way of achieving the same result? Yes: By using the MultiKey class. A MultiKey class combines keys to achieve the same result as a map of maps, as shown in listing 7.14. Listing 7.14 Using the MultiKey class to replace the map of maps package com.manning.commons.chapter07; import java.util.HashMap; import org.apache.commons.collections.keyvalue.MultiKey; public class MultiKeyExampleV2 { private static HashMap codeAndLangToText; public static void main(String args[]) { codeAndLangToText = new HashMap(); addMultiKeyAndValue("en", "GM", "Good Morning"); addMultiKeyAndValue("en", "GE", "Good Evening"); addMultiKeyAndValue("en", "GN", "Good Night"); addMultiKeyAndValue("de", "GM", "Guten Morgen"); addMultiKeyAndValue("de", "GE", "Guten Abend"); addMultiKeyAndValue("de", "GN", "Guten Nacht"); System.err.println("Good Evening in English: " + codeAndLangToText.get(new MultiKey("en", "GE"))); System.err.println("Good Night in German: " + codeAndLangToText.get(new MultiKey("de", "GN"))); } private static void addMultiKeyAndValue( Object key1, Object key2, Object value) { Create German codes in greeting map Create master map using other maps Access greetings based on language and code Create and populate greeting map Access greeting using MultiKey Method to add MultiKey and values Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF MultiKey key = new MultiKey(key1, key2); codeAndLangToText.put(key, value); } } The basic advantage of using the MultiKey class is to reduce the overhead of creating multiple HashMap instances, as was the case in listing 7.13. The MultiKey is constructed by providing its constituent keys in its constructor. Note that there is no limit to the number of keys that can be used. The constructor used in listing 7.14 was for convenience; we could easily have used the alternate constructor, MultiKey(Object[] keys), to provide as many keys as we wanted. In the next section, we’ll cover the more exotic map implementations found in the last Map package of the Collections API. Map heaven—all types of map implementations The last package of the Collections API that has a map-based flavor is org.apache. commons.collections.map. This package contains several implementations of the Map interface that serve varied purposes; it also contains multiple decorators. These implementations are as follows: ƒ CaseInsensitiveMap—Map that performs a key comparison in a case-insensitive manner. ƒ Flat3Map—A targeted map implementation that has faster operations than a HashMap for map sizes of three or less. ƒ HashedMap—General-purpose replacement for Java’s HashMap. ƒ IdentityMap—A map that doesn’t use equals() for comparison of its keys, but instead uses the identity check ==. ƒ LinkedMap—A map that retains the order of its elements as they were inserted. ƒ LRUMap—A map that is fixed in size and removes the least recently used (lru) element when new elements are added, to make space. ƒ MultiKeyMap—Uses MultiKey to create a map based on multiple keys. ƒ ReferenceIdentityMap—A combination of a ReferenceMap and an IdentityMap. ƒ ReferenceMap—A map that’s optimized for memory and lets the garbage collector reclaim map elements once they become unreachable, using the equals method for testing. ƒ SingletonMap—A single-element map implementation that doesn’t change in size once it has been created. The single value can be changed, but not the single key. ƒ StaticBucketMap—A thread-safe implementation of the Map interface. In addition to these classes, this package includes a few new decorators: FixedSize, which ensures that the size of map doesn’t change; Lazy, which ensures that map elements are only created on demand; and ListOrdered, which ensures that the insertion order of elements is retained. Let’s look at an example of using a few of these implementations and a decoration. In listing 7.15, we use two map instances from the previous list along with the Lazy decorator. Create MultiKey Put key in map with value Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 21 Listing 7.15 Using map implementations package com.manning.commons.chapter07; import java.util.Map; import java.util.Date; import java.util.HashMap; import org.apache.commons.collections.map.LazyMap; import org.apache.commons.collections.FactoryUtils; import org.apache.commons.collections.map.IdentityMap; import org.apache.commons.collections.map.CaseInsensitiveMap; public class MapHeavenV1 { public static void main(String args[]) { MapHeavenV1 instance = new MapHeavenV1(); instance.createMaps(); instance.testMaps(); } private void testMaps() { cIMap.put("key1", "value1"); cIMap.put("key2", "value2"); cIMap.put("KeY1", "value3"); System.err.println("Value of key1: " + cIMap.get("key1")); Integer identRef = new Integer(1); Integer identRef2 = new Integer(1); identMap.put(identRef, "value1"); identMap.put(identRef2, "value3"); System.err.println("Value of identRef2: " + identMap.get(identRef2)); System.err.println(lazyMap); lazyMap.get("EmptyBuffer"); System.err.println(lazyMap); } private void createMaps() { cIMap = new CaseInsensitiveMap(); identMap = new IdentityMap(); lazyMap = LazyMap.decorate( new HashMap(), FactoryUtils.instantiateFactory(StringBuffer.class)); } private CaseInsensitiveMap cIMap; private IdentityMap identMap; private Map lazyMap; } Replace previous Key1 value Print value3 because it’s case insensitive Two different elements, even though keys are equal Contains no elements Create and store empty Buffer Lazy Map decorates HashMap Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF Using the rest of the map instances of this package is a similar exercise. So, we’ll now move on to the next section and continue the coverage of the other collection types by looking at the org.apache. commons.collections.list package. 7.2.5 Make a List and check it twice A List collection type holds its elements in a sequence, where the user of the List has precise control over where elements are added and removed from. The Java core Collection library provides three List implementations: ArrayList, Vector, and LinkedList (see table 7.1). The Collections API provides three of its own implementations, in addition to several decorators, in the package org.apache. commons.collections.list. The three implementations are TreeList, CursorableLinkedList, and NodeCachingLinkedList. TreeList is an optimized implementation of the List interface that provides faster performance for most operations than the core List implementations. Internally, it uses a binary search tree (hence the name TreeList) for the storage of its elements. This tree structure is called AVL (named after its inventors, Adelson-Velskii and Landis); you can get more information about it at http://ciips.ee.uwa.edu.au/~morris/Year2/PLDS210/AVL.html. If you’re looking for added speed in your applications, it’s highly recommended that you use this implementation. However, it’s more memory intensive than the other lists. CursorableLinkedList is a List implementation that provides a cursor with which the list can be traversed. This cursor is the same as an iterator (a ListIterator, actually), and any changes made via this cursor are reflected back in the list and vice versa. NodeCachingLinkedList, as the name suggests, is a specialized list that works as an improved LinkedList. It strives for faster response times for large lists that have a lot of element addition and removal, by caching the nodes of the internal linked list in a default-sized cache. In addition to the usual decorators, this package introduces a new decorator called SetUnique. This decorator effectively converts the list it decorates into an ordered set, thereby only storing unique elements. Listing 7.16 shows examples of using the TreeList with the SetUnique decorator and the effect of using the CursorableLinkedList’s cursor (ListIterator). Listing 7.16 Using the List implementations package com.manning.commons.chapter07; import org.apache.commons.collections.list.TreeList; import org.apache.commons.collections.list.SetUniqueList; import org.apache.commons.collections.list.CursorableLinkedList; import java.util.List; import java.util.ListIterator; public class ListExampleV1 { public static void main(String args[]) { ListExampleV1 listExample = new ListExampleV1(); listExample.createLists(); uniqueList.add("Value1"); uniqueList.add("Value1"); System.err.println(uniqueList); cursorList.add("Element1"); cursorList.add("Element2"); 0th element Does nothing, because “Value1” exists in unique list Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 23 cursorList.add("Element3"); ListIterator iterator = cursorList.listIterator(); iterator.next(); iterator.add("Element2.5"); System.err.println(cursorList); } private void createLists() { uniqueList = SetUniqueList.decorate(new TreeList ()); cursorList = new CursorableLinkedList(); } private static List uniqueList; private static List cursorList; } As you can see, using the List implementations isn’t much different from using the other collection types and the Java core Collections API. You can experiment with the other decorators in this package to arrive at varying list types—the SynchronizedList, the TransformedList, the TypedList, and so on. In the next section, we’ll look at the org.apache.commons.collections.set package, which contains implementations of the Set interface. 7.2.6 Decorating your Set The Set package of the Collections API doesn’t contain any new implementation types. This package only defines a series of decorators, almost all of which we have encountered before, except MapBackedSet and CompositeSet. Let’s look at these set decorators with the help of examples. Using a MapBackedSet Recall from table 7.1 that a set is a collection type that contains no duplicate elements and at most one null element. A map, on the other hand, is a key-value based collection of elements, where the keys are guaranteed to be unique. How do you create a map-backed set? Simple: The map is decorated by the set using the keys of the map as its elements. This is the premise of the MapBackedSet, as illustrated in listing 7.17. Listing 7.17 Using a MapBackedSet package com.manning.commons.chapter07; import org.apache.commons.collections.set.MapBackedSet; import java.util.Map; import java.util.Set; import java.util.HashMap; import java.util.Iterator; public class SetExampleV1 { public static void main(String args[]) { Map map = new HashMap(); map.put("Key1", "Value1"); Return cursor on List Cursor now between 0th and 1st element Add element between 0th and 1st element Decorate TreeList Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF Set set = MapBackedSet.decorate(map); map.put("Key2", "Any dummy value"); set.add("Key3"); Iterator itr = set.iterator(); while(itr.hasNext()) { System.err.println(itr.next()); } } } In this code, we create and initialize the map and then create the decoration. We make modifications to both the map and the set, and finally print the set’s contents. The key thing to remember while using the MapBackedSet is that modifications work both ways. Therefore, if an element is put in the map or added to the set, both the set view and the map views are modified. Note that the value, while using the put method for the Map, is immaterial, because it isn’t used anywhere. Using a CompositeSet The CompositeSet implementation works with multiple set implementations and provides a unified view. However, before this implementation can be used, it must be told what to do when elements are added to or removed from it (for example, where are the added elements actually added, from where are the removed elements taken, and so on) or when a conflict occurs due to similar elements in any of the sets, because a set can’t contain duplicate elements. You do so by implementing the SetMutator interface, which defines methods that clearly outline what needs to be done in such situations. Listing 7.18 shows an example of using the CompositeSet with an implementation of the SetMutator. Listing 7.18 Using the CompositeSet class with a SetMutator implementation package com.manning.commons.chapter07; import org.apache.commons.collections.collection.*; import org.apache.commons.collections.set.*; import java.util.Set; import java.util.HashSet; import java.util.Iterator; import java.util.Collection; public class SetExampleV2 { public static void main(String args[]) { Set set1 = new HashSet(); set1.add("Red"); set1.add("Green"); Set set2 = new HashSet(); set2.add("Yellow"); set2.add("Red"); Create two sets with one common element Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 25 CompositeSet composite = new CompositeSet(); composite.setMutator(new CompositeMutator()); composite.addComposited(new Set[] {set1, set2}); composite.add("Pink"); composite.remove("Green"); Iterator itr = composite.iterator(); while(itr.hasNext()) { System.err.println(itr.next()); } } } class CompositeMutator implements CompositeSet.SetMutator { public void resolveCollision( CompositeSet comp, Set existing, Set added, Collection intersection) { added.removeAll(intersection); } public boolean add( CompositeCollection collection, Collection[] collections, Object obj) { return collections[0].add(obj); } public boolean remove( CompositeCollection collection, Collection[] collections, Object obj) { return collections[0].remove(obj); } public boolean addAll( CompositeCollection collection, Collection[] collections, Collection coll) { return collections[0].addAll(coll); } } Using the CompositeSet, decoration is easy if you remember to provide the strategy for resolving conflicts and operations on the composite. In listing 7.18, this functionality is provided by the CompositeMutator Remove conflict elements from added set Resolve “Red” element conflict Add all elements to first set Create class that handles operations Initialize Composite with two sets Add new elements to first set Remove elements from first set Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF class, which implements the interface CompositeSet.SetMutator. Note that this strategy must be set before the composite is initialized with Set instances; otherwise conflicts within the added instances will throw an UnsupportedOperationException. If you’re sure that there will be no conflicts and you only want a view of the added sets without performing any modifications, then you don’t need to set this strategy. Note: This decorator differs from the other decorators we’ve discussed, because it doesn’t provide a decorate method Let’s now look at the final collection type in this API, Buffer, which follow a well-defined order for the removal of elements. 7.2.7 Prioritizing and blocking element removal with a Buffer A Buffer is a collection type that follows a particular order for the removal of elements. There is no order for the addition of elements, nor is an order followed for their iteration—only for removal. The Buffer interface is defined in the main org.apache.commons.collections package, and the implementation classes are in the org.apache.commons.collections.buffer package along with several decorators. There is one implementation class in the main package as well: ArrayStack. The implementation classes are as follows: ƒ ArrayStack—Stack implementation based on an ArrayList (not Vector, so it’s unsynchronized). Because it implements Buffer and Stack, it’s essentially a last-in-first-out (LIFO) Buffer implementation. Accepts Null elements. ƒ BoundedFifoBuffer—A fixed-size first-in-first-out (FIFO) Buffer implementation. Doesn’t accept Null elements. ƒ CircularFifoBuffer—A fixed-size FIFO Buffer implementation that replaces its oldest element with a new element if it reaches its maximum size. Doesn’t accept Null elements. ƒ PriorityBuffer—A Binary heap Buffer implementation, where elements are removed based on their natural order of comparison, or a specially provided Comparator. Accepts Null elements. ƒ UnboundedFifoBuffer—A FIFO Buffer implementation that doesn’t accept Null elements. The only new decorator implementation in this package is the blocking decorator. If a decorated blocking buffer is empty, and a call to either the get or remove method is made, the buffer waits (blocks) until an element is made available by either the add or addAll method. There are no timeouts to be set, so you may wait indefinitely if no element is forthcoming. Listing 7.19 shows an example of using this decorator with the PriorityBuffer. Listing 7.19 Using the blocking decorator with the PriorityBuffer package com.manning.commons.chapter07; import org.apache.commons.collections.Buffer; import org.apache.commons.collections.buffer.BlockingBuffer; import org.apache.commons.collections.buffer.PriorityBuffer; public class BufferExampleV1 { public static void main(String args[]) { Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 27 Buffer buffer = new PriorityBuffer (); buffer.add("2"); buffer.add("1"); buffer = BlockingBuffer.decorate(buffer); buffer.clear(); AddElementThread runner = new AddElementThread(buffer); runner.start(); buffer.remove(); System.err.println(buffer); } } class AddElementThread extends Thread { private Buffer buffer; public AddElementThread(Buffer buffer) { this.buffer = buffer; } public void run() { try { sleep(2000); } catch (InterruptedException ie) {} buffer.add("3"); } } As you can see, the example waits two seconds to add a new element after clearing all the buffer elements. The call to remove blocks approximately two seconds until AddElementThread adds the element. Although this example doesn’t show it, because we used a PriorityBuffer, the elements are removed based on String comparison. Thus, element 1 is removed before element 2, because 1 has a lower order than 2, even though 1 is added after 2. You can reverse this sort order by using the constructor PriorityBuffer(boolean ascendingOrder) and setting the flag ascendingOrder to false. You can also use your own Comparator by using the constructor PriorityBuffer(boolean ascendingOrder, Comparator comparator) and still change the ascendingOrder flag. In the next section, we’ll review the Comparator implementations available in the Collections API. 7.2.8 Chaining and more with Comparators We’ve already discussed some of the utility classes of the Collections API when we talked about the functor and decorator classes. Although comparator classes aren’t strictly for utility only, they do work around the collection types. A comparator, as defined by the Java Core API interface, helps determine the sort order of objects that implement it. This determination is done by implementing the methods int compare(Object o1, Add two elements to PriorityBuffer Create BlockingBuffer decorator Remove all elements Create and start thread Call is blocked for 2 seconds Thread adds element to buffer after 2 seconds Licensed to Tricia Fu 28 JAKARTA COMMONS ONLINE BOOKSHELF Object o2) and boolean equals(Object). The Collections API defines several general-purpose comparators that can be used for sorting in your own applications: ƒ BooleanComparator—Sorts boolean objects. Either all true objects are sorted first or all false objects are sorted first. ƒ ComparableComparator—Sorts objects that implement the Comparable interface. ƒ ComparatorChain—Uses a sequence of comparators to produce a sorting result. ƒ FixedOrderComparator—Lists a fixed set of objects in a fixed order. ƒ NullComparator—Compares objects to Null, and returns them either higher or lower than that. ƒ ReverseComparator—Reverses the objects to compare to arrive at a result. ƒ TransformingComparator—Uses a transformer to transform objects before comparing them. Listing 7.20 shows an example of using the BooleanComparator and the FixedOrderComparator classes from this package. Note that it uses the static utility class ComparatorUtils to create the BooleanComparator. This class is found in the main org.apache.commons.collections package and is a handy and central way to create Comparator instances. You can choose to create these Comparator instances yourself as well, by calling their constructors directly (for example, the FixedOrderComparator is created this way in the listing). Listing 7.20 Using the comparators package com.manning.commons.chapter07; import org.apache.commons.collections.ComparatorUtils; import org.apache.commons.collections.comparators.BooleanComparator; import org.apache.commons.collections.comparators.FixedOrderComparator; import java.util.Arrays; import java.util.Comparator; public class ComparatorExampleV1 { public static void main(String args[]) { ComparatorExampleV1 example = new ComparatorExampleV1(); example.createComparators(); Arrays.sort(boolParams, boolComp); example.printArray(boolParams); Arrays.sort(stringParams); example.printArray(stringParams); Arrays.sort(stringParams, fixedComp); example.printArray(stringParams); } private void createComparators() { boolComp = ComparatorUtils.booleanComparator(true); fixedComp = new FixedOrderComparator(stringParams); Put all true elements together first Sort alphabetically Sort by country size (FixedOrderComparator can’t be modified now) Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 29 } private void printArray(Object[] array) { for(int i = 0; i < array.length; i++) System.err.println(array[i]); } private static Comparator boolComp; private static Comparator fixedComp; private static Boolean boolParams[] = {new Boolean(true), new Boolean(true), new Boolean(false), new Boolean(false)}; private static String stringParams[] = {"Russia", "Canada", "USA", "Australia", "India"}; } ComparatorChain is an interesting class that chains the result of comparisons from its constituent Comparator instances. Comparators are quizzed one by one on the result of comparison between two elements until a nonzero result is reached, at which point the result is returned and no further pass is made. This is a very useful tool, which can be used to sort data that requires more than one comparison for effective ordering. For example, consider the following data, which shows the names of children and the number of cookies each child ate at different times, separated by a hyphen (-): Shashwat-4, Sarthak-8, Kartik-11, Kartik-14, Waris-14, Shashwat-20, Waris-5 Thus, Waris ate 14 cookies one time and 5 cookies the next. If we wanted to sort this data based on each child’s name and then by the number of cookies they ate, we would need two comparators: one for the name part and the other for the number part. Further, the number comparator must only act after the name comparator has done its work. This is where the ComparatorChain comes in. Listing 7.21 shows how to make this chain comparison work. Listing 7.21 Using two comparators in a chain package com.manning.commons.chapter07; import org.apache.commons.collections.comparators.ComparatorChain; import java.util.Arrays; import java.util.Comparator; public class ComparatorExampleV2 { public static void main(String args[]) { prepareData(); ComparatorChain chain = new ComparatorChain(); chain.addComparator(new NameComparator()); chain.addComparator(new NumberComparator()); Create ComparatorChain using two comparators Licensed to Tricia Fu 30 JAKARTA COMMONS ONLINE BOOKSHELF printArray(dataArray); Arrays.sort(dataArray, chain); printArray(dataArray); } private static void prepareData() { dataArray[0] = "Shashwat-4"; dataArray[1] = "Sarthak-8"; dataArray[2] = "Kartik-11"; dataArray[3] = "Kartik-14"; dataArray[4] = "Waris-14"; dataArray[5] = "Shashwat-20"; dataArray[6] = "Waris-5"; } private static void printArray(String[] array) { System.err.println("---- Elements in Array ---- "); for(int i = 0; i < array.length; i++) { System.err.print(array[i] + ", "); } System.err.println(""); } private static String[] dataArray = new String[7]; } class NameComparator implements Comparator { public int compare(Object o1, Object o2) { if(o1 instanceof String && o2 instanceof String) { String s1 = (String)o1; String s2 = (String)o2; s1 = s1.substring(0, s1.indexOf("-")); s2 = s2.substring(0, s2.indexOf("-")); return s1.compareTo(s2); } return 0; } } class NumberComparator implements Comparator { public int compare(Object o1, Object o2) { if(o1 instanceof String && o2 instanceof String) { String s1 = (String)o1; String s2 = (String)o2; Integer i1 = new Integer(s1.substring(s1.indexOf("-"), s1.length())); Integer i2 = new Integer(s2.substring(s2.indexOf("-"), s2.length())); return i1.compareTo(i2); Sort data using chain Create NameComparator Only compare name part Create NumberComparator Only compare number part Licensed to Tricia Fu MODULE 7: ENHANCING JAVA CORE LIBRARIES WITH COLLECTIONS 31 } return 0; } } This sort of chained comparison is very useful in processing data from a database, when data is retrieved in different columns and you want to sort the data based on each column. You can also reverse the order of sorting for any one comparator (descending or ascending) by setting a flag in the method addComparator(Comparator comp, boolean order). This brings us to the end of our discussion of the Collections component. We’ll continue with the discussion of the BeanUtils and Lang components in the next module. 7.3 Summary This module started by talking about the Commons components that enhance the Java core libraries. We’ve just studied the Collections component, which enhances the libraries of the Collections framework in Java. The Collections component provides a variety of new collection types that aren’t found in the core libraries. Some of these types are Bag, Buffer, and BidiMap. It also provides implementations of the core library collection types that are radically different from the defaults. It makes it very easy to use both the new and enhanced types by providing utility classes to create them. The Collections component also introduces several utility functionalities built around these types. For example, decorators and closures, types of functional objects (functors), are useful for providing value-added benefits to these types while maintaining data integrity. This module was just one part of our discussion of the core Java library enhancements. We continue our coverage of these enhancements in the next module, where we discuss the BeanUtils and Lang components. Licensed to Tricia Fu 32 JAKARTA COMMONS ONLINE BOOKSHELF Index AbstractCollectionDecorator, 5 Adelson-Velskii and Landis. See TreeList ArrayStack, 26 BeanUtils component example, 1 Closure Closure functors, 14 using Closure functors with Collections, 14 Collections Bag examples, 9 interfaces and classes, 9 relationship between Bag classes, 9 unique set of elements, 8 Bag interface, 8 bidimap decorating bidimap, 17 bidimap classes, 16 Blocking Decorator, 26 chaining Comparators, 29 decorating a map with a set, 23 enhancing the List package, 21 List using the List classes, 22 list of decorators, 5 Map enhancements, 16 MultiKey using MultiKey, 18 Set decorators of the Set package, 23 using a CompositeSet, 24 using a map-backed set, 23 using Buffers, 26 using Comparators with Collections API, 27 using maps and decorators, 20 using multiple keys for one collection, 18 using the Transformer functors, 12 Collections component example, 1 Decorator pattern, 4 decorators in Collections component, 5 in action, 4 Factory Factory functors, 14 using the Factory functor, 15 functors, 6 in Collections package, 6 Java Collections API classes, 4 Java reflection, 3 Lang component example, 1 Predicate, 6 decorating a Predicate functor, 7 types of Predicate functors, 6 using Predicate functor classes, 7 PredicatedCollection, 6 PriorityBuffer, 26, 27 Transformers example of using transformers, 12 Transformer functors, 11 types of Transformer functors, 11 using transformers with Collections API, 12 TreeList, 22 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Enhancing Java core libraries with BeanUtils and Lang Vikram Goyal MODULE MANNING 8 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 8 Enhancing Java core libraries with BeanUtils and Lang 8.1 BeanUtils: JavaBeans made easy.............................................................................................................................. 1 8.2 Mind your Lang(uage)............................................................................................................................................ 22 8.3 Summary................................................................................................................................................................. 36 Index ............................................................................................................................................................................. 37 This module continues where the previous one ended, by continuing to discuss the Commons components that enhance the Java core libraries. In the previous module, we discussed the Collections component. In this module, we’ll look at the BeanUtils component, which is used to supplement the Java Reflection and Introspection API; and Lang, which provides helper classes for the Java Lang package. Let’s start with BeanUtils. We’ll take a quick tour of its API and packaging and then delve into some examples. 8.1 BeanUtils: JavaBeans made easy Although BeanUtils is really a supplement to the Java Reflection and Introspection API, its main functionality is directed toward the ability to read and write JavaBeans without needing to know the name and signature of their properties or methods. BeanUtils (short for Bean Introspection Utilities) relies on the JavaBeans that it works on to follow the standard JavaBean specification. The specification puts some restrictions on the structure of JavaBeans. However, BeanUtils is lenient in terms of these restrictions and only requires the following two to be enforced: ƒ A JavaBean must have a no-argument constructor, and its class must be public. ƒ The properties of a JavaBean must follow a standard pattern. This pattern ensures that a property can be accessed by using the method getXXX (or isXXX for boolean properties), where XXX is the property name. Similarly, properties can be modified by using the method setXXX. These methods are typically called the accessor and mutator methods, respectively. Note that there must be only one accessor or mutator for each property; overloaded methods will fail to work with BeanUtils. The current version of BeanUtils is 1.7.0. With this release, BeanUtils has been separated into two libraries. The first library, commons-beanutils-core.jar, contains the core BeanUtils classes. The second library, commons-beanutils-bean-collections.jar, is an add-on that allows you to use BeanUtils with the Collections component. If you don’t plan to use BeanUtils with the Collections API, it’s recommended that you use the core library, because it removes the dependency on the Collections component. In this module we’ll use both libraries, using the combined jar file commons-beanutils.jar. This means that the Collections components jar file must also be in your CLASSPATH in order for any of the Collection-BeanUtils examples to work. Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF Before we look at any examples, you must first understand the terminology associated with JavaBeans and BeanUtils. 8.1.1 Understanding the terminology Simple. Scalar. Indexed. Nested. Mapped. DynaBean. Lazy DynaBean. If these terms leave you confused in relation to JavaBeans and BeanUtils, then this section will give you a quick overview of what they all mean. It’s important that you have a clear understanding of these terms, because we’ll use them a lot later in this module. Therefore, with the help of an example, we’ll explain them here. If you already understand these terms, then it’s safe to skip this section, keeping in mind that this section also introduces the JavaBeans we’ll work with while explaining the BeanUtils examples. Property types JavaBean properties can be divided into four types, based on what they represent. These four types are scalar (or simple), indexed, mapped, and nested. Before we describe these types, look at listings 8.1, 8.2, and 8.3, which show three example JavaBeans. Listing 8.1 Going to the movies with the Movie JavaBean package com.manning.commons.chapter08; import java.util.Map; import java.util.List; import java.util.Date; public class Movie { public Movie() { } public Date getDateOfRelease() { return this.dateOfRelease; } public void setDateOfRelease(Date dateOfRelease) { this.dateOfRelease = dateOfRelease; } public String getTitle() { return this.title; } public void setTitle(String title) {this.title = title; } public Person getDirector() { return this.director; } public void setDirector(Person director) { this.director = director; } public List getActors() { return this.actors; } public void setActors(List actors) { this.actors= actors; } public String[] getKeywords() { return this.keywords; } public void setKeyWords(String[] keywords) { this.keywords = keywords; } public Map getGenre() { return this.genre; } public void setGenre(Map genre) { this.genre = genre; } private Date dateOfRelease; private String title; private Person director; Licensed to Tricia Fu 3MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 3 private List actors; private String[] keywords; private Map genre; } No movie business is complete without its actors; listing 8.2 shows the Actor JavaBean. Listing 8.2 Actor JavaBean package com.manning.commons.chapter08; import java.util.List; public class Actor extends Person { public Actor() { } public List getMovieCredits() { return this.movieCredits; } public void setMovieCredits(List movieCredits) { this.movieCredits = movieCredits; } public long getWorth() { return this.worth; } public void setWorth(long worth) { this.worth = worth; } private List movieCredits; private long worth; } Of course, all actors are people. Therefore, they must extend from the Person JavaBean class, shown in listing 8.3. Listing 8.3 All Actors are Persons first package com.manning.commons.chapter08; import java.util.Map; public class Person { public Person() { } public String getName() { return this.name == null ? "NoName" : this.name; } public void setName(String name) { this.name = name; } public int getGender() { return this.gender; } public void setGender(int gender) { this.gender = (gender > 2 || gender < 0) ? 0 : gender; } Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF public Map getContactNumber() { return this.contactNumber; } public void setContactNumber(Map contactNumber) { this.contactNumber = contactNumber; } private String name; private int gender; private Map contactNumber; } These three code listings set the case for establishing a movie management system. (OK, maybe this isn’t something you can sell, but it’s a start.) These three JavaBeans tell us that a Movie has Actor objects, which are derived from the Person superclass. Each JavaBean has its own properties, and each property provides something different. Properties that provide a single direct value, whether a Java language primitive or an object, are referred to as scalar (or simple) properties. For example, title and dateOfRelease are scalar properties of the Movie JavaBean, whereas name and gender are scalar properties of the Person JavaBean. In the rest of this module, we’ll refer to these properties as scalar (not simple). Properties that use a List or Array implementation to store objects are referred to as indexed properties. These properties allow access to their individual objects via an integer-based index, and therefore they are called indexed properties Note that this index starts at 0. The movieCredits property of an Actor JavaBean, and the actors and keywords properties of the Movie bean, are indexed properties. Similar to indexed properties, a mapped property uses a map to store objects. Individual objects of the map are accessed via a unique key. For example, genre in Movie and contactNumber in Person are mapped properties. A nested property is one that isn’t a direct property of the JavaBean that is being worked on but is a (scalar, indexed, or mapped) property of another bean that is itself a property of the first JavaBean. For example, when talking about the Movie JavaBean, the gender property of the director is a nested property for Movie. As you can see, a nested property is valid only for related JavaBeans and only in context. Therefore, even though gender is a scalar property of a Person, it becomes a nested property for the Movie JavaBean. Table 8.1 divides the various properties of our JavaBeans into their respective types. Table 8.1 Property types of our JavaBeans Property type Properties Scalar Movie.director, Movie.title, Movie.dateOfRelease, Actor.worth, Person.name, Person.gender Indexed Movie.actors, Movie.keywords, Actor.movieCredits Mapped Movie.genre, Person.contactNumber Nested Movie.director.name, Movie.director.gender, Movie.director.contactNumber, Movie.actors[n].name, Movie.actors[n].gender, Movie.actors[n].contactNumber, Movie.actors[n].worth, Movie.actors[n].movieCredits Notice the way the properties are qualified with the name of the JavaBean using a period (.). Especially notice the nested properties syntax for indexed properties: Movie.actors[n].name refers to the name property of the nth Actor of the Movie and is a indexed nested property. We can similarly retrieve mapped nested properties using the key for each object in the map. The reverse of this scenario is true as well: For example, Movie.director.contactNumber is a nested scalar property (which returns the map 1= male, 2 = female, 0 = indeterminate Licensed to Tricia Fu 5MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 5 containing all the contact numbers), but Movie.director.contactNumber(key) (which returns a specific contact number) is a nested mapped property. Dynamic JavaBeans (DynaBeans) Dynamic JavaBeans (DynaBeans) represent a JavaBean that hasn’t been created separately in a class of its own but has been put together in code to represent one. There are several advantages to using a DynaBean: ƒ DynaBeans don’t require a dedicated class of their own. They’re created in code and therefore don’t require any compile-time information (except of course, the BeanUtils package). ƒ DynaBeans use generic methods to access or modify a property. The methods typically follow the pattern get(propertyName) and set(propertyName, value). ƒ Because no compile-time information is required for these classes, they enable runtime querying, which allows information about them to be generated dynamically. Of course, there is a penalty to be paid for this convenience. Because these beans are created dynamically, they’re slower to access than normal JavaBeans. Further, errors in accessing property names may not be visible until runtime, which may be a little late in some cases. Keeping these advantages and disadvantages in mind, let’s look at the support for DynaBeans in the BeanUtils package. The basic DynaBean interface is supported by the DynaClass interface, in much the same way that a JavaBean is supported by the java.lang.Class class. However, note that BeanUtils handles JavaBeans and DynaBeans similarly, and almost no change is required to switch from one to the other. DynaBean and DynaClass define the interfaces for dynamic beans, and the BeanUtils package provides several implementations to get you started. Nothing is stopping you from creating your own implementations of these classes; but given the choice, there should be no need. These implementations are as follows: ƒ BasicDynaBean, BasicDynaClass—The base implementations that work together to provide minimal functionality. ƒ LazyDynaBean, LazyDynaClass—Implementations that add a property to the underlying DynaClass when it’s requested by calling the set method, and if it doesn’t already exist. ƒ ResultSetDynaClass—Wrapper DynaClass implementation for java.sql.Row, thereby representing a row of a resultset. The underlying database connection isn’t closed until you retrieve the complete data. ƒ RowSetDynaClass—Similar to ResultSetDynaClass, but creates an in-memory representation of the each row, thereby allowing the underlying database connection to be closed. Also known as a disconnected resultset. ƒ WrapDynaBean, WrapDynaClass—Implementations that are used to wrap around existing JavaBean instances to make them work as DynaBeans. You’ll see detailed examples of these implementations in the examples that follow. Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF 8.1.2 BeanUtils in action BeanUtils provides several ways in which properties of JavaBeans can be accessed or manipulated. Four classes can be directly used for this purpose: BeanUtils, BeanUtilsBean, PropertyUtils, and PropertyUtilsBean. What is the difference between these classes, and when should each be used? Any class in the BeanUtils package that ends with Utils is a wrapper class around the default implementation of that class. Therefore, the BeanUtils class contains static wrapper methods that direct their calls to a default instance of BeanUtilsBean class, which provides actual implementation. Similarly, PropertyUtils provides static wrapper methods for the PropertyUtilsBean class, using a default instance of that class. It’s useful to use the Utils classes, if you’re happy with the default behavior that they provide. For most purposes, the default behavior is the required behavior, and we’ll stick with using the Utils classes. However later in this module you’ll see an example in which we provide our own instance of the BeanUtilsBean class. Now that we have determined the difference between the Utils classes and their implementations classes, a question still remains: What is the difference between the BeanUtils and PropertyUtils classes, and which one should you use for managing JavaBean properties? If you look at the APIs for both classes (or the APIs for the corresponding BeanUtilsBean and PropertyUtilsBean), you’ll notice that almost all the methods signatures look alike and seem to do the same thing! For example, consider the getProperty(Object bean, String name) method in both classes. It has the same signature, except for the return type: BeanUtils returns a String value, and PropertyUtils returns an Object. And herein lies the difference. BeanUtils uses a built-in converter to convert non-String properties to their equivalent String value, whereas PropertyUtils returns these properties as is. These conversions for BeanUtils are performed by the ConvertUtils class (actually, by its ConvertUtilsBean implementation); you can change this default behavior of BeanUtils by switching to BeanUtilsBean and providing a custom conversion utility (discussed earlier). To summarize, here are the rules for using the BeanUtils classes: ƒ Use BeanUtils when the default conversions of properties to String values are acceptable. ƒ Use PropertyUtils when no property conversions are required. ƒ Use BeanUtilsBean when the default conversions of properties to String values are not acceptable and you want to provide your own conversion routines. ƒ Use PropertyUtilsBean when you don’t want to use the static methods of PropertyUtils. Note that the same rules apply when you’re trying to set the values of properties, not just when you’re retrieving them. BeanUtils uses a converter to convert values to their respective types, but PropertyUtils doesn’t. Also, as you can see from the list, there is no difference between using PropertyUtilsBean and PropertyUtils except that using the latter doesn’t require the creation of an instance on your part. In the examples that follow, we’ll use the Utils class. Property access Let’s get up to speed with using BeanUtils by using it to access the properties of some JavaBeans. In this section, we’ll use BeanUtils to access the properties of a Movie instance (created in section 8.1.1). This instance is created in listing 8.4 in a code snippet of its own. Licensed to Tricia Fu 7MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 7 Listing 8.4 Creating a Movie instance for The Italian Job private Movie prepareData() { Movie movie = new Movie(); movie.setTitle("The Italian Job"); movie.setDateOfRelease(new GregorianCalendar(1969, 0, 1).getTime()); Map genre_map = new HashMap(); genre_map.put("THR", "Thriller"); genre_map.put("ACT", "Action"); movie.setGenre(genre_map); Person director = new Person(); director.setName("Peter Collinson"); director.setGender(1); Map director_contacts = new HashMap(); director_contacts.put("Home", "99922233"); director_contacts.put("Mobile", "0343343433"); director.setContactNumber(director_contacts); movie.setDirector(director); Actor actor1 = new Actor(); actor1.setName("Michael Caine"); actor1.setGender(1); actor1.setWorth(10000000); List actor1_movies = new ArrayList(); Movie movie2 = new Movie(); movie2.setTitle("The Fourth Protocol"); Movie movie3 = new Movie(); movie3.setTitle("Shiner"); actor1_movies.add(movie2); actor1_movies.add(movie3); actor1.setMovieCredits(actor1_movies); Actor actor2 = new Actor(); actor2.setName("Margaret Blye"); actor2.setGender(2); actor2.setWorth(20000000); List actors = new ArrayList(); actors.add(actor1); actors.add(actor2); movie.setActors(actors); return movie; } Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF This code shows the prepareData method that is used to create an instance of the Movie object for the movie titled The Italian Job. Almost all properties of the Movie JavaBean are set, along with several properties of the related JavaBeans Actor and Person (Director). Let’s see how to use BeanUtils to access the properties of this JavaBean. Listing 8.5 shows several examples of property access. Listing 8.5 Using BeanUtils to access Movie instance properties package com.manning.commons.chapter08; import org.apache.commons.beanutils.BeanUtils; import java.util.Map; import java.util.Date; import java.util.List; import java.util.HashMap; import java.util.ArrayList; import java.util.GregorianCalendar; public class BeanUtilsExampleV1 { public static void main(String args[]) throws Exception { BeanUtilsExampleV1 diff = new BeanUtilsExampleV1(); Movie movieBean = diff.prepareData(); System.err.println("Movie Title: " + BeanUtils.getProperty(movieBean, "title")); System.err.println("Movie Year: " + BeanUtils.getProperty(movieBean, "dateOfRelease")); System.err.println("Movie Director: " + BeanUtils.getProperty(movieBean, "director.name")); System.err.println("Movie Director Home Contact: " + BeanUtils.getProperty(movieBean, "director.contactNumber(Home)")); System.err.println("Movie Genre (Thriller): " + BeanUtils.getProperty(movieBean, "genre(THR)")); System.err.println("Movie Actor 1 name: " + BeanUtils.getProperty(movieBean, "actors[0].name")); System.err.println("Movie Actor 1 worth: " + BeanUtils.getProperty(movieBean, "actors[0].worth")); System.err.println("Movie Actor 1 other movie 1: " + BeanUtils.getProperty(movieBean, "actors[0].movieCredits[0].title")); } } This listing shows most ways of accessing properties via BeanUtils. All types of property access are covered, including scalar, indexed, mapped, nested, and a mix of these. As you can see, dynamically retrieving the property of a JavaBean requires a reference to the bean and the name of the property being accessed; then you can use BeanUtils’s static method getProperty() method to get the value. There are several variations on this method; these are listed in table 8.2 with examples. Scalar Nested-scalar Nested-mapped Mapped Indexed-nested-scalar Indexed-nested- indexed-scalar Licensed to Tricia Fu 9MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 9 Table 8.2 Variations of the getProperty method Method variation Example getIndexedProperty(Object bean, String name, int index) BeanUtils.getIndexedProperty(movieBean, "actors", 1) Returns Actor object at index 1 getMappedProperty(Object bean, String name, String key) BeanUtils.getMappedProperty(movieBean, "genre", "ACT") Returns the Action genre getArrayProperty(Object bean, String name) BeanUtils.getArrayProperty(movieBean, "keywords") Returns null (because the keywords property isn’t set) getNestedProperty(Object bean, String name) BeanUtils.getNestedProperty(movieBean, "director.name") Returns the name of the director getSimpleProperty(Object bean, String name) BeanUtils.getSimpleProperty(movieBean, "title") Returns the title of the movie It’s probably best to use the getProperty method, because it encapsulates all the other methods; therefore you don’t have to remember the syntax for the other methods. Besides, the results are same in both cases, and the getProperty method is much cleaner to use. If you forget to use the right syntax with the right method, you’ll get a runtime exception. For example, if you use the method getSimpleProperty with a nested property, as in getSimpleProperty(movieBean, "director.name"), a runtime exception will be thrown. Of course, retrieving property values is only part of the BeanUtils story. The other part deals with dynamically setting the value of these properties. Changing properties There are three ways you can change the properties of a JavaBean using BeanUtils. The first way involves setting the individual properties using either the setProperty or the copyProperty method. The second way uses an existing JavaBean to copy the properties and their values, using either the cloneBean or copyProperties method. The final way is to use a map of name-value pairs with the populate method to set the property values. Let’s look at an example of each of these methods. Listing 8.6 uses the Movie JavaBean defined in listing 8.4 to create a new Movie bean and then change some of its properties. Listing 8.6 Creating a new Movie bean package com.manning.commons.chapter08; import org.apache.commons.beanutils.BeanUtils; import java.util.Map; import java.util.Date; import java.util.List; import java.util.HashMap; import java.util.ArrayList; import java.util.GregorianCalendar; public class BeanUtilsExampleV2 { public static void main(String args[]) throws Exception { BeanUtilsExampleV2 diff = new BeanUtilsExampleV2(); Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF Movie movieBean = diff.prepareData(); Movie newMovieBean = new Movie(); BeanUtils.copyProperties(newMovieBean, movieBean); BeanUtils.setProperty(newMovieBean, "title", "Quills"); BeanUtils.setProperty( newMovieBean, "dateOfRelease", new GregorianCalendar(2000, 0, 1).getTime()); BeanUtils.setProperty(newMovieBean, "director.name", "Philip Kaufman"); BeanUtils.setProperty( newMovieBean, "director.contactNumber(Home)", "3349084333"); System.err.println(BeanUtils.getProperty(newMovieBean, "title")); System.err.println(BeanUtils.getProperty(newMovieBean, "director.name")); System.err.println(BeanUtils.getProperty( newMovieBean, "director.contactNumber(Home)")); } } We first use the copyProperties method to copy all the properties of the movieBean to newMovieBean, effectively creating a clone. The same effect can also be achieved using the cloneBean method, except that if cloneBean is used, an existing object isn’t required—it creates one for you. For example, Movie newMovieBean = (Movie)cloneBean(movieBean) creates a new JavaBean. The copyProperties method tries to match the properties between the source and destination beans. However, if it doesn’t find a corresponding property on the destination bean, it doesn’t throw an exception but continues trying to find corresponding methods for the rest of the properties. The third method, populate, is useful for copying properties from a map, which contains the property names mapped to their values, to a JavaBean. This is shown in listing 8.7, where we first use the describe method to get these properties in a map and then use this map to create a new Actor object. Listing 8.7 Using the populate and describe methods package com.manning.commons.chapter08; import org.apache.commons.beanutils.BeanUtils; import java.util.Map; import java.util.Date; import java.util.List; import java.util.HashMap; import java.util.ArrayList; import java.util.GregorianCalendar; public class BeanUtilsExampleV3 { Copy all property values from old to new bean Change title Change date Change nested property Change nested-mapped property Licensed to Tricia Fu 11MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 11 Side Effects of Using CloneBean Be aware of the effect of using copyProperties and cloneBean and subsequent property changes on the two JavaBeans. When properties are copied, a copy of the primitive properties is made, and that makes them safe from modification. However, object properties aren’t copied themselves: Only their reference is copied (this is true for cloneProperties but may or may not be true for copyProperties, depending on how many of the original JavaBean object properties were set before and have their own references to these objects). Thus changes made to the object properties, from either the old JavaBean or the new, will be reflected in the other bean. This may or may not be what you desire. Figure 8.1 shows this conundrum. Figure 8.1 Cloning and its effects public static void main(String args[]) throws Exception { BeanUtilsExampleV3 diff = new BeanUtilsExampleV3(); Actor actor = diff.prepareData(); Map describedData = BeanUtils.describe(actor); System.err.println(describedData.get("name")); Describe Actor properties and values to map Check whether data is correct Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF describedData.put("name", "Robert Redford"); Actor newActor = new Actor(); BeanUtils.populate(newActor, describedData); System.err.println(BeanUtils.getProperty(newActor, "name")); } private Actor prepareData() { Actor actor = new Actor(); actor.setName("Michael Caine"); actor.setGender(1); actor.setWorth(10000000); return actor; } } One thing you need to remember while using the describe and populate methods is that they only work with limited property types: String, boolean, int, long, float, and double, along with their corresponding array types. That is why we choose to do this example with the Actor bean rather than the Movie bean, which contains the Date type. Converters and locales In the rare case when the default conversion of properties from type-specific information to equivalent String values (and vice versa) provided by BeanUtils isn’t good enough for your application, you can write your own converters and register them with the ConvertUtils class. You’ll learn how to do this in this section. You’ll also see how to use LocaleBeanUtils, which is a specialized BeanUtils class that formats properties in a locale-dependent manner. Listing 8.8 shows an example of creating and using a String and Long converter. Listing 8.8 Creating and using a String and Long converter package com.manning.commons.chapter08; import org.apache.commons.beanutils.Converter; import org.apache.commons.beanutils.PropertyUtils; import org.apache.commons.beanutils.BeanUtilsBean; import org.apache.commons.beanutils.ConvertUtilsBean; import org.apache.commons.beanutils.PropertyUtilsBean; import org.apache.commons.beanutils.ConversionException; import java.util.Map; public class BeanUtilsExampleV4 { public static void main(String args[]) throws Exception { BeanUtilsExampleV4 diff = new BeanUtilsExampleV4(); Actor actor = diff.prepareData(); Change map value of name key Populate new Actor with described and modified data Licensed to Tricia Fu 13MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 13 ConvertUtilsBean convertUtilsBean = new ConvertUtilsBean(); convertUtilsBean.deregister(String.class); convertUtilsBean.register(new MyStringConverter(), String.class); convertUtilsBean.deregister(Long.class); convertUtilsBean.register(new MyLongConverter(), Long.class); convertUtilsBean.register(new MyLongConverter(), Long.TYPE); BeanUtilsBean beanUtilsBean = new BeanUtilsBean(convertUtilsBean, new PropertyUtilsBean()); System.err.println("==== Values before calling describe ==== "); System.err.println("By PropertyUtils: " + PropertyUtils.getProperty(actor, "name")); System.err.println("By BeanUtils: " + beanUtilsBean.getProperty(actor, "name")); System.err.println(beanUtilsBean.getProperty(actor, "worth")); Map describedData = beanUtilsBean.describe(actor); System.err.println("==== Values in Map ==== "); System.err.println(describedData.get("name")); System.err.println(describedData.get("worth")); Actor newActor = new Actor(); beanUtilsBean.populate(newActor, describedData); System.err.println("==== Values after calling populate ==== "); System.err.println(beanUtilsBean.getProperty(newActor, "name")); System.err.println(beanUtilsBean.getProperty(newActor, "worth")); } private Actor prepareData() { Actor actor = new Actor(); actor.setName("Michael Caine"); actor.setGender(1); actor.setWorth(10000000); return actor; } } class MyStringConverter implements Converter { public Object convert(Class type, Object value) { if(value == null) { return (String)null; } else { return (value.toString().replaceAll("\\s", "")); } } } class MyLongConverter implements Converter { Create custom ConvertUtilsBean Deregister previous converters; register new ones Create customized BeanUtilsBean instance PropertyUtils doesn’t use converters Create StringConverter Remove all whitespace Create LongConverter Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF private Object defaultValue; private boolean useDefault; public MyLongConverter() { this(true, new Long(0)); } public MyLongConverter(boolean useDefault, Object defaultValue) { this.useDefault = useDefault; this.defaultValue = defaultValue; } public Object convert(Class type, Object value) { if(value == null) { if(useDefault) { return defaultValue; } else { throw new ConversionException("No default value specified"); } } if(value instanceof Long) { return new Long(((Long)value).longValue() + 1000); } else { try { return new Long(new Long(value.toString()).longValue() + 1000); } catch (Exception e) { System.err.println(e); if(useDefault) { return defaultValue; } else { throw new ConversionException(e); } } } } } The first step in using your own converters is to create those converters. In this code, we create two converters. The first, MyStringConverter, removes all whitespace from the String values. The second, MyLongConverter, increments any Long value by 1000. Next, these converters must be registered with the BeanUtilsBean instance. Recall that we can’t use BeanUtils now, because it uses the default ConvertUtilsBean converters, and we want it to use our converters for String and Long (and long) properties. Thus, a ConvertUtilsBean instance is created; then the previous converters are deregistered and the new ones are registered, using the deregister and register methods, respectively. Notice that both the primitive type (long) and the class for Long must be registered. With this new ConvertUtilsBean instance prepared, we now create the required customized BeanUtilsBean instance. The rest of the listing exercises this instance by using it to describe an Actor instance, which would call only the MyStringConverter, and then populating a new Actor instance that Set defaultValue of 0 Increment all values by 1000 Licensed to Tricia Fu 15MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 15 will call both MyStringConverter (for the name property) and MyLongConverter (for the worth property). We point out in the code that PropertyUtils (and PropertyUtilsBean) doesn’t use converters while querying property values. As you can see if you run this listing, it prints the unmodified value of the Actor name property, whereas using the customized BeanUtilsBean will print the modified (all spaces removed) value. Before we move on, let’s look at the locale-specific BeanUtils implementation, LocaleBeanUtils, and its corresponding LocaleBeanUtilsBean; they’re placed in a package of their own, org.apache.commons.beanutils.locale. These classes are useful for formatting results in a locale- sensitive manner. For example, suppose you want your date properties to have formatting applied to them based on the locale in which you’re running your application, rather than having to format the results yourself. To do this, instead of using BeanUtils, you use the counterpart LocaleBeanUtils class. Listing 8.9 shows an example. Listing 8.9 Locale-dependent date formatting package com.manning.commons.chapter08; import org.apache.commons.beanutils.locale.LocaleBeanUtils; import java.util.Date; import java.util.GregorianCalendar; public class BeanUtilsExampleV5 { public static void main(String args[]) throws Exception { BeanUtilsExampleV5 diff = new BeanUtilsExampleV5(); Movie movie = diff.prepareData(); System.err.println( LocaleBeanUtils.getProperty(movie, "dateOfRelease", "dd-MMM-yyyy")); } private Movie prepareData() { Movie movie = new Movie(); movie.setTitle("The Italian Job"); movie.setDateOfRelease(new GregorianCalendar(2000, 0, 1).getTime()); return movie; } } To get the date printed in the format dd-MMM-yyyy for the default locale, all you have to do is specify the format to the getProperty method of LocaleBeanUtils class. This class supports all the methods of the BeanUtils class and retrofits them to allow the pattern format to be specified. Corresponding to LocaleBeanUtilsBean is the LocaleConvertUtilsBean class, which can be overridden to provide custom formatting of locale-dependent properties. Print date in default locale formatted in this pattern Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF Collections and BeanUtils1 The BeanUtils package provides a way to utilize some of the features of the Collections component discussed in the previous module. For example, it provides a custom Comparator implementation that you can use to compare and sort the value of the property of a JavaBean. Similarly, it provides implementations for the functor classes in the BeanPredicate, BeanPropertyValueChangeClosure, BeanPropertyValueEqualsPredicate, and BeanToPropertyValueTransformer classes. Let’s first look at the BeanComparator class. Strictly speaking, this class isn’t related to the Collections component, because it provides a comparator by implementing the Java Comparator interface. The only reason it’s included in the collection-specific BeanUtils classes is because internally it uses the ComparableComparator from the Collections API. Therefore, the examples in this section require commons-collections.jar to be present in your CLASSPATH. Also, if you’ve been using commons-beanutils- core.jar to run the examples until now, you’ll need to put the commons-beanutils-bean-collections.jar optional library in your CLASSPATH as well. Easier still, use the commons-beanutils.jar file, which combines both of these libraries in one file. Listing 8.10 shows an example of using the BeanComparator to sort an array that contains several Actor JavaBeans. The sort is done on the name property. Listing 8.10 Sorting using the BeanComparator package com.manning.commons.chapter08; import org.apache.commons.beanutils.BeanComparator; import java.util.Arrays; public class BeanUtilsCollectionsV1 { public static void main(String args[]) { BeanComparator comp = new BeanComparator("name"); Actor actor1 = new Actor(); actor1.setName("Michael Caine"); Actor actor2 = new Actor(); actor2.setName("Robert Redford"); Actor actor3 = new Actor(); actor3.setName("Harrison Ford"); Actor[] actors = {actor1, actor2, actor3}; Arrays.sort(actors, comp); for(int i=0; i 17MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 17 you can do so by calling the constructor BeanComparator(String propertyName, Comparator comparator). Functors, which we discussed in the last module, are objects that perform some sort of function. We introduced Predicate, Closure, Factory, and Transformer functors. BeanUtils leverages the use of these functors by providing implementations that perform specific functions for JavaBeans. These functors are listed here: ƒ BeanPredicate—Maps a property to a supplied predicate. Whenever the evaluate method on the BeanPredicate is called, the property is retrieved and the supplied predicate’s evaluate method is called on the value, which is then returned. ƒ BeanPropertyValueChangeClosure—A Closure functor that sets the value of a property to a supplied value. ƒ BeanPropertyValueEqualsPredicate—A Predicate functor that evaluates a property value and compares against a supplied value. Returns true or false based on the comparison. ƒ BeanToPropertyValueTransformer—A Transformer functor that only returns the value of a single supplied property when applied on a given object. Let’s see how to use one of these functors. Listing 8.11 uses BeanPredicate to restrict the title property of the Movie JavaBean, using UniquePredicate from the Collections API. Listing 8.11 Using BeanPredicate package com.manning.commons.chapter08; import org.apache.commons.beanutils.BeanPredicate; import org.apache.commons.collections.PredicateUtils; public class BeanUtilsCollectionsV2 { public static void main(String args[]) { BeanPredicate predicate = new BeanPredicate("title", PredicateUtils.uniquePredicate()); Movie movie = new Movie(); movie.setTitle("The Italian Job"); Movie movie1 = new Movie(); movie1.setTitle("The Italian Job"); System.err.println(predicate.evaluate(movie)); System.err.println(predicate.evaluate(movie1)); } } The rest of the functor classes can be similarly used to restrict or evaluate the property values of JavaBeans using BeanUtils. Create BeanPredicate using title property and UniquePredicate Create two movies with same title Evaluates to true Evaluates to false Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF We’ve so far covered most of the functionality available with BeanUtils to manipulate properties for existing JavaBeans. However, you haven’t yet seen how BeanUtils support dynamic beans, although we introduced this topic in section 8.1.1. Let’s now see some examples of using DynaBeans. Working with DynaBeans: BasicDynaClass, ResultSetDynaClass, and RowSetDynaClass Dynamic beans take the drudgery out of defining and creating JavaBean instances. This is especially true where the definition of a dynamic bean can be picked up from an external file, thus removing all the definition logic from application code. Support for DynaBeans is defined in the DynaBean, DynaClass, and DynaProperty interfaces and classes. Let’s start this section with the code example shown in listing 8.12, which creates an instance of our Movie JavaBean (and its support classes) using the DynaBean classes. We’ll use the BasicDynaBean and BasicDynaClass classes as the standard implementations of these interfaces. We’ll also use the DynaProperty class to model the properties. Listing 8.12 Converting a Movie JavaBean to a DynaBean package com.manning.commons.chapter08; import org.apache.commons.beanutils.DynaBean; import org.apache.commons.beanutils.DynaClass; import org.apache.commons.beanutils.BeanUtils; import org.apache.commons.beanutils.LazyDynaBean; import org.apache.commons.beanutils.DynaProperty; import org.apache.commons.beanutils.BasicDynaClass; import java.util.Map; import java.util.List; import java.util.Date; import java.util.HashMap; import java.util.GregorianCalendar; public class DynaBeansExampleV1 { public static void main(String args[]) throws Exception { Object movie = createMovieBean(); System.err.println(BeanUtils.getProperty(movie, "title")); System.err.println(BeanUtils.getProperty(movie, "director.name")); } private static Object createMovieBean() throws Exception { DynaProperty properties[] = new DynaProperty[] { new DynaProperty("title", String.class), new DynaProperty("dateOfRelease", Date.class), new DynaProperty("keywords", String[].class), new DynaProperty("genre", Map.class), new DynaProperty("actors", List.class), new DynaProperty("director", DynaBean.class) }; c Create Dynamic properties Define director as DynaBean f BeanUtils works the same with DynaBeans Licensed to Tricia Fu 19MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 19 DynaClass movieClass = new BasicDynaClass("movie", null, properties); DynaBean movieBean = movieClass.newInstance(); movieBean.set("title", "The Italian Job"); movieBean.set("dateOfRelease", new GregorianCalendar(1969, 0, 1).getTime()); movieBean.set("keywords", new String[] {"Italy", "Bank Robbery"}); Map genre = new HashMap(); genre.put("THR", "Thriller"); movieBean.set("genre", genre); movieBean.set("genre", "ACT", "Action"); DynaBean director = createPersonBean(); director.set("name", "Peter Collinson"); director.set("gender", new Integer(1)); movieBean.set("director", director); return movieBean; } private static DynaBean createPersonBean() { DynaBean person = new LazyDynaBean(); return person; } } c The properties of the DynaClass are defined here by listing them in a DynaProperty array. Each property is defined by two values: the name and the property type. Thus, title is of type String, keywords is type String[], actors is type List, and director is type DynaBean. Why is director of DynaBean type? Shouldn’t it be of type Person? Yes and no. Yes, if we aren’t going to model the Person type using DynaBeans. No, because we’re going to model the director property as a DynaBean (actually as a LazyDynaBean). d The dynamic Movie class is created using the properties defined in c. The class is given the name movie, and its implementation class is set as null. By allowing you to set the implementation class different from itself, the BasicDynaClass can use existing constructors to perform utility functions. However, in this case, we don’t require this functionality, and the implementation class is the movieClass. Now that the blueprint for creating new classes has been set by defining movieClass, we can create instances of it. This is done by calling the newInstance method on movieClass. The properties of this new instance are set using the set(String propertyName, Object propertyValue) method. This method has two variations, which allow the setting of indexed and mapped properties. For example, set("actor", 0, new Actor())sets the value at index 0 to a blank Actor instance. Similarly, as shown in the listing, (movieBean.set("genre", "ACT", "Action")) supplies the key with which the mapped property needs to be set in the map. e The Person DynaBean is created slightly differently. In fact, it’s created as a LazyDynaBean. A DynaBean of this type is created without any underlying properties. Properties are only added to it the d Create dynamic Movie class e Create LazyDynaBean for director Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF first time a property is set on this DynaBean. Thus, calling set("name", "Peter Collinson") on this DynaBean first creates this property on it and then sets it to the desired value. This is extremely useful, as long as you’re confident that all properties of the desired DynaBean will be set before an attempt is made to do a get. f Finally, you don’t have to do anything special to access or modify the properties of a DynaBean using BeanUtils. BeanUtils recognizes that the class is of the DynaBean type and adjusts its dynamic calls accordingly. This makes working with normal JavaBeans and DynaBeans in BeanUtils seamless. After this example of using the basic implementation of the DynaBean and DynaClass, let’s look at some specialized implementations. The ResultSetDynaClass implementation and its counterpart, RowSetDynaClass, can both be used to represent the result of running a query on a database. This is an ideal case for using DynaBeans, because it takes the problem of representing the data from the resultset into a JavaBean suitable for the consuming application, a familiar and time-saving process. You don’t have to write a JavaBean for each possible resultset, and each resultset can be dynamically constructed. Let’s start with ResultSetDynaClass. Conceptually, using the two is same; you execute your query to get the resultset, wrap it in the appropriate class, and then process the results. However, the key difference lies in the fact that a ResultSetDynaClass returns the data row by row while still holding on to the underlying resultset; whereas a RowSetDynaClass wraps the data in memory, which allows you to close and return the connection used to retrieve the resultset before processing the result. We’ll make this difference obvious in the two examples that follow. Listing 8.13 shows an example of using ResultSetDynaClass. Note that it utilizes DBCP for making a connection to the database; therefore, you’ll need the commons-dbcp.jar library in your CLASSPATH. Listing 8.13 Using ResultSetDynaClass package com.manning.commons.chapter08; import org.apache.commons.dbcp.BasicDataSource; import org.apache.commons.beanutils.DynaBean; import org.apache.commons.beanutils.ResultSetDynaClass; import java.util.Iterator; import java.sql.ResultSet; import java.sql.Connection; import java.sql.PreparedStatement; public class DynaBeansExampleV2 { public static void main(String args[]) throws Exception { Connection conn = getConnection(); PreparedStatement ps = conn.prepareStatement( "SELECT * from movie, person " + "WHERE movie.director = person.Id"); ResultSet rs = ps.executeQuery(); ResultSetDynaClass rsdc = new ResultSetDynaClass(rs); Iterator itr = rsdc.iterator(); Iterator to go through resultset Wrap resultset in DynaClass Licensed to Tricia Fu 21MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 21 while(itr.hasNext()) { DynaBean bean = (DynaBean)itr.next(); System.err.println(bean.get("title")); } conn.close(); } private static Connection getConnection() throws Exception { BasicDataSource bds = new BasicDataSource(); bds.setDriverClassName("com.mysql.jdbc.Driver"); bds.setUrl("jdbc:mysql://localhost/commons"); bds.setUsername("root"); bds.setPassword(""); bds.setInitialSize(5); return bds.getConnection(); } } A ResultSetDynaClass holds the resultset returned by executing the query, and each row in the resultset represents a DynaBean. But how is this DynaBean created? Does it use a LazyDynaBean? No, but the DynaClass information for this DynaBean is constructed by querying the resultset for metadata (resultSet.getMetaInformation()) information. This information creates an array of DynaProperties, which form the properties of the DynaClass. However, all this is transparent to the end user; all you have to do to get each DynaBean representing each row, with the column names representing a property each, is traverse through it. A specialized Iterator implementation is provided, called ResultSetIterator. The ResultSetIterator holds a pointer to each row in the resultset and, as requested by the user, returns the DynaBean requested at that particular row. Once you move away from a particular row, you can’t go back to it. This implies that the DynaBean represented by each row can’t be accessed once you’ve moved past it. Worse, once you close the underlying resultset, no DynaBean is accessible, because the resultset is no longer valid. This implies that a ResultSetDynaClass is only useful in situations where you want to work on the data sequentially while holding the underlying connection to the database open. This isn’t the best-case scenario for obvious reasons. The connection remains open throughout the time you’re processing the data. The solution to this problem lies in the RowSetDynaClass. This DynaClass wraps the underlying resultset, queries it to create a DynaClass representation of the column information, traverses it to convert each of its rows to a BasicDynaBean based on this DynaClass, and stores these DynaBean instances in an ArrayList in memory. This allows the underlying connection to be safely closed. Using this class requires only a small change from the ResultSetDynaClass, in addition to the class itself. Listing 8.14 shows an example of using this class; it’s stripped of the side requirements of getting the connection. Listing 8.14 Using RowSetDynaClass package com.manning.commons.chapter08; import org.apache.commons.dbcp.BasicDataSource; import org.apache.commons.beanutils.DynaBean; Retrieve value using get Each item is of type DynaBean Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF import org.apache.commons.beanutils.RowSetDynaClass; import java.util.Iterator; import java.sql.ResultSet; import java.sql.Connection; import java.sql.PreparedStatement; public class DynaBeansExampleV3 { public static void main(String args[]) throws Exception { Connection conn = getConnection(); PreparedStatement ps = conn.prepareStatement( "SELECT * from movie, person " + "WHERE movie.director = person.Id"); ResultSet rs = ps.executeQuery(); RowSetDynaClass rsdc = new RowSetDynaClass(rs); conn.close(); Iterator itr = rsdc.getRows().iterator(); while(itr.hasNext()) { DynaBean bean = (DynaBean)itr.next(); System.err.println(bean.get("title")); } } } There is, of course, a tradeoff in using RowSetDynaClass as opposed to ResultSetDynaClass. Because all the data from the underlying connection is copied into memory, large volumes of data can eat up computer resources. The advantage lies in not tying up the underlying connection, which can be released immediately for other queries. This brings us to the end of our discussion of BeanUtils. We’ve covered dynamic management of JavaBean properties, both normal and dynamic. The next section introduces the Lang component. 8.2 Mind your Lang(uage) The Lang component provides utility functions for working with the Java Lang package, which provides some of the core functionality of the Java language. In this section, we’ll start with a brief description of the classes of the Java Lang package, and we’ll use it as the basis for comparison with the Commons Lang package classes. We’ll then move straight into the section on using the Lang package with a variety of examples. 8.2.1 What’s wrong with the java.lang package? The java.lang package contains the core class of the Java Runtime. For example, it contains definition classes like Object and Class; it also contains Java primitive type wrapper classes like Boolean, Long, and Byte. Further, it provides classes that work with character data: String and StringBuffer. Finally, it includes classes that assist in system operations, like ClassLoader, Process, and Runtime. getRows retrieves ArrayList Connection can now be closed Hold resultset in ArrayList of DynaBeans Licensed to Tricia Fu 23MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 23 This is by no means an exhaustive rendition of the classes in this package. But it gives a fair indication of this package’s importance. With such a star list of classes, this package requires support in terms of utility functions to derive maximum benefit. There is nothing wrong with this package’s classes—nothing needs to be fixed. The classes do the business they’re entrusted with, and they do it well. External libraries provide services beyond those expected of the java.lang package, but they keep the classes of this package as the basis for these services. For example, consider the StringUtils class of the Commons Lang package. One of the methods (static of course) that it provides is indexOf(String str, String searchStr), which returns the index of the (first) occurrence of the search parameter in the supplied String. Of course, the String class provides a similar nonstatic method, indexOf(String searchStr). Why would you use the former over the latter? Is there any value added in using the former? Yes, because of the way the former handles null input. All methods of the StringUtils class handle null input gracefully. This means that if you by mistake pass a null value to these methods, StringUtils will return –1, which means it didn’t find any occurrence; whereas the normal String method will throw a NullPointerException. Needless to say, a runtime NullPointerException can bring your application to its knees, while a graceful exit can leave the application functioning normally. (Note that the –1 response is technically not incorrect; therefore, application integrity is maintained.) Once again, we must emphasize that the String class does its business as it’s supposed to. It throws a NullPointerException to indicate incorrect data. You can handle it either by programming defensively, to ensure that a null value is never passed into this method, or by using StringUtils. Using StringUtils—in fact, using any of the classes of the Lang package—is tantamount to programming defensively with the classes of the Java Lang package. Having established the need for the Lang component classes, let’s look at this component’s API. The package structure is as follows: ƒ org.apache.commons.lang—Contains classes that provide static utility functions for the classes of the java.lang package ƒ org.apache.commons.lang.builder—Contains classes that help in the creation of equals, hashCode, toString, and compareTo methods ƒ org.apache.commons.lang.enum—Provides an implementation that can help in the creation of enums ƒ org.apache.commons.lang.exception—Provides support for nested exceptions ƒ org.apache.commons.lang.math—Provides support for common math functions ƒ org.apache.commons.lang.time—Supports date operations and formatting We’ll examine this packaging structure in the examples that follow. 8.2.2 Lang in action The Lang package is all about convenience. As we said before, it promotes defensive programming by providing alternatives to common Java core language practices that may result in unexpected runtime problems. It makes sense, therefore, that most of its classes are structured in a way that makes them easy to use. Let’s start the examples with the static methods of the org.apache.commons.lang package. Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF Using the org.apache.commons.lang package The org.apache.commons.lang package is central to the Lang component. This package handles most of the functionality and gives this component its name. Let’s start our exploration of this package with the StringUtils class. Listing 8.15 shows the usage of the StringUtils class with a few of its static methods. Listing 8.15 Examples of using the StringUtils class package com.manning.commons.chapter08; import org.apache.commons.lang.StringUtils; public class StringUtilsExampleV1 { public static void main(String args[]) { System.err.println( StringUtils.abbreviate("Take time off working", 0, 10)); System.err.println( StringUtils.capitalize("vanderLust")); System.err.println(StringUtils.center("MTV", 7, '=')); System.err.println(StringUtils.chomp("temperature", "ure")); System.err.println(StringUtils.chop("Dane")); System.err.println(StringUtils.contains("Dorothy", "oro")); System.err.println( StringUtils.containsNone("r u m t", new char[] {'r', 'o'})); System.err.println( StringUtils.containsOnly("r u m t", new char[] {'r', 'o'})); System.err.println( StringUtils.countMatches("arthur", "r")); System.err.println( StringUtils.deleteWhitespace("f f f f")); System.err.println(StringUtils.difference("govern", "government")); System.err.println( StringUtils.getLevenshteinDistance("govern", "government")); } } As you can see in these examples, static methods of the StringUtils class provide several utility functions and are inherently easy to use. It would be impossible to list all of these methods in this book; this example uses some of the better-known ones. Listing 8.16 shows some examples of using the ArrayUtils class. As with the StringUtils class, null values are handled gracefully. Listing 8.16 Using the ArrayUtils class package com.manning.commons.chapter08; import org.apache.commons.lang.ArrayUtils; public class ArrayUtilsExampleV1 { Prints “Take ti...” Prints “==MTV==” Prints “Dan” Prints “false” Prints “2” Prints “ffff” Prints “ment” Prints “4” Licensed to Tricia Fu 25MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 25 public static void main(String args[]) { long[] longArray = new long[] {10000, 30, 99}; String[] stringArray = new String[] {"abc", "def", "fgh"}; long[] clonedArray = ArrayUtils.clone(longArray); System.err.println( ArrayUtils.toString( (ArrayUtils.toObject(clonedArray)))); System.err.println(ArrayUtils.indexOf(stringArray, "def")); ArrayUtils.reverse(stringArray); System.err.println(ArrayUtils.toString(stringArray)); } } All methods of ArrayUtils that act on one primitive type are overloaded to act for all primitive types. For example, the longArray was cloned, but we could as easily have cloned an intArray using the same method signature. Moving on, listing 8.17 shows examples of using the StringEscapeUtils class, which provides utility functions for escaping (and unescaping) Strings for Java, JavaScript, HTML, XML, and SQL. Listing 8.17 Using the StringEscapeUtils class package com.manning.commons.chapter08; import org.apache.commons.lang.StringEscapeUtils; public class StringUtilsEscapeExampleV1 { public static void main(String args[]) { String unescapedJava = "Are you for real?"; System.err.println( StringEscapeUtils.escapeJava(unescapedJava)); String unescapedJavaScript = "What's in a name?"; System.err.println( StringEscapeUtils.escapeJavaScript(unescapedJavaScript)); String unescapedSql = "Mc'Williams"; System.err.println( StringEscapeUtils.escapeSql(unescapedSql)); String unescapedXML = ""; System.err.println( StringEscapeUtils.escapeXml(unescapedXML)); } } Clone array Print String representation of wrapper array Find index of “def” Reverse array Tab between “you” and “for” Print “Are you\tfor real?” Escape single quote Escape single quote Print “<data>” Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF As you may expect, escapeSql isn’t very useful for constructing SQL queries, because if you’re using PreparedStatement instances to construct your queries, they will handle these cases automatically. Listing 8.18 combines using the CharRange and CharSet classes. CharRange represents a continuous range of characters. If you optionally negate this range, it will contain everything except the range. CharSet is a handy representation of a set of characters. Listing 8.18 Using the CharSet and CharRange classes package com.manning.commons.chapter08; import org.apache.commons.lang.CharSet; import org.apache.commons.lang.CharRange; import org.apache.commons.lang.ArrayUtils; public class CharSetExampleV1 { public static void main(String args[]) { CharSet set = CharSet.getInstance("the apprentice"); System.err.println(set.contains('q')); CharRange[] range = set.getCharRanges(); System.err.println(ArrayUtils.toString(range)); } } The set doesn’t contain duplicate characters. Therefore, the characters t, p, and e appear only once in the set and the related ranges. Finally, before we finish this section, look at listing 8.19, which shows examples of using the Validate class. If you’re familiar with using JUnit, and you like the way assertions are made in it, then you’ll like the methods of this class. These methods make validations convenient and let you compose a message mixed with input variables. All methods throw an IllegalArgumentException (with a predefined message, if any) if the assertion that is being made doesn’t turn out to be true. Listing 8.19 Using the Validate class package com.manning.commons.chapter08; import org.apache.commons.lang.Validate; import java.util.List; import java.util.ArrayList; public class ValidateExampleV1 { public static void main(String args[]) { int i = 35; Validate.isTrue(i > 30); Validate.isTrue(i > 40, "Invalid value: ", i); Create set of characters with given String Print “false” Get continuous ranges Pass validation Throw custom exception Licensed to Tricia Fu 27MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 27 List data = new ArrayList(); Validate.notEmpty(data, "Collection cannot be empty"); data.add(null); Validate.noNullElements(data, "Collection contains null elements"); } } There are other classes in this package that we haven’t discussed. However, most offer generic static methods and are as easy to use as those in the examples shown here. Using the org.apache.commons.lang.builder package The classes of the org.apache.commons.lang.builder package provide functionality that creates consistent equals, toString, hashCode, and compareTo methods. What these methods are and why they need to be consistent is a matter for whole other book (already written by Joshua Bloch: Effective Java Programming Language Guide, http://java.sun.com/developer/Books/effectivejava/), but in this section we’ll lay down a few basics before getting into the builder package classes. Admittedly, you probably know about these methods. You may even have written them at some stage. But were they always consistent? The equals, toString, and hashCode methods are the methods of the java.lang.Object class. Thus, every class inherits them from this superclass, in the absence of explicitly overridden methods of their own. The default behavior of these methods is as follows: ƒ equals—Two objects are equal if their reference (handle) points to the same object in memory. Thus the default behavior is to return x == y, where x and y are the references to the two objects. ƒ toString—Returns Class Name + "@" + Integer.toHexString(hashCode()). ƒ hashCode—Returns the internal address of the object, represented as an int. The Java language lays down some general rules to use when you’re overriding these methods in classes of your own. Let’s first look at the rules for the equals method, which is supposed to return true or false based on the equality of two objects: ƒ An object must be equal to itself; x.equals(x) should always return true. ƒ The equals method must be symmetric; if x.equals(y) is true, then y.equals(x) must also be true. ƒ The equals method must be transitive; if x.equals(y) is true and y.equals(z) is true, then x.equals(z) must also be true. ƒ Equality should not be a function of time; x.equals(y) should always return the same true or false value, provided x and y haven’t changed. ƒ An object must never be equal to the null value; x.equals(null) should always return false. Similarly, the rules for the hashCode method, which is supposed to return a unique integer value representing the hash code of an object, are as follows: ƒ Always override the hashCode method when the equals method is overridden. Throw custom exception Licensed to Tricia Fu 28 JAKARTA COMMONS ONLINE BOOKSHELF ƒ If two objects have been deemed equal by the equals method, they must return the same hash code. ƒ An object must return the same hash code consistently, provided the object hasn’t changed. ƒ It is not required that two different objects return different hash codes. Finally, although there are no rules for the toString method, generally it should return a consistent textual representation of the object in question. It’s easy to violate the rules of the equals and hashCode methods and write code for them that doesn’t perform as expected. The consequences of writing such methods can be subtle, taking hours to figure out. Listing 8.20 shows part of the Person class from listing 8.3 with an equals method. Listing 8.20 Part of the Person class with the equals method added public boolean equals(Object o) { if(o == this) return true; if(!(o instanceof Person)) return false; Person otherPerson = (Person)o; if(otherPerson.getName().equals(this.name) && otherPerson.getGender() == this.gender) return true; return false; } Although this method works perfectly, the fact that we haven’t added a hashCode method violates one of the rules defined earlier. Listing 8.21 shows this problem in action. Listing 8.21 Working with an equals method but no hashCode method package com.manning.commons.chapter08; import java.util.Map; import java.util.HashMap; public class EqualsHashCode { public static void main(String args[]) { Person p1 = new Person(); p1.setName("Steven Spielberg"); p1.setGender(1); Person p2 = new Person(); p2.setName("Steven Spielberg"); p2.setGender(1); System.err.println("Is P1 equal to P2? " + p1.equals(p2)); Map personMap = new HashMap(); personMap.put(p2, "available"); System.err.println("Is P2 available? " + personMap.get(p1)); } } P1 and P2 are Put P2 in HashMap Try to retrieve P2 information using equal P1 Licensed to Tricia Fu 29MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 29 As you can see if you run the listing, the last step prints null, instead of available, even though P1 and P2 are equal objects. In the absence of a hashCode method, the hash codes of these two equal objects evaluate to different values, which is an erroneous situation. The EqualBuilder and HashCodeBuilder classes of the org.apache.commons.builder package help to write equals and hashCode methods that are consistent and follow the rules identified by Joshua Bloch in his book. The critical part of these rules for the equals method is as follows: Check whether each significant field matches the corresponding field in the argument. For primitives, excluding floats and doubles, use == for a shallow check; for objects, invoke their equals method, for floats and doubles, get their int and long values respectively (using Float.FloatToIntBits and Double.DoubleToLongBits), and compare those. If a field is an array, each element of the array must be compared with these rules in mind. Let’s see how to use these classes to write the equals and hashCode methods. Listing 8.22 shows a snippet of these two methods added to the Person class. Listing 8.22 Using HashCodeBuilder and EqualsBuilder public boolean equals(Object o) { if(!(o instanceof Person)) return false; Person otherPerson = (Person)o; return new EqualsBuilder() .append(name, otherPerson.getName()) .append(gender, otherPerson.getGender()) .isEquals(); } public int hashCode() { return new HashCodeBuilder() .append(name) .append(gender) .append(contactNumber) .toHashCode(); } The EqualsBuilder and HashCodeBuilder classes make the task of creating consistent equals and hashCode methods easier by following a consistent pattern themselves. As you can see, each class’s new instance is created and significant fields are included by using the append method. The append methods produce the respective equals or hashCode result for different types using overloaded methods. The final result is brought out by the isEquals and toHashCode methods, respectively. The fields to include are up to you, but you must be careful to include those that make the most sense. For example, the equals method doesn’t include the contactNumber field, but the hashCode method does. As we pointed out earlier, this package also contains implementations that allow consistent creation of toString and compareTo methods. Usage of these classes is similar to that in listing 8.22. For example, the following code snippet shows the toString method created for the Person object using the ToStringBuilder class: public String toString() { return new ToStringBuilder(this) Return true or false Return int hashCode Licensed to Tricia Fu 30 JAKARTA COMMONS ONLINE BOOKSHELF .append("Name", name) .append("Gender", gender) .append("Contact Details", contactNumber) .toString(); } Using this toString method, an example invocation results in the following String: com.manning.commons.chapter08.Person@1be2d65[Name=Steven Spielberg,Gender=1,Contact Details={Home=444444444, Mobile=457993444}] As you can see, the format that the ToStringBuilder class uses to output the result is fully qualified class name@ internal address[fieldName=fieldValue ...] You can change the field names that are printed by changing the append method for that field. Further, you can change the format of this output by using the ToStringStyle class. Finally, all the Builder classes provide a way to build their relevant information through reflection. This allows you to, for example, build a toString implementation without needing to specify all the fields by using ToStringBuilder.reflectionToString(this). Of course, this puts the value of all fields in the output, which may or may not be what you want, but it’s definitely easier than specifying all the fields yourself. Using the org.apache.commons.lang.math package The classes of this package provide business mathematical functions that let you work with a range of numbers and use Math.random in a more functional way. The range of numbers is specified in the abstract class Range, and five implementations are provided: DoubleRange, FloatRange, IntRange, LongRange, and NumberRange. The names specify the type of numbers they represent. The Range abstract class represents a collection of numbers, with one added restriction: The numbers are sequential. This class then provides several utility functions to traverse this range; for example, range1.containsRange(range2) compares two ranges to find out whether the second range is contained wholly within the first range; and range1.overlapsRange(range2) confirms whether the second range overlaps the first. In addition to range-specific operations, type-specific operations are also supported for type-specific implementations. For example, containsInteger(int intValue) tells whether a range contains the specific integer, and getMaximumInteger() returns the maximum int value in the range. If you call getMaximumDouble on an int range, the maximum int value will be returned as a double. Listing 8.23 shows an example of using the Range classes. Listing 8.23 Using the Range classes package com.manning.commons.chapter08; import org.apache.commons.lang.math.IntRange; public class RangeCheckV1 { public static void main(String args[]) { IntRange range1 = new IntRange(35, 70); IntRange range2 = new IntRange(35); IntRange range3 = new IntRange(69, 75); Create range from 35–70, inclusive Create range with only one number, 35 Licensed to Tricia Fu 31MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 31 System.err.println(range1.containsRange(range2)); System.err.println(range1.overlapsRange(range3)); System.err.println(range1.getMaximumInteger()); } } The core Java API lacks a dedicated class for working with fractional numbers. Numbers of the type of 3/4, 2 2/3, and 27/98 can only be worked on as their double equivalent. This package provides a class called Fraction, which you can use to model these numbers. Numbers can be modeled based on their double value or their fractional parts. This class also provides several methods to operate on these numbers as well. Listing 8.24 provides some examples of working with this class. Listing 8.24 Using the Fraction class package com.manning.commons.chapter08; import org.apache.commons.lang.math.Fraction; public class FractionExampleV1 { public static void main(String args[]) { Fraction twoThirds = Fraction.TWO_THIRDS; Fraction fraction_whole = Fraction.getFraction(2, 2, 3); Fraction fraction = Fraction.getFraction(27, 98); Fraction fraction_double = Fraction.getFraction(4.56); Fraction fraction_string = Fraction.getFraction("2 1/3"); System.err.println(twoThirds.doubleValue()); System.err.println(fraction_string.getNumerator()); System.err.println(fraction_whole.divideBy(fraction_double)); System.err.println(fraction.divideBy(fraction)); } } c The acceptable String formats for creation of Fraction instances are doubles that must contain a period (.), X Y/Z, and X/Y. Using the org.apache.commons.lang.time package The classes in this package provide convenient methods to deal with date and time operations. One of the classes is FastDateFormat, which is a replacement for SimpleDateFormat class of the core Java API. This class should be used in all instances instead of SimpleDateFormat where threadsafe formatting is required. Using this class is identical to using the SimpleDateFormat, except that instead of using a constructor to create an instance, you use any variation of the getInstance method; for example, getInstance("dd-mm-yyyy") returns a formatter suitable for the default locale with the specified pattern. Using the DateFormatUtils class indirectly uses the FastDateFormat class. This class provides static methods to format Date instances in both long and normal date types. For example, using Print true Print true Print 70 Use whole, numerator, and denominator Use numerator and denominator Convert double to Fraction c Convert String to Fraction Licensed to Tricia Fu 32 JAKARTA COMMONS ONLINE BOOKSHELF System.err.println(DateFormatUtils.format(new Date(), "dd-MMM-yyyy")) prints the current date in the given format. The most useful class in this package is DateUtils. It provides three operations on Date objects (also Calendar objects) that are useful in dealing with Date instances: ƒ truncate truncates a given date. ƒ round rounds the given Date instance to the next highest date. ƒ iterator provides a way to create an iterator over a given date. Listing 8.25 shows how to use these methods. Listing 8.25 Using the DateUtils class, truncate, round, and iterator package com.manning.commons.chapter08; import org.apache.commons.lang.time.DateUtils; import java.util.Date; import java.util.Calendar; import java.util.Iterator; import java.util.GregorianCalendar; public class DateUtilsV1 { public static void main(String args[]) { GregorianCalendar calendar = new GregorianCalendar(1974, 5, 25, 6, 30, 30); Date date = calendar.getTime(); System.err.println("Original Date: " + date); System.err.println("Rounded Date: " + DateUtils.round(date, Calendar.MONTH)); System.err.println("Truncated Date: " + DateUtils.truncate(date, Calendar.MONTH)); Iterator itr = DateUtils.iterator(date, DateUtils.RANGE_WEEK_CENTER); while(itr.hasNext()) { System.err.println(((Calendar)itr.next()).getTime()); } } } The DateUtils round method takes two parameters. The first is, of course, the date that needs to be rounded. The second is the part of the date that should be rounded. Therefore, in this example, we’ve decided to round the MONTH part. This makes the resulting date equal to Sat Jun 01 00:00:00 EST 1974 from Wed Jun 05 06:30:30 EST 1974. If we were to change the HOUR part, the date would Round MONTH part up to July Truncate MONTH part to June Create iterator over date Licensed to Tricia Fu 33MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 33 be rounded to Tue Jun 25 07:00:00 EST 1974. Note that not all parts of a date are supported for this operation. Using the truncate method is similar to using the round method. Instead of rounding up or down, truncate chops off the rest of the parts of the date, leaving only the part that is specified as the most significant part. Thus, this example prints Sat Jun 01 00:00:00 EST 1974 for Tue Jun 25 06:30:30 EST 1974 when told to truncate the MONTH part. The iterator method returns an iterator that is useful to iterate over a range of dates. However, you only supply one date to this method, and by supplying the range as the second parameter, you specify how the iterator should operate. For example, DateUtils.RANGE_WEEK_CENTER means that using the date supplied in the first parameter, the iterator should be constructed with it as the center date, over a week. Thus, the iterator will have seven iterations, the first iteration starting at: Sat Jun 22 00:00:00 EST 1974 and the last at: Fri Jun 28 00:00:00 EST 1974. Several ranges are available, as follows: ƒ RANGE_MONTH_SUNDAY—A whole-month range, with the each week assumed to be starting on a Sunday ƒ RANGE_MONTH_MONDAY—A whole-month range, with the week starting on a Monday ƒ RANGE_WEEK_CENTER—A week range (seven days), with the center of the week being the date supplied ƒ RANGE_WEEK_MONDAY—A week range, with the Monday of the week in which the supplied date lies as the starting date for the week ƒ RANGE_WEEK_SUNDAY—A week range, with the Sunday of the week in which the supplied date lies as the starting date for the week ƒ RANGE_WEEK_RELATIVE—A week range, with the supplied date as the starting date All three methods discussed earlier provide overloaded options that take a Calendar object or a String in place of the Date parameter. Using the org.apache.commons.lang.exception package Java Development Kit (JDK) 1.4 introduced the concept of nested exceptions. As the name suggests, this allows exception classes to be nested within one another. This in turn allows exceptions to be created that are carriers for other exceptions. With multiple exception chaining, new methods were introduced that make information about the cause of an exception to be readily available. These methods are getCause(), which returns the exception that caused the exception on which this method is called; and initCause(Throwable throwable), which marks the current exception’s cause to the one specified by the throwable parameter. Because these methods were introduced in JDK 1.4, previous versions of Java didn’t derive the full benefit of this chaining mechanism. The classes of the org.apache.commons.lang.exception package assist these early versions by supplying a mechanism to create nested exceptions. The interface Nestable is the main definition class; it’s implemented with the NestableException, NestableRuntimeException, and NestableError classes, which are counterparts to the Exception, RuntimeException, and Error classes in Java (Nestable is the counterpart to Throwable). There is also a utility class, ExceptionUtils, that you can use to examine exception messages, but it doesn’t provide any functionality beyond that presented by JDK. Listing 8.26 shows how to use the NestableException class. Licensed to Tricia Fu 34 JAKARTA COMMONS ONLINE BOOKSHELF Listing 8.26 Using the NestableException class package com.manning.commons.chapter08; import org.apache.commons.lang.exception.ExceptionUtils; import org.apache.commons.lang.exception.NestableException; import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; public class ExceptionUtilsV1 { public static void main(String args[]) { try { loadFile(); } catch(Exception e) { e.printStackTrace(); } } public static void loadFile() throws Exception { try { FileInputStream fis = new FileInputStream(new File("nosuchfile")); } catch (FileNotFoundException fe) { throw new NestableException(fe); } } } If you run this listing, you’ll see that the output shows the nested nature of the exception. The first part is the NestableException, which clearly says that the FileNotFoundException caused it. Of course, you don’t have to use NestableException in the code if you’re using JDK 1.4 or above—you can just as easily use new Exception(fe) and get the same result. Using the org.apache.commons.lang.enum package An enum is a C language construct that is used to define a sequence of integer constants. There is no equivalent in Java (versions before 5.0); the classes of this package allow you to mimic the enum functionality. Listing 8.27 shows an example enum created by extending the Enum interface of this package. Listing 8.27 Creating a minimum enum package com.manning.commons.chapter08; import org.apache.commons.lang.enum.Enum; public final class EmotionEnum extends Enum { public static final EmotionEnum JEALOUSY = new EmotionEnum("Jealousy"); public static final EmotionEnum HAPPINESS = new EmotionEnum("Happiness"); public static final EmotionEnum FEAR = new EmotionEnum("Fear"); Print complete exception and root cause Exception thrown because of nosuchfile Exception rethrown as NestableException Licensed to Tricia Fu 35MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 35 private EmotionEnum(String emotion) { super(emotion); } } The listing shows a minimum enum EmotionEnum created, which is useful only if you’re going to use it with the EnumUtils class. Listing 8.28 shows how to use this enum with the EnumUtils class. Listing 8.28 Using the EnumUtils class with the minimum enum package com.manning.commons.chapter08; import org.apache.commons.lang.enum.EnumUtils; import java.util.Iterator; public class EnumUtilV1 { public static void main(String args[]) { System.err.println(EnumUtils.getEnum( EmotionEnum.class, "Fear")); Iterator itr = EnumUtils.iterator(EmotionEnum.class); while(itr.hasNext()) { System.err.println(itr.next()); } } } If you plan to use your Enum class directly—that is, without the EnumUtils class—you’ll need to provide several more methods. Listing 8.29 shows the EmotionEnum from listing 8.27 extended to include these methods. Listing 8.29 Providing other EmotionEnum methods package com.manning.commons.chapter08; import org.apache.commons.lang.enum.Enum; import java.util.Map; import java.util.List; import java.util.Iterator; public final class EmotionEnum extends Enum { public static final EmotionEnum JEALOUSY = new EmotionEnum("Jealousy"); public static final EmotionEnum HAPPINESS = new EmotionEnum("Happiness"); public static final EmotionEnum FEAR = new EmotionEnum("Fear"); private EmotionEnum(String emotion) { super(emotion); } Return specific enum Iterate over enum values Licensed to Tricia Fu 36 JAKARTA COMMONS ONLINE BOOKSHELF public static EmotionEnum getEnum(String emotion) { return (EmotionEnum) getEnum(EmotionEnum.class, emotion); } public static Map getEnumMap() { return getEnumMap(EmotionEnum.class); } public static List getEnumList() { return getEnumList(EmotionEnum.class); } public static Iterator iterator() { return iterator(EmotionEnum.class); } } The four return statements all call the superclass method. You can now use EmotionEnum anywhere as an enum type. For example, you can call System.err.println(EmotionEnum.getEnum("Fear")) and get the right result without having to use EnumUtils. 8.3 Summary This module on BeanUtils and Lang continued from the previous module, in which we talked about Collections. These two modules explored the Commons components that enhance the Java Core libraries and were therefore discussed as a whole. The BeanUtils Commons component lets you work dynamically with JavaBeans. It allows you to do this by providing utility methods that manage the properties of JavaBeans at runtime without knowing about the structure of these beans before hand. It provides a structure to create JavaBeans dynamically as well, using classes known as DynaBean and DynaClass. In this module, we discussed the BeanUtils component and how to use it for dynamic JavaBean access and modification. You also learned how to create Dynamic JavaBeans and some specialized versions. We then moved to the Lang component, which provides helper classes to work with the classes of the Java Lang API. This was a very hands-on section, and we provided several examples. In the next module, we’ll look at the Pool and DBCP components, which provide object pool management functionality. Licensed to Tricia Fu 37MODULE 8: ENHANCING JAVA CORE LIBRARIES WITH BEANUTILS AND LANG 37 Index BeanUtils accessing JavaBean properties, 6 BeanComparator, 16 classes, 6 Collection-based classes, 16 difference between PropertyUtils and BeanUtils, 6 DynaBeans, 18 advantages of using DynaBeans, 5 Dynamic JavaBeans, 5 indexed properties, 4 introduction, 1 JavaBean property access syntax, 4 JavaBean property types, 2 JavaBean requirements, 1 JavaBean terminology, 2 LazyDynaBean, 19 locale-specific classes, 15 mapped properties, 4 nested properties, 4 packaging, 1 registering converters, 14 ResultSetDynaClass, 20 RowSetDynaClass, 21 rules for using classes, 6 scalar properties, 4 setting JavaBean properties, 9 side effect of using cloneBean, 11 using a map to set JavaBean properties, 10 using converters, 12 variations of getProperty method, 9 Builder package, 27 converters, 12 date formatting, 32 Date package, 32 DynaBean, 2, 5, 18, 19, 20, 21, 36 DynaBeans, 5 DynaClass, 5, 18, 19, 20, 21, 36 Dynamic JavaBeans, 5 enums, 34 JavaBean property types, 2 JavaBean terminology, 2 Lang ArrayUtils, 24 Builder package, 27 Char modification classes, 26 creating a range of numbers, 30 creating enums, 34 enhancing the Date package, 32 Fraction classes, 31 Math package, 30 need for the Lang package, 22 NestableException, 34 nested exceptions, 33 Range classes, 31 requirements for equals method, 27 requirements for hashCode method, 28 StringEscapeUtils, 25 StringUtils, 24 thread-safe date formatting, 32 Validate, 26 locale-specific classes, 15 Math package, 30 nested exceptions, 33 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Pool and DBCP: Creating and using object pools Vikram Goyal MODULE MANNING 9 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 9 Pool and DBCP: Creating and using object pools 9.1 The Pool component ................................................................................................................................................. 1 9.2 The DBCP component............................................................................................................................................ 18 9.3 Summary................................................................................................................................................................. 37 Index ............................................................................................................................................................................. 39 Pool and DBCP are the Jakarta Commons components that deal with creating, managing, and using object pools. DBCP (which stands for Database Connection Pool and uses the Pool component to manage database connections) represents a successful implementation of the Pool component because it builds on it to supply a connection-pooling mechanism for managing database connectivity. The generalized nature of the Pool component is brought into narrow focus by DBCP. It’s therefore natural for us to discuss these two components together. The Pool component represents a generic API for implementing pools of objects. These objects can represent any Java class, and the Pool API provides the infrastructure to maintain these objects in a pool for easy access, easy return, and lifecycle maintenance. The DBCP component also represents a generic API for maintaining connections to a database. As we said, it uses the Pool component to maintain these connections, but it also provides support services for maintaining interactions between a database and the application that is accessing it. It does so in a seamless fashion without needing to know the underlying database structure. In this module, we’ll start with the Pool component. We’ll discuss object pooling and look at the structure of the Pool API, and we’ll use it to create a basic employee pooling mechanism. We’ll then move on to explore DBCP and build some concrete examples. 9.1 The Pool component The Pool component contains a set of APIs that let you manage objects in a pool for reuse and maintainability. The idea of using a pool is to provide reusable objects. A pooling mechanism should not make any assumptions about the objects it contains and should provide facilities for easy creation, management, reuse, and destruction of those objects. The Pool component of Jakarta Commons is such a mechanism; it goes a step further by providing a means to create a factory for creation of the pool itself. But we’re getting ahead of ourselves. Let’s first look at the need for object pooling and consider some of the issues involved. We’ll then examine the structure of the Pool API and discuss in detail the facilities provided by the Pool component. 9.1.1 The case for object pooling Object pools provide a better way to manage available resources. You’ve probably come across object pools before in one form or another, most notably as connection pools for managing access to databases. However, Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF most developers use other object pools without realizing they’re doing so: Application servers, servlet engines, mail servers, even database servers, all use some sort of thread-pooling mechanism to service the requests they receive. Thread pooling is a way by which most high-end servers manage to respond to requests for services quickly. These servers don’t create a thread for each request that comes in; instead, they maintain a pool of ready threads that can be rapidly assigned to a request. The thread does its business and then is returned to the pool for future requests. Object pooling relies on three development requirements as strong cases for its use: ƒ Responsiveness—Because objects are already created and in a pool, there’s no overhead associated with object creation. Your application can quickly dip into the pool, pick an object, and assign it to the task at hand. Contrast this with the tasks needed if object pooling isn’t used: Objects must be created and initialized, and only then can they be assigned to the task at hand. Complex objects are the worst offenders and will cause your application to suffer from poor response times. As with most things, there is a tradeoff in this scenario. Heavy objects are a drain on system resources, and maintaining them in system memory waiting for tasks can be problematic. You need to maintain a fine balance between the minimum objects required in the pool and your system’s responsiveness. ƒ Reusability—Objects in a pool are reused. Although this may seem similar to the first feature, there is difference because of the way object pools are managed. Consider the case of an object pool where each time you took out an object, a new object was created to replace it (behind the scenes, by a separate thread). The object you took out wasn’t returned to the pool but was discarded. Although this process would serve the first purpose of a responsive object pool, it would create overhead for the system. To avoid the overhead of object creation and subsequent garbage collection for the discarded objects, objects pools implement a mechanism by which the object borrowed from the pool can be returned. This returned object can then be used by other tasks. ƒ Controlled access—Objects in a pool are controlled because they’re accessed and maintained within the pool. While the pool is sitting idle, it may perform maintenance operations like repairing broken objects, releasing unusable objects, and so on. Access to these objects is limited and controlled via the pool. A request for an object from the pool is handled internally using any protocol, without the application knowing anything about it. Further, by restricting access, you limit the number of these objects in the system at any time, thus improving performance and security. An object pool is an example of the Factory Pattern for creation of objects. An object pool may not be the best way to manage your objects, however. Object pools are best used in cases where the objects that are being pooled are complex; have large initialization, creation, and destruction times; are used frequently for short small tasks; and can be recycled. Examples of some generic objects that should be pooled include database connections, socket connections, and threads in application servers. In the next section, we’ll examine the difference between using an object pool to manage objects and letting the garbage collector in your JVM do its object-collection trick. Object pooling vs. the garbage collector The alternate to using an object pool is to let the garbage collector in Java do its business. You use the new keyword to create your objects, and the garbage collector cleans them up when they aren’t needed. The humble garbage collector has seen many performance advances since its inception, to the stage that it’s now a rival to object pools. A fast and efficient garbage collector can quickly reclaim objects that are no longer needed by your application, thus freeing up memory and making that memory available to other objects. This Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 3 makes it possible for you to create your objects as and when they’re needed; they aren’t lying in a pool consuming resources with the possibility that they may never be used. One obvious disadvantage of relying on the garbage collector to reclaim unused objects is that the behavior of garbage collectors is nondeterministic. This means there are no guarantees as to when the garbage collector will be run and when it will reclaim your unused objects. Although garbage collectors are fast and efficient, they can’t know exactly in advance when an object will be cleaned up, which may be a requirement for some applications. An object pool may be a better bet when you can use objects in a deterministic fashion: You always know the state of the pool in advance, and you can plan ahead. Object pool requirements Before we delve into the Pool API, let’s see what an object pool should support. An object pool should let you configure the way the pool is managed. At a minimum, the pool should allow you to set the following two parameters: ƒ Minimum number of objects to maintain—This setting lets you control the way the pool behaves during slow periods when you don’t expect much traffic. This minimum set of objects is enough to serve the expected tasks and quick enough to recycle them. ƒ Maximum number of objects to create—This setting strikes a balance between your system’s responsiveness and your system’s resources. If the pool keeps creating objects for use as requests come in without waiting for these objects to be returned, you’re bound to eventually run out of resources; the system will then become nonresponsive. In addition to these configuration parameters, an object pool must provide at least two methods: One must allow users to borrow objects from the pool, and the second must let users return objects to the pool. The pool should, if possible, provide another method to validate these objects (to make sure they conform to an accepted structure), either when they’re borrowed from the pool or when they’re returned to the pool—or even when they’re sitting idle in the pool. Easily changeable configuration parameters should let you configure the pool for these settings. 9.1.2 The Pool API The Pool API is a simple API divided into two packages. The first package, org.apache. commons.pool, contains interfaces and abstract classes that define the Pool API. This package defines the contract that implementations must support. The second package, org.apache.commons.pool.impl, contains concrete implementations of these interfaces and abstract classes. Thus, the Pool API represents a complete solution. There are enough implementations in the second package to satisfy most needs. The org.apache.commons.pool package The org.apache.commons.pool package contains four abstract classes and six interfaces. The base interface, ObjectPool, defines methods for the operation of the pool. However, you only need to define two of its methods for a bare minimum functioning pool (the rest of the methods can default to doing nothing). As we discussed earlier, these methods are Object borrowObject() to borrow objects from the pool and returnObject(Object object) to return objects to the pool. Both of these methods throw java.lang.Exception, which allows all actual implementations to throw implementation-specific exceptions that may occur on either borrowing or returning of objects. Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF This package also defines classes for lifecycle management of objects within the pool. This is done by setting factory classes for the creation of objects. The PoolableObjectFactory interface specifies the methods that a factory must implement so that the ObjectPool can use it to create, initialize, passivate, destroy, and validate objects within the pool. Without a factory, an object pool can’t create and maintain its objects. The ObjectPoolFactory interface defines a single method called createPool, which implementations can override to define how the ObjectPool is created. This factory is useful in cases where you want to define a generic way that applications create and manage objects—for example, in application servers. The application server may provide functionality for developers to create pools of objects using separate object pools for each object, and this interface can help define a standardized approach. The ObjectPool, PoolableObjectFactory, and ObjectPoolFactory interfaces together form the basic contract needed to implement your own object pools. The KeyedObjectPool, KeyedPoolableObjectFactory, and KeyedObjectPoolFactory interfaces provide a variation on this combination. These interfaces, which are present in the same package, provide a way for objects to distinguish themselves from one another while in the pool. The original set of interfaces provide for a pool where one object isn’t distinguishable from another. However, in some cases objects are kept and accessed using a distinguishing feature, or key. Hence the name of these instances starts with Keyed. An interesting point to note is that you may choose to have at most one object per key in the pool or several objects per key, in essence creating several object pools distinguished using the key. Figure 9.1 shows the distinction between these two types, and the nonkeyed pool. Figure 9.1 Two variations of KeyedObjectPool and a non-KeyedObjectPool Since objects in the keyed pool are distinguished using a key, it’s no surprise that the borrow and return methods use this key when accessing this pool. Thus, the signatures of these methods are borrowObject(String key) and returnObject(String key), respectively. Both the keyed and nonkeyed pool interfaces define optional functions. One of these is the ability to set the factory class for creation of objects in the pool, which we discussed earlier. This method’s signature can be either of the following, as the case may be: setFactory(PoolableObjectFactory factory) setFactory(KeyedPoolableObjectFactory factory) Other optional functions include the following: Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 5 ƒ getNumActive() returns the number of active objects (that is, objects that have been borrowed from the pool but not yet returned). ƒ getNumActive(Object key) applies to a particular key. ƒ getNumIdle() returns the number of objects in the pool that are doing nothing. ƒ getNumIdle(Object Key) acts similarly for keyed objects. ƒ clear() clears the pool of all objects. ƒ clear(Object key), as expected, clears all the objects corresponding to a particular key. Actual implementations may throw a java.lang.UnsupportedOperationException if they don’t wish to support a particular method. However, a pool wouldn’t be very useful if it didn’t support most of these functions. At the beginning of this section, we said that this package contains six interfaces and four classes. We’ve discussed the interfaces but not the classes: BaseObjectPool, BaseKeyedObjectPool, BasePoolableObjectFactory, and BaseKeyedPoolableObjectFactory. (Note that all these classes are abstract.) They’re adapter classes that implement four of the interfaces we’ve discussed, making it easier for concrete implementations to define only the desired methods. The interfaces that aren’t implemented are ObjectPoolFactory and KeyedObjectPoolFactory, since they contain only one method each, and defining basic classes for them would serve no purpose. Let’s now look at the concrete reference implementation of these classes, housed in the org.apache.commons.pool.impl package. The org.apache.commons.pool.impl package This package, as expected, contains concrete implementations for the Pool API, using the set of interfaces and abstract base classes discussed in the previous section. As a developer and user of the Pool API, you’ll consider these implementation classes, because they serve most object-pooling needs—especially GenericObjectPool and its keyed counterpart, GenericKeyedObjectPool. However, these two classes are no good on their own; they must be coupled with the correct factory classes for the creation of objects they’re pooling. Application developers are left to supply a factory class for the creation of these objects by implementing the PoolableObjectFactory and KeyedPoolableObjectFactory interfaces, respectively, and setting this factory on the GenericObjectPool by either using the method setFactory(PoolableObjectFactory factory) or passing it through the GenericObjectPool constructor (or the GenericKeyedObjectPool constructor, as the case may be). The GenericObjectPool and GenericKeyedObjectPool classes can be configured using the various setter methods for their properties. However, to make it easy to set the vast number of properties, an inner class called Config can be used. You’ll see how to use this class, and the properties that can be set, in the examples that follow this section. In addition to the GenericObjectPool (and its keyed counterpart), there are two more implementations of the ObjectPool interface, which give a different flavor to this API. StackObjectPool (and StackKeyedObjectPool), based on the java.util.Stack, let you create a pool that puts no limit on the number of objects that are created. Initial objects lay idle in the pool, and more are created if heavy demand ensues. The underlying pool is, as we said, based on the Stack class, and the GenericObjectPool is based on the org.apache.commons.collections.list. CursorableLinkedList class. There is also a StackObjectPoolFactory for managing the objects in this pool (StackKeyedObjectPoolFactory for the keyed pool). Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF The second class with a different flavor is SoftReferenceObjectPool, which is based on the java.lang.ref.SoftReference class. Objects in this pool are maintained using an ArrayList, but these objects are marked as softly reachable for the garbage collector. When these objects are lying idle in the pool, they can be reclaimed by the garbage collector if it determines that there is a need to reclaim memory for other memory-intensive operations. Note that there is no keyed counterpart to this class. Now that you know more about the classes of the Pool API, let’s see how to use them in your applications. 9.1.3 The Pool API in action Before we look at some code that exercises the Pool API, it’s important that you understand the configuration parameters that affect its functioning. This will allow you to fully understand this API and use it more efficiently. We’ll concentrate on the GenericObjectPool class, because the properties that it exposes are central to the Pool API’s functioning. Table 9.1 shows all the configuration parameters that affect GenericObjectPool class. This table lists each configuration parameter’s name, the effect the parameters have on the pool, their valid values, their types, and their default values. The properties listed in table 9.1 are representative of the GenericObjectPool and GenericKeyedObjectPool implementations. GenericKeyedObjectPool contains another property, maxTotal, which limits the maximum number of objects that can exist, idle or active, regardless of the key. Its default value is -1, which means that there is no such limit. Active and Idle Objects Looking at table 9.1, it’s important to understand the difference between an active and an idle object. An active object is out there doing some work. An idle object is sitting around, waiting to be called up for work. An idle object resides in the pool and is eligible to be borrowed from the pool. It becomes an active object when a caller has borrowed it from the pool. After the object has been used, it can be returned to the pool and becomes an idle object again, waiting to be called up for work. An idle object must be activated before it can be borrowed. Activation allows an object to be ready for the real world. For example, if your objects hold connections to a remote server, activation may involve letting the remote server know that the connection is now live. It’s the duty of the pool to return an active object, but it delegates this task to the factory used for creation of these objects. Similarly, when an object is returned to the pool, it must be passivated (deactivated) by the factory before it can be put back in the pool. (Objects may or may not be validated before they’re borrow from or returned to the pool, but they’re always activated and passivated on borrow and return, respectively. Of course, both activation and passivation of objects may do nothing and return the objects as they are.) Objects make the transition from idle to active using the first-in-first-out (FIFO) principle for GenericObjectPool and its keyed counterpart implementations. StackObjectPool, its keyed counterpart, and SoftReferenceObjectPool use the last-in-first-out (LIFO) principle. Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 7 Table 9.1 Configuration parameters for GenericObjectPool and GenericKeyedObjectPool Configuration parameter Effect on pool Valid values Type Default value maxActive This parameter defines the maximum number of objects that can be borrowed from the pool at a given time. For GenericKeyedObjectPool, it defines the maximum number of objects that can be borrowed per key from the pool. If this value is reached, the pool is exhausted and can’t give out any more objects. However, this behavior can be controlled by the property described next. Values should be greater than or equal to 0. If a negative value is specified, there is no limit on the borrowed objects, and theoretically the pool can never be exhausted. int 8 whenExhaus tedAction If the maxActive value has been reached for a pool (no more objects are available to be borrowed), this parameter specifies the action to be taken when a request for an object is made to the pool. Fail (byte value of 0)—The caller is told that no objects are available, by throwing the exception NoSuchElementExcept ion. Grow (byte value of 2)—The pool automatically grows (it creates a new object and returns it to the caller). Block (byte value of 1)—The caller waits until an object is returned to the pool and becomes available. The length of time the caller waits is specified by the maxWait property. If this property is negative (the default), the caller waits indefinitely. If no object is available even after the wait is over, a NoSuchElementExcept ion exception is thrown. byte Block (byte Value of 1). maxIdle The property is used to configure the maximum number of objects that can be in the pool at any given time (per key for GenericKeyedObjectPool). If a caller tries to return an object to the pool and the pool already contains the maximum number of objects sitting idle, specified by this property, the returned object is rejected. As with maxActive, values should be greater than or equal to 0. A negative value implies no limit on the number of objects that can stay idle in the pool. int 8 timeBetwee nEvictionR unsMillis This property controls the spawning of a thread and the time between its active runs. This thread can examine idle objects in the pool and destroy these objects to conserve system resources. A negative or 0 value for this property ensures that no such thread is started. Even if such a thread is started, your objects can be prevented from being evicted by using the next property. Note that GenericKeyedObjectPool gets only one thread and not a thread per keyed object. This value is in milliseconds and should be greater than 0. A value of 0 or less ensures that this thread isn’t started. long -1 (eviction thread isn’t started by default) minEvictab leIdleTime Millis This property specifies the minimum time in milliseconds that objects can stay in a pool. After this time is up, these objects are eligible to be destroyed, and will be destroyed, the next time the eviction thread is run, as specified by the previous property. You can set a negative value for this property, which stops objects from being destroyed. However, objects can still be destroyed using the next property. Note that this value is reset when a borrowed object is returned to the pool. This value is in milliseconds and should be greater than 0. A value of 0 or less means that objects aren’t eligible for destruction based on the time they have spent in the pool alone. long 1800000 millisecond s or half an hour Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF Table 9.1 Configuration parameters for GenericObjectPool and GenericKeyedObjectPool (continued) testWhileId le This property indicates to the eviction thread (if such a thread is running) that objects must be tested when they’re sitting idle in the pool. They’re tested for validity using the factory that creates them, by calling the validateObject method on the factory. It’s up to the factory to decide how objects are validated. Invalidated objects are destroyed by calling the destroyObject method. Objects can also be tested for validity when they’re borrowed from the pool by setting testOnBorrow or on their return to the pool by setting testOnReturn. Note that validateObject can only be called on objects that have been activated by calling the activateObject method. Boolean true or false. boolean false numTestsPer EvictionRun This property limits the number of objects to examine in each eviction run, if such a thread is running. This allows you to enforce a kind of random check on the idle objects, rather than a full- scale check (which, if the number of idle objects is large, can be processor intensive). 0 or greater. If it’s 0, then of course, no objects will be examined. However, If you supply a negative value, you can enable fractional checking. Thus, if you supply a value of -2, it tells the evictor thread to examine 1/2 objects (roughly). int 3 minIdle This property controls the minimum objects that must be maintained in the pool at all times (per key for GenericKeyedObjectPool). If the number of idle objects in the pool falls below this value, then the evictor thread (if running) starts creating new objects to rise to this level. 0 or greater. int 0 StackObjectPool and StackKeyedObjectPool contain two configurable properties. These are listed in table 9.2. Table 9.2 Configuration parameters for StackObjectPool and StackKeyedObjectPool implementations Configuration parameter Effect on pool Valid values Type Default value maxIdle This property limits the maximum number of idle objects in the pool, similar to the maxIdle property discussed in table 9.1. 0 or greater. Unlike the corresponding property in table 9.1, a negative value doesn’t mean unlimited idle objects are allowed. Instead, a negative value is ignored and the default value is used. int 8 initIdleCapacity This property specifies the initial size of the underlying stack that is used for this pool. Note that this doesn’t prepopulate this stack with actual objects; it only ensures that the stack is ready to accept these objects, by calling the ensureCapacity method on the Stack class. Greater than 0. A value of 0 or less is converted to the default value. int 4 SoftReferenceObjectPool is the only implementation that allows you to initialize your pool with objects by specifying the initSize property. Using this property, the pool is initialized with softly reachable objects when the pool is constructed. There is no default value for this property; therefore, no objects are constructed if you don’t specify it. Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 9 There is no point knowing about all the properties of a Pool implementation, if you don’t know how to use them. The next section builds a basic pool for creating employees. Creating an employee pool Listing 9.3 shows a simple example of using the GenericObjectPool. As you may expect, you can’t use GenericObjectPool without a factory class for creation of the objects the pool is supposed to manage. Listing 9.2 shows an example factory class. But what objects does this factory create? Well, the objects are recyclable skilled employees. We’re going to create a skilled recyclable employee pool and enhance it as we present more examples of the Pool API. Listing 9.1 shows this Employee class. These employees are picked up from the pool, asked to do some work, and then returned to the pool. Employees in the pool are indistinguishable from one another, as we’ll discuss more after the code. Listing 9.1 First draft of the Employee class, which is used to form a pool of employees package com.manning.commons.chapter09; public class Employee { private String name; public Employee() { } public String getName() { return this.name; } public void setName(String name) { this.name = name; } public void doWork() { // does nothing } public String toString() { return this.name; } } Listing 9.2 shows the first draft of the factory class for creating employees. Listing 9.2 First draft of the factory class for creating employees package com.manning.commons.chapter09; import org.apache.commons.pool.BasePoolableObjectFactory; public class EmployeeFactory extends BasePoolableObjectFactory { public Object makeObject() { return new Employee(); } } Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF Finally, listing 9.3 shows how to use the Employee pool we just created. Listing 9.3 First draft of the TestObjectPool class package com.manning.commons.chapter09; import org.apache.commons.pool.impl.GenericObjectPool; public class TestObjectPool { public static void main(String args[]) throws Exception { GenericObjectPool pool = new GenericObjectPool(); pool.setFactory(new EmployeeFactory()); pool.setMinIdle(5); pool.setTimeBetweenEvictionRunsMillis(500); Thread.currentThread().sleep(600); System.err.println("Number of employees in pool: " + pool.getNumIdle()); Employee employee = (Employee)pool.borrowObject(); employee.doWork(); System.err.println("Employee: " + employee); pool.returnObject(employee); } } The pool is created with the default configuration settings, and only the employee factory is installed. The pool is initialized to contain five employees only. The five employees are created after the eviction thread has run, so the sleep code must let this thread run at least once before the program quits. This isn’t the preferred way to initialize the pool, however. To borrow an employee from the pool, we call the borrowObject() method and cast the resulting object to the proper type. The employee is then made to do some work and, after that, returned to the pool. Listing 9.3 shows a way of initializing the pool that relies on setting the minIdle property of the pool and starting the eviction thread. After the eviction thread has run once, it creates this minimum number of objects. However, this is probably not the best way to initialize the pool, because it creates a dependency on starting an eviction thread. The preferred way to initialize the pool is to add the initial objects yourself, as shown here: for(int i = 0; i < 5; ++i) { pool.addObject(); } You can still set the minIdle property and start the eviction thread, but then these values serve the purpose they were meant for. If you run the code in listing 9.3, you’ll see that the state of the employee is printed as having a null value. This is to be expected, since the toString method prints the name of the employee, and this name Default settings pool Allow only five objects Borrow object Return object Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 11 isn’t set anywhere. Perhaps listing 9.3 can be modified to set the employee’s name and alter its state. However, if we then return this same employee to the pool, it will be clearly distinguishable from the rest of the employees in the pool. This isn’t desirable, because other applications may want to use this employee in its initial state; if we return a modified employee to the pool, the pool serves no purpose. After all, the goal is to create a pool of similar employees from which an employee can be borrowed at any time without worrying about its internal state. This is where the idea of validating objects comes into the picture. Recall that objects may be validated when they’re borrowed, returned, or sitting idle in the pool. We’d like to validate objects when they’re returned, so that if the borrowing application has modified the internal state of the borrowed employee, it can be validated to make sure that it can be returned to the pool. This may lead to objects being dropped from the pool, because objects that fail validation are automatically dropped. If you’d rather have the objects be fixed to their internal state rather than be dropped from the pool, you can add a passivate method to the factory. Note that if you want objects to be validated when they’re returned, you can’t rely on passivation to happen. Either you can rely on objects being validated and risk them being dropped if they fail validation, or you can have them passivated. Ultimately, this means is that object validation can only be performed on activated objects. Listing 9.4 shows the modified version of the factory class from listing 9.2 to accommodate these methods. Listing 9.4 Modifying the factory to add validation and passivation methods package com.manning.commons.chapter09; import org.apache.commons.pool.BasePoolableObjectFactory; public class EmployeeFactory extends BasePoolableObjectFactory { public Object makeObject() { return new Employee(); } public boolean validateObject(Object obj) { if(obj instanceof Employee) { if(((Employee)obj).getName() == null) return true; } return false; } public void passivateObject(Object obj) throws Exception { if(obj instanceof Employee) { ((Employee)obj).setName(null); } else throw new Exception("Unknown object"); } } Validation involves checking whether the returned employee is unnamed. Passivation involves explicitly setting the employee’s name to null so that it’s indistinguishable from other employees in the pool. However, passivation is done only if the pool is configured not to perform validation on returning objects. Listing 9.5, a modified version of listing 9.3, shows how to use validation. Perform validation Perform passivation Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF Listing 9.5 Modifying listing 9.3 to use validation on returned and initialized objects package com.manning.commons.chapter09; import org.apache.commons.pool.impl.GenericObjectPool; public class TestObjectPool { public static void main(String args[]) throws Exception { GenericObjectPool pool = new GenericObjectPool(); pool.setFactory(new EmployeeFactory()); /*First way of initializing pool pool.setMinIdle(5); pool.setTimeBetweenEvictionRunsMillis(500); Thread.currentThread().sleep(600);*/ /* second, and preferred way */ for(int i = 0; i < 5; ++i) { pool.addObject(); } pool.setTestOnReturn(true); System.err.println("Number of employees in pool: " + pool.getNumIdle()); Employee employee = (Employee)pool.borrowObject(); employee.setName("Fred Flintstone"); employee.doWork(); System.err.println("Employee: " + employee); pool.returnObject(employee); System.err.println("Number of employees in pool: " + pool.getNumIdle()); } } As we noted earlier, objects may be tested while they’re sitting idle in the pool and not only when they’re returned or borrowed. For that to happen, you must run the eviction thread by setting a nonnegative value, as shown here: pool.setTimeBetweenEvictionRunsMillis(500); pool.setTestWhileIdle(true); Objects may also be given a live-by date. For example, if we think that employees in the example outlive their usefulness by the end of a second, we can make sure they’re no longer part of the employee pool by using the following snippet: Preferred pool initialization method Force validation on object return Invalid object is dropped Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 13 pool.setMinEvictableIdleTimeMillis(1000); pool.setTimeBetweenEvictionRunsMillis(600); This code tells the pool that employees that have lived for a second are dispensable; and that every 600 milliseconds the eviction thread runs, the thread may inspect these employees and remove them from the pool. The minEvictableIdleTimeMillis property sets a timestamp on these employees. As you can see, the eviction thread, if started, does several things: 1. The thread sleeps for the milliseconds specified. 2. When awakened, it examines finite number of objects from the pool. The number of objects to be examined is determined by the numTestsPerEvictionRun property. These finite objects are examined and evicted based on the following criteria: The first objects to be removed are the ones that have outlived their usefulness because they have passed the live by date as specified by a positive minEvictableIdleTimeMillis property. Next, objects are evicted if they fail validations. This only happens if the testWhileIdle property is set to true. 3. It ensures that the minimum number of idle objects is maintained in the pool. It uses the minIdle property, along with the count of the currently active and idle objects, to arrive at a final figure. Objects are created if the count falls below the minIdle mark. We have created a basic employee pool where employees are created, borrowed, and returned without any problems and all employees are the same. However, employees are hired based on skill; not all employees are similar, so it makes sense to consider a skilled employee pool. Let’s look at how we can create such a pool. Creating a skilled employee pool Suppose that we want to create a pool of employees where the employees are pooled based on their skills. Instead of creating several pools for each skill set, we can use the GenericKeyedObjectPool class to create one pool and use the skill as the key for accessing employees with the same skill. This way, any external application can dip into our pool and request an employee of a particular skill, and the pool will return the employee from the pool. (The pool will create an employee of that skill, if it doesn’t exist already. Of course, you can restrict the pool to allow only a particular set of skills by modifying the factory makeObject method.) Listing 9.6 shows the SkilledEmployee class, which extends the Employee class from listing 9.1 by adding a skill property. Listing 9.6 SkilledEmployee, which extends the Employee class by adding a skill property package com.manning.commons.chapter09; public class SkilledEmployee extends Employee { private String skill; public SkilledEmployee(String skill) { this.skill = skill; } Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF public String getSkill() { return this.skill; } public String toString() { return getSkill() + " -- " + getName(); } } We need a new factory for creating these skilled employees. Listing 9.7 shows such a factory, which mirrors listing 9.2. Listing 9.7 SkilledEmployeeFactory for creating skilled employees package com.manning.commons.chapter09; import org.apache.commons.pool.BaseKeyedPoolableObjectFactory; public class SkilledEmployeeFactory extends BaseKeyedPoolableObjectFactory { public Object makeObject(Object key) { if(key == null || !(key instanceof String) || ((String)key).length() == 0) throw new IllegalArgumentException("Invalid key specified"); return new SkilledEmployee(key.toString()); } } We could restrict the types of skilled employees that we create by making sure the key is part of a predecided list. Here, the employee is created using a key that is converted to type String. Listing 9.8 shows how this skilled employee pool can be used. Listing 9.8 Using the skilled employee pool package com.manning.commons.chapter09; import org.apache.commons.pool.impl.GenericKeyedObjectPool; public class TestKeyedObjectPool { public static void main(String args[]) throws Exception { GenericKeyedObjectPool pool = new GenericKeyedObjectPool(); pool.setFactory(new SkilledEmployeeFactory()); pool.addObject("Java"); pool.addObject("Java"); pool.addObject("VB"); pool.addObject("C++"); System.err.println("Number of Java employees in pool: " + pool.getNumIdle("Java") + " out of total employees: " + pool.getNumIdle()); Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 15 Employee employee = (Employee)pool.borrowObject("Java"); employee.doWork(); System.err.println("Employee: " + employee); pool.returnObject("Java", employee); } } The skilled employee pool is created using the GenericKeyedObjectPool with a skilled employee factory. The pool is initialized by adding 2 Java-skilled employees, 1 VB-skilled employee, and 1 C++-skilled employee. The skills become the keys to access these employees. The employees are borrowed and returned based on the skill as the key. As you can see, using a skilled employee pool is no different from using a nonskilled employee pool, as discussed earlier. The only difference lies in using a key for every operation, whether we borrow employees from the pool or return them, adding them to the pool initially or removing them completely. The configuration parameters for this GenericKeyedObjectPool-based pool are same as the parameters for GenericObjectPool, and we won’t discuss them separately. Suffice to say that these parameters affect the pool as a whole, and not individual keyed object clusters; therefore, they’re set as a whole for the pool without using a key. In the next section, we’ll create an employee pool that automatically removes employees from the pool when they’re redundant by excess memory requirements. Creating a redundant employee pool In this section, we’ll explore a SoftReferenceObjectPool-based employee pool. This pool implementation is useful for storing objects that can be destroyed by the system garbage collector when memory available to a Java program is in short supply. This allows memory to be allocated for other memory- intensive operations by freeing these objects. The objects are stored in the pool by wrapping them in a java.lang.ref.SoftReference object. Listing 9.9 shows a modification to the Employee class. We have introduced an ID number for each employee, so they can be differentiated from each other (only for internal demonstration purposes). In all other aspects, different employee objects are exactly the same when they’re borrowed from the pool. Because of this, the ID number is a read-only property; the toString() method has been changed to show this ID along with the name. Finally, the finalize method has been added to this class, to provide notification when the garbage collector removes an object. Listing 9.9 Modified Employee class, with an ID number property and a finalize method package com.manning.commons.chapter09; public class Employee { private static int base = 0; private int id; private String name; public Employee() { this.id = base++; } Base employee ID Unique employee ID Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF public int getId() { return this.id; } public String getName() { return this.name; } public void setName(String name) { this.name = name; } public void doWork() { // does nothing } public String toString() { return ("Id: " + this.id + ", Name: " + this.name); } public void finalize() { System.err.println("Employee " + toString() + " made redundant"); } } Listing 9.10 shows the code for the SoftReferenceObjectPool-based pool. Employees are made redundant if memory is in short supply. Listing 9.10 SoftReferenceObjectPool-based pool package com.manning.commons.chapter09; import java.util.HashMap; import org.apache.commons.pool.impl.SoftReferenceObjectPool; public class TestRedundantObjectPool { public static void main(String args[]) throws Exception { SoftReferenceObjectPool pool = new SoftReferenceObjectPool(new EmployeeFactory(), 5); try{ System.err.println( "Number of employees in pool: " + pool.getNumIdle()); c Create pool with 5 employees Finalize called on object destruction d Confirm pool size Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 17 Employee employee = (Employee)pool.borrowObject(); System.err.println("Borrowed Employee: " + employee); employee.doWork(); pool.returnObject(employee); employee = null; HashMap map = new HashMap(); System.err.println("Running memory intensive operation"); for(int i = 0; i < 1000000; i++) { map.put(new Integer(i), new String("Fred Flintstone" + i)); } }catch(OutOfMemoryError e) { System.err.println("Borrowed employee after OutOfMemory: " + pool.borrowObject()); System.err.println("Error: " + e); } } } c This pool is created using the original Employee factory. Recall that SoftReferenceObjectPool is the only pool that lets you initialize the pool with objects; therefore we utilize this functionality to create this pool with five employees. d This statement confirms the number of employees in the pool that are currently available to be borrowed. e These statements confirm that this pool can be used just as we would use other implementation-based pools. f Before we discuss this notation, look at the following snippet, which shows the result of running the code in listing 9.10: Number of employees in pool: 5 Borrowed Employee: Id: 4, Name: null Running memory intensive operation Employee Id: 4, Name: null made redundant Employee Id: 0, Name: null made redundant Employee Id: 1, Name: null made redundant Employee Id: 2, Name: null made redundant Employee Id: 3, Name: null made redundant Borrowed employee after OutOfMemory: Id: 5, Name: null Error: java.lang.OutOfMemoryError As soon as we start the memory-intensive operation, the garbage collector begins cleaning up the employee objects, as you can see by the messages that are printed when the finalizer method is called on each employee in the pool. All five employees are made redundant, because all these employees are softly reachable even though we borrowed employee 4 and made it do work. By setting the borrowed employee’s reference to null, the employee is no longer strongly reachable; therefore, it’s cleaned up as well. As a test, try commenting out this line: This object won’t be cleaned up e Normal pool usage f Mark for garbage collection g Simulate heavy load Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF because our application holds a live reference to it, as shown in the following output (after commenting out this line and compiling/running the code): Number of employees in pool: 5 Borrowed Employee: Id: 4, Name: null Running memory intensive operation Employee Id: 2, Name: null made redundant Employee Id: 0, Name: null made redundant Employee Id: 1, Name: null made redundant Employee Id: 3, Name: null made redundant Borrowed employee after OutOfMemory: Id: 4, Name: null Error: java.lang.OutOfMemoryError All employees except employee 4 are cleaned up and made redundant. g Here we simulate heavy activity on the application, which increases memory requirements and ultimately results in an OutOfMemory error. When an employee is borrowed from the pool after an OutOfMemory error has occurred, the pool first tries to return employees that are still available. If no employee is available, the pool creates a new employee and returns it. If all the borrowed employee’s references were cleared up by setting these references to null before the memory intensive operation began, chances are that the employee returned is a new employee, as can be seen from the first example output (a new employee with id 5 is created and returned). However, if these employees weren’t cleaned up, the employee returned was previously created, as you can see in the second output (employee 4, who was borrowed previously, is returned). Note: To understand softly reachable objects and how the garbage collector operates with these objects, refer to the API documentation for the java.lang.ref package. This brings our discussion of the Pool component to an end. Let’s move onto the DBCP component, which relies on the Pool component for its pooling engine. 9.2 The DBCP component DBCP, or DataBase Connection Pool, is the Commons component that builds on the Pool component to provide a connection-pooling mechanism that lets you connect to relational databases. The fact that it builds on the Pool component provides you with a flexible way of managing these connections. In the context of this module, this also lets us focus on showing you how to use it by presenting concrete examples, rather than spending too much time learning its internals. We’ll start with a review of the DBCP API and then jump into some code. DBCP implements the principles of object pooling for the specific case of managing connection objects to a database. Database connections are expensive objects in terms of system memory and resources that they require, because they represent a physical connection over a network; therefore, they’re ideal candidates for pooling. DBCP grew out of a need to consolidate the various connection pools that existed in the early days of open source Java and was tightly integrated with development of the Tomcat Servlet Engine. Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 19 9.2.1 The DBCP API DBCP provides support for both java.sql.Driver- and javax.sql.DataSource-based connection pools. If you don’t know the difference between these two ways of managing connections, read the sidebar “DriverManager, Driver, and DataSource: the many ways to a database.” DriverManager, Driver, and DataSource: the many ways to a database The Driver and DriverManager classes together form one of two ways to connect to a database in Java. The Driver interface in the java.sql package defines the interface that database-specific driver providers implement to allow connectivity to databases. The Driver class is registered with the DriverManager class when it’s loaded. When a request for connection is made to the DriverManager class, it tries to load as many drivers as possible that have been registered with it. Think of the Driver interface as the blueprint for making connections, whereas the DriverManager class is the service that manages these drivers. You don’t get a connection directly from a Driver class; instead, a request is made to the DriverManager class for a connection using the getConnection method. It’s up to the DriverManager class to give you an actual connection using any of the Drivers registered with it. JDBC 2.0 introduced a new way of connecting to databases that made the task of managing connections a separate entity from the application code. It utilized the Java Naming and Directory Interface (JNDI) to access datasources, which have since become the preferred way to connect and manage database connectivity. This is primarily because application code doesn’t need to know the details of database drivers and their locations when a datasource is used. All that an application needs to know is the logical name of the datasource in a JNDI context. The application code then gets its connections from this datasource, much as it would from a DriverManager. The difference lies in the fact that a DriverManager is still part of the application code, whereas the datasource is implemented and maintained externally. This allows you to change the properties of the driver used to connect to the database without changing the application code. The connection objects obtained through the basic DataSource and DriverManager classes aren’t different from one another. They can differ, however, if the datasource participates in connection pooling or distributed transactions. The datasource is an interface, and, as with the Driver interface, implementations are provided by third-party database driver manufacturers. If the basic DataSource interface is implemented, the connections it provides are no different from those obtained from the DriverManager class. ConnectionPoolDataSource and XADataSource, on the other hand, provide PooledConnection and XAConnection Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF DriverManager, Driver, and DataSource: the many ways to a database (continued) connections, respectively, which automatically participate in connection pooling and distributed transactions with the help of a middle-tier manager utility like DBCP. The Tomcat servlet engine uses DBCP as the default connection-management utility. Other application servers and servlet engines provide their own managers. Figure 9.2 shows the domain objects in a database connection environment and how they interact with each other. Figure 9.2 How DBCP fits in the database access environment The DBCP API is divided into four packages. Let’s explore each of these packages to see how they fit in the overall picture. The org.apache.commons.dbcp package This is the main package of DBCP. It provides the primary DBCP classes like PoolingDriver, PoolingDataSource, and BasicDataSource. BasicDataSource is the most important class in the DBCP package, and it’s the class we’ll use for most purposes. Tomcat uses it internally to configure the connection pooling. In addition to these basic classes, this package provides classes for tracking and maintaining abandoned connections. These classes are simple wrappers around existing classes; for example, DelegatingResultSet is a wrapper around ResultSet. Delegating classes wrap around existing classes after providing hooks for tracking the actual usage of these classes. You can use these classes in place of normal classes if you’re interested in tracking your database connection code. Connections that are pooled in DBCP are created using the PoolableConnectionFactory. It implements lifecycle methods for creating database connections by implementing the methods of the PoolableObjectFactory (from the Pool component). The connections that it creates are PoolableConnections, which are delegating connections that are returned to the pool when closed by the application code. Unfortunately, it gets slightly more complicated than that, because the Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 21 PoolableConnectionFactory uses a ConnectionFactory to create the underlying connections. ConnectionFactory is an interface, and it has three implementations based on the type of connections you’re making: DataSourceConnectionFactory, DriverConnectionFactory, and DriverManagerConnectionFactory. If all this sounds confusing, it is. Look at figure 9.3, which shows part of the way in which this API and the classes we’ve have just talked about are used. Figure 9.3 Using the classes in the org.apache.commons.dbcp package If all the steps required to create a connection pool seem too confusing, you can try to re-create the same steps using an XML configuration file. The Java Object Configuration Language (JOCL) specified in the org.apache.commons.jocl package is used to read an external XML file containing the details of your required connection pool. The org.apache.common.jocl package Curiously, this package isn’t under the DBCP namespace. It defines two classes, JOCLContentHandler and ConstructorUtil. JOCLContentHandler is the main class and is a content handler for JOCL. JOCL is a simple XML syntax for creating arbitrary Java objects from XML configuration files. It’s a convenient language (in XML) for creating objects, setting parameters, calling methods, and the like. Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF For example, to create an object of type java.awt.Frame, you list the following in a .jocl file: This creates an object of type Frame when this file is read via the JOCLContentHandler using the default Frame constructor. It’s possible to pass parameters by nesting them within the parent element. Thus, you can create a Frame with the title “Hello World”: You may notice that JOCL seems to do the same thing that Digester (module 4) does. In fact, JOCL is supposed to eventually be replaced by Digester. (For more information about JOCL, read the API description for the JOCLContentHandler class.) The DTD for the JOCL language is as follows: Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 23 JOCL is used in DBCP to read configuration values for connection pools from a .jocl file. This file must be in your application CLASSPATH, and the PoolingDriver class reads the details of the connection pool from it when a connection is requested from it. The name of this file is used to reference it. For example, if you decide to call this file myDataSource.jocl in your code, myDataSource will be used as the name by which it’s referenced. You’ll see an example of using JOCL later in this module; however, next we’ll look at the specialized datasources in the DBCP package. The org.apache.commons.dbcp.datasources package Whereas the package org.apache.commons.dbcp contains some basic implementations of datasources, the package org.apache.commons.dbcp.datasources contains some specialized versions of datasources. The datasources in this package are perfect for deployment in an application server. The reason is that these datasources are created keeping in mind applications where there may be multiple users not only per application, but also across applications. This allows you to configure the settings of connection pools based on either an individual user, or a group of users. There are two specialized pools in this package along with a base class from which both these pools are derived. The base class is InstanceKeyDataSource, and the classes are PerUserPoolDataSource and SharedPoolDataSource. Both pools are supported by their respective factory classes: PerUserPoolDataSourceFactory and SharedPoolDataSourceFactory. As the name suggests, PerUserPoolDataSource, is configurable on a per-user basis. This class provides two sets of methods for each configurable property. For example, to find the number of idle connections within this pool for a single user, you use the getNumIdle(String username, String password) method. To find the number of idle connections for the default connection pool, you use getNumIdle() method instead. This datasource maintains its pools within a HashMap with the username as the key to each pool. The SharedPoolDataSource is used to share the same connection pool between different users on an application server. The users are distinguished based on their usernames. However, unlike PerUserPoolDataSource, no separate configuration can be done on a per-user basis. The org.apache.commons.dbcp.cpdsadapter package A JDBC driver provider normally provides implementations of the three datasource interfaces (DataSource, ConnectionPoolDataSource, and XADataSource) that represent actual connections to databases. However, it’s possible for driver providers to miss providing implementations of ConnectionPoolDataSource and XADataSource and only provide implementations for the basic DataSource and basic DriverManager classes. If the JDBC driver for your database doesn’t contain implementations for the missing interfaces, how do you go about creating ConnectionPoolDataSources using DBCP? The answer lies in this package’s sole class, DriverAdapterCDPS. This class acts as a crossover class for DriverManager-based driver implementations. You can use it in place of the missing ConnectionPoolDataSource interface implementation, because it implements this interface as well as javax.naming.Referenceable. The second interface allows it to use itself for referencing purposes in a JNDI environment. Later in this module, you’ll see an example of using this class. 9.2.3 DBCP in action To begin our examples of DBCP in action, we’ll introduce the BasicDataSource class, which provides a one-stop shop for most cases. Listing 9.11 shows an example of using the BasicDataSource class. After Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF this, you’ll see an example of how to use DBCP directly. BasicDataSource provides a convenient way to create datasources; however, nothing stops you from doing the basics yourself. Note: To run any of these examples, you need to make sure you have the commons-collections.jar, commons-dbcp.jar, and commons-pool.jar libraries in your CLASSPATH. Further, you’ll require the database driver classes in the CLASSPATH, along with access to a database. We’ve used MySQL as the database and Connector/J as the database driver in these examples. You can run the examples on any database, provided that you have the drivers to access the database and that the required tables are added in a database called commons. Listing 9.11 BasicDataSource example: the simplest way to use DBCP import java.sql.Connection; package com.manning.commons.chapter09; import org.apache.commons.dbcp.BasicDataSource; public class BasicDataSourceExample { public static void main(String args[]) throws Exception { BasicDataSource bds = new BasicDataSource(); bds.setDriverClassName("com.mysql.jdbc.Driver"); bds.setUrl("jdbc:mysql://localhost/commons"); bds.setUsername("root"); bds.setPassword(""); bds.setInitialSize(5); Connection connection = bds.getConnection(); connection.close(); } } As you can see, it’s easy to create a BasicDataSource that pools connections to a database. All you need to do is set the basic properties of a single connection, and the BasicDataSource manages them in a pool for you. c Here, a BasicDataSource is created without any properties. You can instead use a factory class called BasicDataSourceFactory to create an instance of the BasicDataSource by using the method createDataSource(Properties properties). This method accepts a fully configured Properties class that contains the acceptable properties of the underlying connection. d Since we aren’t using the factory class, we need to set the properties ourselves here. These are the bare minimum properties that must be set. They include the name of the Driver class that is used to connect to the database, the connection string, and the username/password combination. e Here, we set the initial size of the pool to 5. This preallocates the pool with five connection objects. At this stage, you may wonder about the other properties of the pool. For example, what happens to the other essential properties of the pool, like maximum idle and active objects, maximum waits c Create BasicDataSource d Configure BasicDataSource g Close connection e Set initial size f Retrieve connection Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 25 for evicting idle connections, and so forth? Well, since BasicDataSource uses the GenericObjectPool from the Pool component as its basic pooling mechanism, it uses the default values provided by it. Refer to table 9.1 for a full list of these default values. f Now that the pool has been created, it can be used to do the basic task of generating connections. We get a connection by calling the method getConnection(). Note, however, that the connections returned are instances of the org.apache.commons.dbcp.PoolableConnection class (and not instances of javax.sql.PooledConnection). g A call to the close method of the PoolableConnection class returns the connection to the pool from which it came. This releases the particular connection so it can be recycled. To really close this connection, call the reallyClose() method. However, do this when you’re absolutely sure that you don’t want this connection to be recycled, because it will be closed at the physical level. Now that you know how to do connection pooling the easy way by using the BasicDataSource class, let’s look at how to do it the hard way. The hard way gives you more control over the basics, like the internal pool used, the connection factory used, and whether a Driver or a datasource is used. Listing 9.12 shows a complete example. Listing 9.12 Doing the connection pooling basics ourselves package com.manning.commons.chapter09; import java.sql.Connection; import java.util.Properties; import org.apache.commons.dbcp.ConnectionFactory; import org.apache.commons.dbcp.PoolingDataSource; import org.apache.commons.dbcp.DriverConnectionFactory; import org.apache.commons.dbcp.PoolableConnectionFactory; import org.apache.commons.pool.impl.GenericObjectPool; public class ConnectionPoolBasics { public static void main(String args[]) throws Exception { GenericObjectPool gPool = new GenericObjectPool(); Properties props = new Properties(); props.setProperty("User", "root"); props.setProperty("Password", ""); ConnectionFactory cf = new DriverConnectionFactory(new com.mysql.jdbc.Driver(), "jdbc:mysql://localhost/commons", props); PoolableConnectionFactory pcf = new PoolableConnectionFactory(cf, gPool, null, null, false, true); e Create ConnectionFactory d Properties for ConnectionFactory c Create object pool f Create PoolableConnectionFactory Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF PoolingDataSource pds = new PoolingDataSource(gPool); for(int i = 0; i < 5; i++) { gPool.addObject(); } Connection conn = pds.getConnection(); // do some work with the connection conn.close(); } } If you compare figure 9.3 with this code listing, you’ll see that the code follows the exact process described in that figure. In fact, the BasicDataSource that we discussed in listing 9.11 uses this same process internally. Let’s go through this process: c To create a connection pool from scratch, you first need to create an object pool. The object pool in this case is based on the GenericObjectPool class provided in the org.apache.commons.pool.impl class. This is the most common object pool and serves our purposes well. de Next, we need to create an instance of the ConnectionFactory interface. A ConnectionFactory instance is used to create the physical connections to a database. Recall that this can be done three ways: You can create physical connections by using the java.sql.DriverManager class, the java.sql.Driver class, or the javax.sql.DataSource class. Hence, DBCP provides you with implementations of all these three mechanisms as factory classes of the ConnectionFactory interface. In this listing, we use DriverConnectionFactory. It requires three parameters: the name of the database driver provider class, the database connection URL, and a properties object that contains, at a minimum, the username and password for making the connections. To use the DriverManagerConnectionFactory instead, you would provide only the database connection URL and the properties object. However, the database driver must still be loaded, because the DriverManager interface is a wrapper for actual drivers. The following code snippet shows how a DriverManagerConnectionFactory can be used: Class.forName("com.mysql.jdbc.Driver"); DriverManagerConnectionFactory cf = new DriverManagerConnectionFactory( "jdbc:mysql://localhost/commons", props); You can also use a DataSourceConnectionFactory: To do so, provide a DataSource object to this factory (and a username and password, if required) that points to the actual database connections. f A PoolableConnectionFactory provides the lifecycle methods for working with connection objects. For example, this factory provides implementations of methods for activating and passivating connections, for validating connections, and for creating new connections (which it delegates to a ConnectionFactory as discussed earlier). To do all this, it needs to know the object pool that is being used to keep the connections in and the actual connection-making factory that makes the connections. Both of these values are provided as the first two parameters. h Initialize pool g Create datasource i Use connection Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 27 The next two parameters are set to null in this case. The first parameter accepts an instance of a KeyedObjectPoolFactory (discussed later). The second parameter accepts a validation query String. A validation query is used to validate that a database connection is up and running and can return data. A simple Select statement that returns at least one row can be a valid validation query. If you turn on validation for the internal object pool by setting the testOnBorrow, testOnReturn, or testWhileIdle flag, each connection will be validated using this query. Connections that fail validation, either because the connection has been closed at the physical level or the validation query doesn’t return even a single row, are dropped from the pool. The first parameter that accepts a KeyedObjectPoolFactory, for which we passed a null value, is interesting. Why do you need a factory for creating keyed object pools? Well, it’s required if you want the connection pool to manage and pool prepared statements. Pooling prepared statements saves you the round-trip network call from an application server to the database server by caching the statement on the client side. The query becomes the key for the pool; and by using an instance of the KeyedObjectPoolFactory interface, you can let your base connection pool know that you’re interested in pooling your prepared statements. However, the only implementation offered in the Pool package is the GenericKeyedObjectPoolFactory class. This class in turn requires that you pass it an instance of the KeyedPoolableObjectFactory interface. Where do you get an instance of a KeyedPoolableObjectFactory that will create and manage prepared statements around a connection object? From the PoolingConnection class. This class serves two purposes: It not only acts like a wrapper for connection objects (by implementing java.sql.Connection), but at the same time, by implementing the KeyedPoolableObjectFactory interface, it also acts as a factory for managing prepared statements. However, because this class is a wrapper around connection objects and requires a connection object around which the prepared statements are pooled, you probably won’t need to use it directly. Instead, you should let the PoolableConnectionFactory manage their creation internally. This allows each pooled connection to have its own pool of prepared statements. Thus, by adding the following line of code KeyedObjectPoolFactory kopf = new GenericKeyedObjectPoolFactory(null, 8); and then adding the resulting factory as a parameter to the PoolableConnectionFactory class, you can pool prepared statements around their connection objects. By keeping it null, as we did, prepared statements won’t be pooled. The last two parameters in the PoolableConnectionFactory class are boolean flags. The first indicates whether the connections created by this factory should be read-only. The second sets the status of the default auto-commit flag. A true value for this flag indicates all connections by default are set to auto-commit all transactions. g Having created the factories and the base pool, we’re ready to create the PoolingDataSource that will automatically pool connections. Notice that it takes as a parameter the object pool that was created in c. This pool knows how to create connections, manage them, and evict those that fail validation, as specified in the previous steps, using the factories that were created and the properties that were set. h Our pool is still without any initial connections, and this step adds five of them. Note that we wouldn’t have been able to add these initial connection objects before the factories were set in f. After all, the pool wouldn’t have known how to create these connections without the benefit of the factories. i Finally, we can get connections from the datasource we just created. As expected, closing the connection returns it to the pool rather than closing it completely at the physical level. Licensed to Tricia Fu 28 JAKARTA COMMONS ONLINE BOOKSHELF The two examples you’ve seen so far let you get connections from a datasource, which internally pools connections. However, you may not want to use a datasource for pooling your connections and may prefer the connections from a DriverManager class instead (which pools its connections). You’ll see an example of this in the next section. PoolingDriver vs. PoolingDataSource Recall that you can get connections from a DriverManager by calling the DriverManager. getConnection(String connectionURL) method, as specified in the java.sql package. However, the connections from the default implementation of java.sql.DriverManager aren’t pooled. It would be nice if you could get an implementation of DriverManager that does this. DBCP allows you to roll your own version of the DriverManager class (or rather, the Driver class) that lets you do this. Listing 9.13 shows the steps in creating a PoolingDriver class, which, when registered with the DriverManager, automatically pools its connections. The basic steps in creating the pool are the same as outlined in listing 9.12. The difference lies in how the pool is registered and accessed. Listing 9.13 Using a DriverManager/Driver combination to pool connections package com.manning.commons.chapter09; import java.sql.Connection; import java.sql.DriverManager; import java.util.Properties; import org.apache.commons.dbcp.PoolingDriver; import org.apache.commons.dbcp.ConnectionFactory; import org.apache.commons.dbcp.DriverConnectionFactory; import org.apache.commons.dbcp.PoolableConnectionFactory; import org.apache.commons.pool.impl.GenericObjectPool; public class ConnectionPoolDriverBasics { public static void main(String args[]) throws Exception { GenericObjectPool gPool = new GenericObjectPool(); Properties props = new Properties(); props.setProperty("User", "root"); props.setProperty("Password", ""); ConnectionFactory cf = new DriverConnectionFactory(new com.mysql.jdbc.Driver(), "jdbc:mysql://localhost/commons", props); PoolableConnectionFactory pcf = new PoolableConnectionFactory(cf, gPool, null, null, Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 29 false, true); PoolingDriver pd = new PoolingDriver(); pd.registerPool("driverexample", gPool); for(int i = 0; i < 5; i++) { gPool.addObject(); } Connection conn = DriverManager.getConnection( "jdbc:apache:commons:dbcp:driverexample"); // do some work with the connection conn.close(); } } c As you can see, the steps before this point are exactly the same as those shown in listing 9.12. (Of course, you can make variations for the connection factory you want; we’ve also added import statements for the Driver and DriverManager classes.) A PoolingDriver is created at this point, and the pool created in the earlier steps is registered with this driver. The pool is given a name, and this is the name that becomes part of the database connection URL. This name is also used to distinguish between different pools within the same driver. Thus, you could associate multiple pools within the same driver, with differing properties, and separate them using this name. When you want to access a connection from a specific pool, you ask for it by name, as shown next. d Now we can get pooling connections from the DriverManager. Notice the connection URL that is used. The jdbc:apache:commons:dbcp part is fixed and is the DBCP URL. Thus, if you registered two or more drivers (or one driver with multiple pools), you can access each one by first prefixing the DBCP URL and then adding the name of pool/driver. In this case, we registered the pool with the name driverexample, and that is the name we use along with the DBCP URL from which to get pooled connections. DBCP lets you read the configuration properties for the pool from an external file as well, instead of only from the application code. We’ll show you how to do this in the next section. Using JOCL If setting the details of a connection pool-based Driver/DriverManager in code seems like a lot of work to you, then there is hope. If you’re using an application server/servlet engine, you can let the server handle the connection pooling for you. All you need to do is to specify the details of the pool in a configuration file. We’ll show you how in the next section. However, if you aren’t using a server for your application, you can still escape the drudgery of writing the connection pool code by specifying the details in a .jocl file. This solution is available only if you’re using a PoolingDriver/DriverManager combination as described in the previous section. Using a .jocl file is relatively simple. All the configuration parameters for creating the pool are set in the file whose syntax was described in section 9.2.2. The name that you give this file is important. Of course, the file extension should be ,jocl, but the name before the extension should be name you want to use for locating c Create PoolingDriver d Retrieve pooling connections Licensed to Tricia Fu 30 JAKARTA COMMONS ONLINE BOOKSHELF connections from the DriverManager. Continuing with the example from the previous section, if we decide to give this file the name driverexample.jocl, we get connections from it in code by calling DriverManager.getConnection("jdbc:apache:commons:dbcp:driverexample"); or DriverManager.getConnection("jdbc:apache:commons:dbcp:/driverexample") as the case may be, depending on where we save the jocl file. The PoolingDriver class uses the Class.getResourceAsSteam(poolName + ".jocl") method to locate this file, so be conscious of where the file is located. The .jocl file should contain the definition of a PoolableConnectionFactory, following which it should list all the parameters accepted by this factory in the order in which they appear in the PoolableConnectionFactory constructor. So, for example, the first parameter is the constructor definition for a connection factory, with all its parameters listed in order. Note that you must list all the parameters, even if they’re null. Listing 9.14 shows a complete .jocl file, which mirrors the connection pool created in listing 9.13. Listing 9.14 JOCL file that mirrors the connection pool created in listing 9.13 To use the connection pool described by this .jocl file, we need the following code snippet. Class.forName("org.apache.commons.dbcp.PoolingDriver"); if(null == System.getProperty("org.xml.sax.driver")) { System.setProperty("org.xml.sax.driver", "org.apache.xerces.parsers.SAXParser"); } Connection conn = DriverManager.getConnection( "jdbc:apache:commons:dbcp:/driverexample"); // do some work with the connection conn.close(); // returns to pool Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 31 Since a pooling driver is being used, it must be loaded for the DriverManager to know that it can be accessed. Thus the first line of code loads a generic pooling driver. However, when a connection is requested from this pooling driver using the name of the actual driver, it loads the .jocl file and parses it. The first two lines of code are only required the first time a connection is accessed. The org.xml.sax.driver system property can be set by any means. Here we do it in code. You may choose to pass it at runtime as a command-line parameter. (Note that you’ll need to have the Xerces-based SAXParser for this to work, because the Java-supplied SAX parser [in the javax.xml.parsers package] fails while parsing the file.) Application server configuration When you use DBCP in a web-based environment (or any environment where an application server/servlet engine is used), you can let the server manage the pool. The Tomcat servlet engine makes it even easier for you by automatically using DBCP as the default connection pooling mechanism for your applications. Different servers have different ways of configuring the connection pool; but all servers provide a way for you to do it. Some, like Tomcat, support this by having the classes for DBCP already in their path, whereas in others you’ll have to install DBCP as an addition to an existing connection pool mechanism. Listing 9.15 shows a minimal server.xml listing for Tomcat. It shows you how to configure a pooling datasource for a web application called dbcp-example. Listing 9.15 Minimal but working server.xml for DBCP factory org.apache.commons.dbcp.BasicDataSourceFactory driverClassName com.mysql.jdbc.Driver url Licensed to Tricia Fu 32 JAKARTA COMMONS ONLINE BOOKSHELF jdbc:mysql://localhost/commons username root password maxActive 5 The relevant parts in this server.xml file define the resource named jdbc/dbcpExample. Tomcat knows that a resource defined with a type of javax.sql.DataSource needs to be made available in the java:/comp/env/jdbc naming context. It also sets up this datasource with the properties that are defined for it as part of the ResourceParams element tag. These properties dictate that a BasicDataSourceFactory class be used for creating BasicDataSources that automatically participate in connection pooling, along with other properties. To use this connection pool, we need to make an entry in the web.xml file of the dbcp-example application, as shown in listing 9.16. Listing 9.16 web.xml for dbcp-example application Resource reference for a connection pooling datasource jdbc/dbcpExample javax.sql.DataSource Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 33 Container Finally, this pool can be accessed anywhere in our code. As an example, the following JSP file in the dbcp- example web application uses this pool by looking up the relevant datasource in the java:/comp/env/jdbc/dbcpExample context. Connections that are accessed from this datasource are retrieved from the underlying BasicDataSource and are of type PoolableConnection, as should be expected. Closing the connections returns them to the pool: <%@ page import = "javax.naming.*, javax.sql.*, java.sql.*" %> <% Context initContext = new InitialContext(); DataSource ds = (DataSource)initContext.lookup("java:/comp/env/jdbc/dbcpExample"); Connection conn = ds.getConnection(); // do something with this connection conn.close(); // returns to pool %> In the next section, we’ll look at how to adapt older driver implementations that don’t provide datasource support. We’ll also discuss how to provide different datasources for different levels of users using the specialized datasource support in DBCP. Specialized datasources The specialized datasource support in DBCP is provided in two packages: org.apache.commons. dbcp.datasources and org.apache.commons.dbcp.cpds. Recall that the javax.sql package contains an interface definition called ConnectionPoolDataSource. Does this mean that connections created using this datasource are automatically part of a connection pool and that you don’t need DBCP if your physical driver provider (like Connector/J for MySQL) provides an implementation? No. This interface is a factory for creating PooledConnections (javax.sql.PooledConnection). These connections provide lifecycle event notifications when they participate in connection pooling but are themselves not pooled. The interface still needs a third-party connection pool manager like DBCP to provide pooling in conjunction with the implementation of ConnectionPoolDataSource from your driver provider. But what happens if the driver provider doesn’t include such an implementation and you still want to use pooling based on this interface? DBCP provides an answer in the DriverAdapterCPDS class. Before we look at an example of how to use this class, let’s consider another issue. It’s a common practice for application developers to provide multiple datasources for different levels of users. For example, administrative users, who are few in number, may access an application at infrequent times. There are many more normal users of an application, and they access your application more frequently. It makes sense to provide different connection pools for these different needs; you want higher maximum active and idle connections for normal users than administrative users. It would be nice if you could use the same pool- management interface to maintain these two different needs. DBCP provides an answer again: Licensed to Tricia Fu 34 JAKARTA COMMONS ONLINE BOOKSHELF PerUserPoolDataSource. Listing 9.17 shows an example of server.xml for Tomcat in which the PerUserPoolDataSource is created. Listing 9.17 server.xml for creating a PerUserPoolDataSource factory com.mysql.jdbc.jdbc2.optional.MysqlDataSourceFactory serverName localhost databaseName commons port 3306 explicitUrl false user root d Define datasource parameters c Define datasource Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 35 password password factory org.apache.commons.dbcp.datasources.PerUserPoolDataSourceFactory dataSourceName java:comp/env/jdbc/dbcpExampleCPDS Phew! This server.xml doesn’t look easy. Let’s break it down. Essentially, it contains two resource definitions for the dbcp-example web application. e defines the resource name jdbc/dbcpExample, and f defines its parameters. This is similar to listing 9.15, with the important difference that this listing has no definition for the actual physical connection; rather, there is pointer to the datasource using the keyword dataSourceName parameter. This points to a datasource as a JNDI reference in the same web application at java:comp/env/jdbc/dbcpExampleCPDS. This datasource is defined in c, and its parameters are defined in d. When you look at the definition and parameters of the dbcpExampleCPDS reference, you’ll notice that they look very different from those you’re used to. This is because we’re using the Connector/J-provided ConnectionPoolDataSource implementation called com.mysql.jdbc. jdbc2.optional.MysqlConnectionPoolDataSource. It has its own com.mysql.jdbc.jdbc2. optional.MysqlDataSourceFactory factory and parameters, which are named slightly differently than before (instead of username, we have to use user). f Define resource name parameters e Define resource name for later use Licensed to Tricia Fu 36 JAKARTA COMMONS ONLINE BOOKSHELF Note: You must type all the parameters exactly as they’re shown here. MySQL’s Connector/J implementation of ConnectionPoolDataSource is very unforgiving if you omit even one of these parameters (or mess up the case of a parameter, like using explicitURL for explicitUrl). This still doesn’t set up our PerUserPoolDataSource, however. Although the basic pool is set up, we still need to configure it for each set of users. This needs to be done in code only once. Notice the emphasis: You aren’t allowed to change the settings of an already-set PerUserPoolDataSource after a connection has been requested from it. Listing 9.18 shows an example of a setup.jsp file; it’s used to create two sets of pools in the PerUserPoolDataSource, one for an admin user (using the root username) and another for a normal user with different settings. Before you run this JSP, make sure that the web.xml from listing 9.16 is in the WEB-INF directory of the dbcp-example application. Listing 9.18 setup.jsp that sets up PerUserPoolDataSource <%@ page import = "javax.naming.*, javax.sql.*, java.sql.*, org.apache.commons.dbcp.datasources.PerUserPoolDataSource" %> <% Context initContext = new InitialContext(); PerUserPoolDataSource ds = (PerUserPoolDataSource)initContext.lookup("java:/comp/env/jdbc/dbcpExample"); ds.setPerUserMaxActive("root", new Integer(5)); ds.setPerUserMaxActive("normal", new Integer(15)); out.println("Done"); %>
DBCP Test 2 The PerUserPoolDataSource is extracted from a lookup in the initial context. It’s then used to set up the properties for each set of users. In this case, we’re just setting the limit on the maximum number of active connections. Finally, the following code snippet shows how to get connections from this datasource in code for each set of users. Notice that we have to pass the username and password for each connection. These usernames and passwords correspond to database username and passwords for each user: Connection conn = ds.getConnection("root", "password"); Connection connNormal = ds.getConnection("normal", "normal"); Now, let’s get back to the issue we started this section with: using the DriverAdapterCPDS class for instances where there is no implementation of the ConnectionPoolDataSource interface. This is now easily accomplished by changing the first resource reference and its parameters in listing 9.17. The resource’s type is changed to Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 37 and the resource parameters are changed to factory org.apache.commons.dbcp.cpdsadapter.DriverAdapterCPDS user root password password driver com.mysql.jdbc.Driver url jdbc:mysql://localhost/commons The rest of the application remains the same. Voila! We have transformed our application from using the Connector/J-provided ConnectionPoolDataSource implementation to a Connector/J + DBCP-provided Driver + Adapter implementation. This allows us to use DBCP in instances where there is no support for ConnectionPoolDataSource. DBCP contains one more specialized datasource: SharedPoolDataSource. You can roll your own by extending the base class for all specialized datasources, InstanceKeyDataSource. 9.3 Summary This module covered two Commons components: Pool and DBCP. We discussed these two together because DBCP derives from Pool. DBCP is a specific implementation of the Pool component, and it made sense to show these together. The Pool component allows arbitrary objects to be pooled. This module covered the grounds for pooling objects and showed why pooling is required. It also described the difference between object pooling and garbage collection. We explained the Pool API in detail and discussed the basic requirements of any pooling application. We rounded off the Pooling section with detailed examples showing how to use and extend the Pool API. Licensed to Tricia Fu 38 JAKARTA COMMONS ONLINE BOOKSHELF We discussed the DBCP component next. Because we covered much of the theory behind pooling while discussing the Pool component, this section dove straight in to the DBCP API. We walked through a detailed examination of the API, to help you understand the examples that followed. We provided several examples of DBCP in action covered and the whole API. The next module will cover the Codec component of the Jakarta Commons suite. Licensed to Tricia Fu MODULE 9: POOL AND DBCP: CREATING AND USING OBJECT POOLS 39 Index active and idle objects, 6 BasicDataSource, 20 connection creation, 20 CPDS adapter, 23 Database Connection Pool. See DBCP DataSource, 19 DBCP adding initial connections, 27 API, 19 configuration in Tomcat, 31 connection creation, 20 CPDS adapter, 23 creating a connection pool by hand, 25 creating a per-user connection pool, 36 externalizing connection pool properties, 29 JOCL, 21 managing prepared statements, 27 setting a validation query, 27 simple example, 23 specialized DataSources, 23 usage, 21 using a DriverManager connection pool, 28 using connection factories, 26 using JOCL, 29 using older Drivers to pool connections, 33 using specialized datasources, 33 using the BasicDataSource, 23 using the KeyedObjectPoolFactory for managing prepared statements, 27 DelegatingResultSet, 20 difference between DriverManager, Driver, and DataSource, 19 Driver, 19 DriverManager, 19 GenericObjectPool classes, 5 Java Object Configuration Language (JOCL), 21 keyed object pools, 4 object pool minimum methods, 3 minimum parameters, 3 object pooling need for, 1 requirements, 2 controlled access, 2 responsiveness, 2 reusability, 2 vs. the garbage collector, 2 Pool API, 3 contract classes, 3 factory for object pool creation, 4 implementation classes, 5 keyed object pool classes, 4 lifecycle classes, 3 configuration parameters, 7, 8 creating a keyed object pool, 13 creating a simple pool, 9 creating memory sensitive object pool, 15 difference between active and idle objects, 6 GenericObjectPool classes, 5 introduction, 1 keyed object pools, 4 setting initial objects, 10 SoftReferenceObjectPool, 5 StackObjectPool, 5 starting an eviction thread, 12 transitioning from an active to an idle object, 6 using GenericKeyedObjectPool, 13 using GenericObjectPool, 9 using SoftReferenceObjectPool, 15 validating objects, 11 reusable objects, 1 SoftReferenceObjectPool, 5 StackObjectPool, 5 thread pooling, 2 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Codec: encoders and decoders Vikram Goyal MODULE MANNING 10 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 10 Codec: encoders and decoders 10.1 What are encoding and decoding?............................................................................................................................ 1 10.2 Understanding the Codec algorithms........................................................................................................................ 2 10.3 Getting to know Codec ............................................................................................................................................. 9 10.4 Codec at work ........................................................................................................................................................... 14 10.5 Summary................................................................................................................................................................... 18 Index ................................................................................................................................................................................. 19 Jakarta Codec is an effort to provide implementations for various encoders and decoders that a developer is likely to encounter. At the time of this writing, encoders and decoders are provided for Base64, Hex, phonetic encodings, and URLs. It’s possible to extend this API to provide your own implementations for either the supported encodings or any other new encoding scheme. In this module, we’ll look at how to use the Codec component for various encoding/decoding scenarios. We’ll start with the basics of each supplied encoding mechanism to help you understand the underlying technology; and we’ll show an example of how to implement it using Codec. 10.1 What are encoding and decoding? If you examine the headers of an email message, a typical header you’re likely to encounter is Content- Transfer-Encoding. What is this header used for, and who puts it there? The header is added by the mail client you use to compose the message (in other words, the mail client of the message sender). It’s added so that when the mail message passes through the various Internet mail transport mechanisms, it can be identified as having been encoded. This encoded message can then pass through these systems, which may have limitations on the data or character sets that they can handle. For example, binary data can’t be transferred through these systems if it isn’t encoded in a certain way (Base64). This header also identifies to the message recipient the decoding to be applied to re-create the original message. This is the basic crux of encoding/decoding. A system encodes a piece of information for safe transmission, conciseness, or security. The receiving system decodes that information based on the encoding used and re-creates the original information. In this section, we’ll elaborate on these concepts. Digital transmission of information requires textual data to be encoded so that differing systems can agree on and understand the information being transmitted. Encoding, however, isn’t just required for textual data. Systems require information to be encoded in order to make it concise or unambiguous. Phonetic encoding of related words is an example of such encoding. The basic premise of encoding is to convert information from one form to another. As we stated earlier, this second form may be relevant for either transmitting this data over channels that are more conducive to the converted form, or it may represent a more concise representation of the original data. Information that is encoded must be marked with the rules used to do the encoding so that another system can use this marking to understand the encoding rules applied. This other system, can then apply these rules in reverse to arrive at the original information—in effect, decoding the encoded information. Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF The most prevalent form of digital encoding of textual data, and the one that you may have come across, is the American Standard Code for Information Interchange (ASCII). This encoding method was developed to represent English characters as numbers, with each letter assigned a number from 0 to 127. For example, the ASCII code for uppercase A is 65. With 26 letters in the English alphabet and room for more characters like the space and special symbols (such as $ and %), ASCII is well suited for representing and transmitting textual English information. ASCII also works well to represent Latin-based languages. However, it falls short for other languages, like Arabic, Chinese, and Hindi, which are based on separate scripts and have hundreds of characters that can’t be represented by the encoding scheme used in ASCII. The Unicode encoding scheme (http://www.unicode.org) overcomes this problem. It defines three encoding forms that allow the same data to be transmitted in 8-bit, 16-bit, or 32-bit formats. With a theoretical limit of 232 – 1, Unicode is well suited to represent all the languages of the world. 10.2 Understanding the Codec algorithms Before we delve into the Codec component, let’s look at the algorithms supported by Codec. In the next few sections we’ll look at the basics of each encoding scheme supported by Codec. 10.2.1 Base64 An email message that you type and send to a distant recipient is transported across the world through a series of Internet mail gateways that route your message based on header information. This information needs to be in a format that these gateways understand and can interpret for routing forward. However, considering the vast number of gateways around the world and the languages they work in, it’s possible that messages would be lost if they weren’t sent with headers that contain information in a standard way. It isn’t just the headers that need to be in a standard language, though; transporting images and multimedia files requires the conversion of these binary files into transport-safe versions. Toward this end, the Base64 encoding mechanism is defined in rfc2045 (http://www.ietf.org/rfc/rfc2045.txt; see module 2 for the definition of an RFC). This RFC deals with the large issues of mail transport and provides other useful information as well. Base64 is only one of the recommendations that is included as part of the Content-Transfer-Encoding header mandate; the other encoding is quoted-printable. Base64 allows you to convert binary (8-bit) data into a 7-bit short line format. This is necessary because mail transport disallows any other format. For example, you can’t transfer mail messages that are binary in format or are represented using 8-bit encoding. This would typically happen if you were transporting media files or images, which, in their natural format, are either binary or 8-bit. Also, by encoding the images and other multimedia files with Base64, you get a textual representation of these files, which may be useful when you want to store the images in an archive. Base64 works by dividing three continuous octets in the input data into sets of 6 bits each and then representing each 6 bits with the ASCII equivalent. This is better explained by an example. Consider the binary input for some arbitrary data shown in figure 10.1. Figure 10.1 Arbitrary binary input Our task is to convert this binary data into its Base64 equivalent. We start by splitting these 24 bits (3 octets = 24 bits) into groups of 6 bits each to arrive at 4 groups, as shown in figure 10.2. Licensed to Tricia Fu MODULE 10: CODEC: ENCODERS AND DECODERS 3 Figure 10.2 Our arbitrary input is divided into groups of 6 bits each. Each group of 6 bits now can be replaced with the equivalent US ASCII representation to arrive at the Base64-encoded value. This replacement is done according to table 10.1. Table 10.1 Base 64 encoding Binary input Decimal value Base64 value Binary input Decimal value Base64 value 000 000 0 A 100 001 33 h 000 001 1 B 100 010 34 i 000 010 2 C 100 011 35 j 000 011 3 D 100 100 36 k 000 100 4 E 100 101 37 l 000 101 5 F 100 110 38 m 000 110 6 G 100 111 39 n 000 111 7 H 101 000 40 o 001 000 8 I 101 001 41 p 001 001 9 J 101 010 42 q 001 010 10 K 101 011 43 r 001 011 11 L 101 100 44 s 001 100 12 M 101 101 45 t 001 101 13 N 101 110 46 u 001 110 14 O 101 111 47 v 001 111 15 P 110 000 48 w 010 000 16 Q 110 001 49 x 010 001 17 R 110 010 50 y 010 010 18 S 110 011 51 z 010 011 19 T 110 100 52 0 010 100 20 U 110 101 53 1 010 101 21 V 110 110 54 2 010 110 22 W 110 111 55 3 010 111 23 X 111 000 56 4 011 000 24 Y 111 001 57 5 011 001 25 Z 111 010 58 6 011 010 26 a 111 011 59 7 011 011 27 b 111 100 60 8 011 100 28 c 111 101 61 9 011 101 29 d 111 110 62 + 011 110 30 e 111 111 63 / 011 111 31 f -- (pad) = 100 000 32 g Using this table, our original binary data from figure 10.1 is transformed into the data shown in figure 10.3. Figure 10.3 Arbitrary data converted into Base64-encoded data Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF Rfc2045 mandates that Base64 data must not contain lines more than 76 characters in length. As input data is processed and encoded into Base64, once 76 characters are reached, a line break is appended consisting of two characters (CR and LF). A new line is started for the rest of the data; this continues until the input data is finished. Some special cases require slightly different processing. Remember that Base64 encoding is done on a group of three octets to arrive at a group of four sextets (6 bits). Thus each processing requires the presence of 24 bits. What happens when the input doesn’t contain enough bits to make up 24? Zeros are padded at the end to make a complete 6-bit integral value. For example, if the final input contains 9 bits, then 3 more 0 bits are added at the end so the input can be treated as a group of two sextets. Even though by adding 0s at the end you can form integral 6-bit groups, it isn’t guaranteed that the final group will contain a total of four sextets, as shown by our earlier 9+3 example. This problem is solved by using the padding character = which is the sixty-fifth character in the Base64 conversion table. You can see both these special cases in action with a single ASCII character like A, as illustrated in figure 10.4. Figure 10.4 Base64 encoding of character A illustrates special processing. What is 7-bit, 8-bit, or binary data? In the digital world, all data is, in effect, binary data—a series of 1s and 0s. Used in our present sense, binary means data that is represented as a series of octets without any restriction on which octets are used (octet = one 8-bit byte). Each octet has a related decimal value: For example, 0011 1100 is an octet whose decimal value is 60. Binary data is a series of octets without any restrictions on what decimal value the octets represent. Seven-bit data is also binary data. However, the restriction is that in 7-bit data, no octet whose decimal value is greater than 127 or equal to 0 can be used. So, 7-bit data will never have a 0000 0000 octet, and neither will it have any octet greater than 0111 1111 (127 decimal value). Seven-bit also has the restriction that each line of data must be terminated with a carriage return (CR, decimal value 13) and line feed (LF, decimal value 10) and that each line must not contain more than 998 octets. Thus each line is no longer than 1000 characters (1 byte = 1 octet = 1 character) when you include the CR and LF. Eight-bit data is similar to 7-bit data. The only difference is that 8-bit data allows octets whose decimal value is greater than 127. Thus 1111 1011 (decimal value 251) is a valid octet for 8-bit data. 10.2.2 Hex If you’re a computer programmer (and given that you’re reading this book, you probably are), you know what Hex is. In fact, at some point in time when you were in college, you surely did heaps of tedious exercises that converted decimals to Hex in all sorts of formats and vice versa. Well, it’s time to do a quick review. Hexadecimal (Hex for short) refers to the system of representing binary digits using the numbers 0–9 and the letters A–F. Why would you want to do this? Because it’s easier to work with hexadecimal data than a bunch of 0s and 1s. The data is human readable and takes less space. Every byte (8 bits) can be represented Licensed to Tricia Fu MODULE 10: CODEC: ENCODERS AND DECODERS 5 using two characters from the hexadecimal alphabet. In print and code, hexadecimal values are generally preceded with 0x to clearly distinguish them from decimal and octal number systems. Hexadecimal numbers have a base of 16 as compared to 10 for decimal, 8 for octal, and 2 for binary numbers. Table 10.2 shows the conversion chart between binary and hexadecimal systems and vice versa. Table 10.2 Hexadecimal to binary conversion chart Binary value Hexadecimal value 0000 0 0001 1 0010 2 0011 3 0100 4 0101 5 0110 6 0111 7 1000 8 1001 9 1010 A 1011 B 1100 C 1101 D 1110 E 1111 F Using this table, you can easily decipher that the hexadecimal value 0xA43E equals 1010 0100 0011 1110 in binary. Similarly a value of 0001 1100 translates to 0x1C in Hex. If the binary input lacks enough numbers to make a four-character value, 0s are added at left to make up for it. For example, 110011 translates to 0011 0011, which is 0x33 in Hex. 10.2.3 Soundex Suppose one day you decide to draw up your family tree. To start with, you need to know what year your great-grandfather Geoffrey Wilson was born. You end up at the US Census Bureau web site and type in his name. To your surprise, the search result has no listings for a Geoffrey Wilson in the state of Indiana for the period you suggest. You’re adamant that the period and region are right and there must be a record of your great-grandfather. What is going on? A note at the bottom of the search results suggests that although no records were found with the spelling you entered, there is a record of a Geoffrey Willson (notice the double L in Willson as opposed to the single L in Wilson) living in the area at that time. Unknown to you, your great-grandfather used a double L in his surname. This was shortened as the years went by to the current version, Wilson. No problem—you got your record, and now you can draw up that family tree. So how did the search result come up with the alternate spelling? It quite likely used the Soundex encoding algorithm. Soundex is an encoding algorithm that groups together words that sound similar but may be spelled differently. It was originally developed as an indexing system for the scenario we just described—namely, to group together similar sounding surnames for easy access. Soundex encoding is useful in any situation where the way words sound is more important than the way they’re spelled. Soundex works by converting a word to its soundex value. Each word is translated into a special code, which is composed of four characters. Similar sounding words equate to the same code. The first letter of the code is always the first letter of the word that is being converted. Thus the code for Wilson starts with the letter W. The remaining three characters of the code are numbers, which are based on the rules shown in table 10.3. Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF Table 10.3 Soundex coding guide Letters of the alphabet Number assigned B, F, P, V 1 C, G, J, K, Q, S, X, Z 2 D, T 3 L 4 M, N 5 R 6 As you can see, some of the letters of the alphabet are missing from this table, because they aren’t required for Soundex purposes: All the vowels (A, E, I, O, U) and the letters H, W, and Y are discarded. To convert a word into its soundex value, you begin by writing down the first letter of the word. Then, for each successive letter in the word, consult table 10.3 and write down the corresponding number. Do this until you have a four-character code (one letter and three numbers); if there are any letters remaining in the word, discard them. Using these rules, the code for Wilson is W425. Similarly, the code for Austin is A235. There are some variations to the general rules, as follows: ƒ If the word contains double letters, they must be treated as one letter. So, the double L in Willson is effectively reduced to a single L. ƒ If the word doesn’t contain enough letters to arrive at a complete four-character code, zeros are appended to the end to make up the shortfall. (See the next example.) ƒ If the word contains side-by-side letters that have the same number value according to table 10.3, they’re treated as one letter. Consider the surname Jackson. Using the rules, we start with J as the first character of the code, ignore a, and treat c, k, and s as one letter, giving us 2 as the second character of the code. We ignore o, and use the character 5 for n. Finally, since no letters are left to process in the original word, we add 0 to complete the code: J250. Note that this rule holds for the first character of the code as well. For example, if a word starts with the letters PV, then both these letters are considered one character since they have the same assigned number. ƒ If the word contains consonants that have the same number value, separated by either H or W, then only the consonant to the right of the separation is taken into consideration. Consider the surname Ashcraft. Using this rule, its code is A261, not A226. 10.2.4 Metaphone Although the Soundex algorithm returns good language approximations over a small set of words, its failure rate as the number of words increase becomes larger. The biggest problem with Soundex is its dependence on using the first letter of the original word as the first character of the code. This is a problem because a lot of words that have a different first letter sound similar. Consider the words Cathy and Kathy. Both sound the same, yet they evaluate to different Soundex codes—thus breaking one of the reasons why we use Soundex in the first place, which is to group together similar sounding words. The Metaphone algorithm improves on Soundex. This algorithm was originally published by Lawrence Philips in the now-defunct magazine Computer Language, in the December 1990 issue. The rules for Metaphone encoding are much more complex than the Soundex rules. The Metaphone algorithm works by converting the original word into a code with one to four characters. Unlike Soundex, the Metaphone code contains no numbers except 0. The resulting code can contain only the following 16 characters: B, X, S, K, J, T, F, H, L, M, N, P, R, W, Y, and 0. Licensed to Tricia Fu MODULE 10: CODEC: ENCODERS AND DECODERS 7 It’s best to explain the Metaphone rules using an example. Let’s convert the words Willson and Ithicca into their respective Metaphone codes. This process is shown in table 10.4, using the conversion chart found in table 10.5. Table 10.4 Metaphone example Rule to be applied Rule applied to Willson Rule applied to Ithicca 1. Remove all vowels that occur after the first letter. Wllsn Ithcc 2. Drop all double letters except double C. Wlsn Ithcc 3. Consult table 10.5 to convert remaining letters. W -> W l -> L s -> S n -> N Result: WLSN I -> I th -> 0 c -> K c -> K Result: I0KK Table 10.5 Metaphone conversion chart Letter of the alphabet Converted to Comment B B Unless at the end of a word after m as in dumb. The rules don’t specify what happens if the letter B does appear at the end after M. What is it replaced with? Since the rule is silent, all Bs, wherever they occur, are converted to B regardless. C X X (sh) if -cia- or -ch-. S S if -ci-, -ce-, or -cy-. Silent if -sci-, -sce-, -scy-. K Otherwise, including -sch-. D J If in -dge-, -dgy-, -dgi. T Otherwise. F F G Silent if in -gh- and not at end or before a vowel. In -gn or -gned. In -dge-, and so on, as in above rule for the letter D. J If before I, E, or Y, if not double GG. K Otherwise. H Silent if after vowel and no vowel follows. H Otherwise. J J K Silent if after C. K Otherwise. L L M M N N P F If before H. P Otherwise. Q K R R S X If before H (think sh) or in -sio- or -sia-. S Otherwise. T X If -tia- or -tio- (think sh). 0 (the number zero) If before H (think th). Silent if in tch-. T Otherwise. V F W Silent if not followed by a vowel. W If followed by a vowel. X KS Y Silent if not followed by a vowel. Y If followed by a vowel. Z S Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF As with other algorithms, there are some exceptions to the rules shown in table 10.5. These are listed in table 10.6. All these exceptions relate to the first letter of the word being converted. Table 10.6 Metaphone coding exceptions Exception Rule Example The word starts with any of the following: KN, GN, PM, AE, WR. Drop the first letter (don’t consider the first letter of the word). Knuthtranslates to N0 by ignoring the initial K. The word starts with X. Change X to S instead of KS. Xylophone translates to SLFN instead of KSLF. The word starts with WH. Change it to W. White translates to WT. Note that the other rule of W being silent if not followed by a vowel still applies. This means that Whyte translates to a single letter T in this case. 10.2.5 Double Metaphone Even though Metaphone is an improvement over Soundex, it still leaves some errors in translating and encoding certain similar-sounding words. Consider the case of Auto and Otto. Even though both sound the same and should equate to the same code, Metaphone incorrectly encodes them to AT and OT, respectively. Soundex was developed in early 1900s. Metaphone was developed in 1990. Double Metaphone came about in 2000. This timeline is a reflection of the fact that as the number of possible surnames grew, the usability of a particular algorithm diminished. Newer surnames expose the inadequacies of previous algorithms as the population grows and becomes more diverse. Interestingly, the same person (Lawrence Philips) developed both Metaphone and Double Metaphone. Double Metaphone accounts for non-English words like European and Asian names and consists of only 12 of the original 16 characters used in Metaphone. To account for the different ways in which originally non-American words (Schmidt, Kuczewski) are pronounced in countries of origin, Double Metaphone produces two codes for each word; hence this algorithm is called Double Metaphone. The primary code produced is a reflection of the way the word is pronounced in American English, whereas the secondary code is the way the word is pronounced in the country of origin. So, to compare two words and decipher if they evaluate to the same code—or rather, if they should be categorized together because they’re phonetically alike—it’s necessary to compare both the primary code and the secondary code. If the primary codes are alike, it shows the maximum correlation between the words. This correlation decreases with other matches. We’ll show an example of using Double Metaphone in the examples later, which will help you understand the primary and secondary codes. A full discussion of the rules behind Double Metaphone is beyond the scope of this book. If you’re interested, you can refer to the original article that presented this algorithm: http://www.cuj.com/documents/s=8038/cuj0006philips/ (note that this web site requires a subscription). 10.2.6 URL encoding URL encoding, or www-form-urlencoded, is a type of encoding mechanism used to encode data when an HTML form is submitted over the Internet. This encoding support is provided in the Java2 platform. Until the 1.4 release of the Java2 SDK, the encoding provided in the java.net.URLEncoder class was platform- dependent. This meant that unsafe characters in HTML forms were encoded based on the charset encoding of the computer on which the encoding was being done. This can result in problems because the charset encoding required at the decoding end may not be the same. Since the 1.4 release of the SDK, the original method has been deprecated and a new method introduced that allows you to specify the encoding to be used Licensed to Tricia Fu MODULE 10: CODEC: ENCODERS AND DECODERS 9 (the recommendation is use UTF-8 encoding). The URLCodec class in the Codec package is useful for programming with older versions of Java that don’t support encoding with different charsets. www-form-urlencoded is the default encoding used to encode HTML forms. When no encoding is supplied, form data is encoded according to this scheme to prepare the data for sending to the server. We used another encoding, multipart/form-data, in module 2. The rules for www-form-urlencoded encoding are simple: ƒ Letters A–Z and a–z, digits 0–9, and punctuation characters - _ . ! ~ * ‘ ( ) in the form data are left as is. ƒ Any space characters are replaced with the + sign. ƒ Any other characters are replaced with their equivalent ASCII hexadecimal value, which is preceded by the % character. For example, if a + character appears within the data, it’s replaced with %2B. ƒ A line break (CRLF) is represented as %0D%0A (its hexadecimal value). 10.2.7 MD5 and SHA Although Message Digest 5 (MD5) and Secure Hash Algorithm (SHA) as such aren’t encoding algorithms, implementation of these algorithms is provided as part of the utilities in Codec. We’ll briefly cover these algorithms here. MD5 and SHA are algorithms that let you develop the hash of a message. A hash, which is also known as a message digest, is a smaller, unique representation of the original message and doesn’t contain any information from the original message. A hash of a message is unique in the sense that no two messages evaluate to the same hash value. Even a small change in the original message produces a different hash value. Further, the original message can’t be reconstructed from the hash. These two properties make a hash of a message an ideal signature of the message; this is a central concept in digital signatures. MD5 was developed in 1991 by Professor Ronald Rivest and was proposed in rfc1321. It was developed as an enhancement to MD4. You can access full details of this algorithm at http://www.ietf.org/rfc/rfc1321.txt. SHA was developed by the National Institute of Standards and Technology (NIST), which is a US government department. SHA produces a 20-byte digest as compared to 16 bytes in MD5. It’s slower than MD5 but slightly more secure (because of the large size of the digest). You can learn more about SHA at http://csrc.nist.gov/CryptoToolkit/tkhash.html. 10.3 Getting to know Codec The Codec component is divided into four packages. The org.apache.commons.codec package contains the interfaces that define the behavior of the rest of the packages. This package also contains two exception classes that are general catchall exceptions for anything that goes wrong while encoding or decoding. Finally, this package contains a comparator class for comparing encoded Strings. The org.apache.commons.codec.binary package contains the implementation classes for the Base64 and hexadecimal algorithms. Similarly, the org.apache.commons.codec.language package contains the implementation classes for Soundex, Refined Soundex, Metaphone, and Double Metaphone encodings. The org.apache.commons.codec.net package contains the URL encoding algorithm. The digest utilities for creating the MD5 and SHA digests are in the org.apache.commons. codec.digest package. Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF 10.3.1 The org.apache.commons.codec package The org.apache.commons.codec package contains the interfaces that define the way the rest of the classes are implemented. The base interfaces are the Encoder and Decoder interfaces. Encoder The Encoder interface defines a single method with the following signature public Object encode(Object pObject) throws EncoderException All classes in the Codec package that provide encoding implement this method. Actually, the implementation classes implement a subclass of this interface, depending on the type of encoding provided by the class. Notice that the signature is generic in its capabilities; It allows for encoding of any object to any other object and thus makes no restrictions. Decoder The Decoder interface defines a single method, similar to the Encoder interface: public Object decode(Object pObject) throws DecoderException Again, the Decoder interface is generic and thus allows for the decoding of any object to any other object. The decoding object must of the same type (class, interface) as expected by the implementation class. If this isn’t the case, a ClassCastException occurs, which is thrown as a DecoderException. EncoderException, DecoderException Any parsing, processing, or conversion errors in the Codec implementation raise these errors. EncoderException is for errors that occur while trying to encode data. These errors may be related to bad input, invalid data streams, characters out of range, and so on. DecoderException occurs while trying to decode previously encoded data. This exception can arise due to bad input data, invalid casts, and so on. These exceptions get more specific about the error condition with the implementation classes. BinaryEncoder, BinaryDecoder These interfaces are specific and are targeted toward encoding and decoding binary data. Specifically, the input and output data is expected to be a byte array. The interfaces implement the Encoder and Decoder interfaces, respectively, and therefore define the following method signatures: public byte[] encode(byte[] pArray) throws EncoderException public byte[] decode(byte[] pArray) throws DecoderException StringEncoder, StringDecoder These interfaces are specific toward encoding and decoding of String literals. As with their binary counterparts, these interfaces implement the Encoder and Decoder interfaces. The input and output are expected to be Strings. The method signatures are listed here: public String encode(String pString) throws EncoderException public String decode(String pString) throws DecoderException Licensed to Tricia Fu MODULE 10: CODEC: ENCODERS AND DECODERS 11 StringEncoderComparator A very useful class provided in the Codec package is StringEncoderComparator. This comparator class lets you compare the encoded values of Strings for various String encodings. This is useful when you want to sort Strings based on their encodings rather than literal lexicographically. This is, of course, applicable only to classes that implement the StringEncoder interface. Thus the StringEncoderComparator must be initialized with the StringEncoder being used with the constructor public StringEncoderComparator(StringEncoder en). 10.3.2 The Binary package The org.apache.commons.codec.binary package contains implementations for two binary encoding algorithms. As you’d expect, these two binary packages implement the BinaryEncoder and BinaryDecoder interfaces. Base64 The Base64 class provides encoding/decoding functions for the Base64 algorithm by implementing the BinaryEncoder and BinaryDecoder interfaces. In addition to the methods dictated by the interfaces, this class provides several utility methods to encode input data. Thus there are the expected methods, like these: public byte[] encode(byte[] pArray) throws EncoderException public Object encode(Object pObject) throws EncoderException However, these methods defer to the method public static byte[] encodeBase64(byte[] binaryData, boolean isChunked) This method is interesting because not only is it static, which allows you to call it without creating an instance of the class, it also allows lets you specify that you want output in groups of 76 characters by setting the flag isChunked. By setting this flag to true, you make the output be in blocks of 76 characters followed by a new line and another block. This is in strict adherence to the Base64 RFC, which mandates that the output must be in 76-character blocks. You can call this method directly and set the flag to true if you want the output in 76-character blocks, or you can call the method public static byte[] encodeBase64Chunked(byte[] binaryData) By default, this flag is set to false. Using the public static boolean isArrayByteBase64(byte[] arrayOctect) method, you can test an encoded byte array for corruption of data and validity of the characters before attempting to decode it. Hex The Hex class in the Binary package provides encoding/decoding for converting byte arrays into hexadecimal values. By encoding, the resulting byte array is reduced to half, because two bytes are represented by one Hex symbol. You can use the following static methods for encoding and decoding, respectively: public static char[] encodeHex(byte[] data) throws EncoderException public static byte[] decodeHex(char[] data) throws DecoderException Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF Or, you can use the interface-dependent methods, as with the Base64 class. 10.3.3 The Language package The org.apache.commons.codec.language package provides phonetic encoding and decoding of the algorithms we discussed earlier. Soundex, RefinedSoundex Both Soundex and RefinedSoundex follow the familiar pattern of providing similar methods for encoding Strings. Both implement the StringEncoder interface and thus both have the method public String encode(String pString) in addition to the method public Object encode(Object pObject), which is mandated because StringEncoder extends the Encoder interface. Note that no decoding methods are available, which makes sense because once a word is encoded, the resulting code can relate to several possible words and therefore can’t be decoded to a set vocabulary. Soundex and RefinedSoundex provide constructors that let you specify the mapping to be used for mapping characters to code: public Soundex(char[] mapping) public RefinedSoundex(char[] mapping) This can be useful in cases where you want to use a mapping different than the US English mapping of characters, which is the default. You can call the method public String soundex(String str) directly to encode your word if you don’t want to call the interface-mandated methods. This is the method that is called internally by the interface-mandated methods. Metaphone, DoubleMetaphone Metaphone and DoubleMetaphone are again similar in terms of methods provided. In addition to the expected encoding methods mandated by the implemented interface of StringEncoder and Encoder, there are methods to get/set the maximum lengths of the resulting code and to compare whether two words are similar based on their Metaphone values. You can use public void setMaxCodeLen(int maxCodeLen) to set and public int getMaxCodeLen() to retrieve the length of the resulting code after encoding, which by default is 4 for both algorithms. To compare two strings for equal codes, use the method public boolean isMetaphoneEqual(String str1, String str2) for Metaphone and public boolean isDoubleMetaphoneEqual(String str1, String str2) for DoubleMetaphone. For DoubleMetaphone, another method public boolean isDoubleMetaphoneEqual(String value1,String value2, boolean alternate) can check whether the alternate codes are equal as well (true means to check using secondary or alternate codes; false means to check using the primary code). Licensed to Tricia Fu MODULE 10: CODEC: ENCODERS AND DECODERS 13 With DoubleMetaphone, you can use the method public String doubleMetaphone(String value, boolean alternate) to arrive at the secondary encoding for a word, by providing true for the boolean alternate. Finally, you can call the method public String metaphone(String txt) for Metaphone and public String doubleMetaphone(String value) for DoubleMetaphone if you want to get the encodings directly. 10.3.4 The Net package The org.apache.commons.codec.net package contains a single class at the time of writing. This class is an implementation for the www-form-urlencoded encoding. URLCodec The URLCodec class implements both binary and String-based Codec interfaces. This means that it provides implementations for encoding and decoding both String and binary-based inputs. With this in mind, you can find methods to encode using String, byte[], and Object. You’ll find similar methods to decode, using String, byte[], and Object. The URLCodec class provides methods to encode and decode using a different charset than the default charset of US-ASCII. Use these methods, specifying the encoding that you want to use: public String encode(String pString, String charset) throws UnsupportedEncodingException public String decode(String pString, String charset) throws DecoderException, UnsupportedEncodingException All the encode and decode methods internally call these methods, respectively: public static final byte[] encodeUrl(BitSet urlsafe, byte[] pArray) public static final byte[] decodeUrl(byte[] pArray) You can call these methods directly yourself. However, in case of the encode method, you need to provide a list of URL-safe characters. Internally, a BitSet is maintained by default that contains all the characters specified in the first row of table 10.7. 10.3.5 The Digest package The final package, org.apache.commons.codec.digest, contains a single class called DigestUtils. This class provides utility methods to work on creating MessageDigests, or hashes of regular Strings. DigestUtils This class doesn’t implement any of the interfaces specified in the Codec package. In that way, it’s a little different from the rest of the classes. This makes sense, though; creating digests of messages isn’t strictly encoding of data. This class provides static methods to generate the hash for given data. Two types of hash algorithms are supported: MD5 and SHA. For each type, you can generate the hash when the input is given as either a byte array or a String. Again, for each type, you can generate the result as a byte array or a Hex representation in a String. Keeping these needs in mind, the following methods are part of this class: Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF ƒ public static byte[] md5(byte[] data)—Input is a byte array, output is a 16-element MD5 byte array ƒ public static byte[] md5(String data)—Input is a String, output is a 16-element MD5 byte array ƒ public static String md5Hex(byte[] data)—Input is a byte array, output is a 32-character MD5 hex String ƒ public static String md5Hex(String data)—Input is a String, output is a 32-character MD5 hex String ƒ public static byte[] sha(byte[] data)—Input is a byte array, output is a SHA byte array ƒ public static byte[] sha(String data)—Input is a String, output is a SHA byte array ƒ public static String shaHex(byte[] data)—Input is a byte array, output is a SHA Hex String ƒ public static String shaHex(String data)—Input is a String, output is a SHA Hex String Note that internally, all the methods that return a Hex value use the Hex encoder from the binary package. Further, the digests are calculated using the Java-supplied security package. 10.4 Codec at work Phew! We’ve covered a lot of theory! It’s definitely time for some examples. Using Codec is simple; after you learn the way one class works, you’ll be able to apply the same knowledge to other Codec classes as well (except, of course, DigestUtils). To run the following examples of Codec in action, make sure you have a copy of the commons-codec.jar in your CLASSPATH. To begin, let’s look at a few simple binary examples. The following code snippet is used to run these examples: package com.manning.commons.chapter10; import java.io.File; import java.io.IOException; import java.io.FileInputStream; import java.io.FileNotFoundException; import org.apache.commons.codec.binary.Hex; import org.apache.commons.codec.net.URLCodec; import org.apache.commons.codec.binary.Base64; import org.apache.commons.codec.DecoderException; import org.apache.commons.codec.EncoderException; import org.apache.commons.codec.language.Soundex; import org.apache.commons.codec.digest.DigestUtils; import org.apache.commons.codec.StringEncoderComparator; import org.apache.commons.codec.language.DoubleMetaphone; public class CodecExample { private Base64 b64Encoder = new Base64(); private Hex hexEncoder = new Hex(); Licensed to Tricia Fu MODULE 10: CODEC: ENCODERS AND DECODERS 15 private FileInputStream fis; private String result; public static void main(String[] args) { CodecExample app = new CodecExample(); app.doMain(args); } private void doMain(String[] args) { test1(); test2(); test3(); test4(); test5(); test6(); } } Listing 10.1 shows an example of using Codec to encode and decode Base64 and Hex data using the String “Hello World”. The String is encoded/decoded through both Base64 and Hex encoders. The final result gives back the original String. Listing 10.1 Encoding/Decoding using the Binary package classes private void test1() { Base64 b64Encoder = new Base64(); Hex hexEncoder = new Hex(); byte[] b64Result = b64Encoder.encode("HelloWorld!!".getBytes()); System.err.println("Hello World!! got encoded as: " + new String(b64Result)); byte[] hexResult = hexEncoder.encode(b64Result); System.err.println("Hex representation of Base64 result: " + new String(hexResult)); try { hexResult = hexEncoder.decode(hexResult); } catch (DecoderException e) { e.printStackTrace(); } System.err.println("Hex decoding on the encoded data: " + new String(hexResult)); System.err.println("The same encoding should give the original string: " + new String(b64Encoder.decode(hexResult))); } In Listing 10.1, we’ve created the constructors and then applied the encoding and decoding on the resulting instance. You could use the static methods provided in the classes directly instead. Listing 10.2 shows how to get the Base64 representation of a truly binary object, like an image file. Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF Listing 10.2 Encoding chunked and non-chunked Base64 representations private void test2() { File imageFile = new File("hills.jpg"); byte[] imageBytes = new byte[(int)imageFile.length()]; try { FileInputStream fis = new FileInputStream(imageFile); fis.read(imageBytes); fis.close(); } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e1) { e1.printStackTrace(); } System.err.println("Printing Base64 representation of: " + imageFile.getName()); System.err.println(new String(Base64.encodeBase64(imageBytes))); System.err.println("Printing Base64 CHUNKED representation of: " + imageFile.getName()); System.err.println(new String(Base64.encodeBase64Chunked(imageBytes))); } Recall that chunked output means Base64 data that is constrained at a maximum of 76 characters per line. The output of running this listing reflects that, as shown in figure 10.5. Figure 10.5 Partial output of chunked Base64 encoding of an image Listing 10.3 shows examples of using the Soundex encoding and the StringEncoderComparator class. Listing 10.3 Soundex and StringEncoderComparator encoding examples private void test3() { Soundex sndx = new Soundex(); Licensed to Tricia Fu MODULE 10: CODEC: ENCODERS AND DECODERS 17 System.err.println("Soundex Code for Wilson is: " + sndx.encode("Wilson")); System.err.println("Soundex Code for Wylson is: " + sndx.encode("Wylson")); StringEncoderComparator comparator1 = new StringEncoderComparator(sndx); System.err.println("Are Wilson and Wylson same based on Soundex? " + comparator1.compare("Wilson", "Wylson")); System.err.println("Are Auto and Otto same based on Soundex? " + comparator1.compare("Auto", "Otto")); } If you run this listing, the output shows that using the comparator, the Soundex codes for Wilson and Wylson are equal. Not so for Auto and Otto, for which we have to use the Double Metaphone encoding as shown in Listing 10.4. Listing 10.4 Double Metaphone example private void test4() { DoubleMetaphone doubleMetaphone = new DoubleMetaphone(); StringEncoderComparator comparator2 = new StringEncoderComparator(doubleMetaphone); System.err.println("Are Auto and Otto same based on DoubleMetaphone? " + comparator2.compare("Auto", "Otto")); System.err.println("Double Metaphone primary code for Schmidt: " + doubleMetaphone.doubleMetaphone("Schmidt")); System.err.println("Double Metaphone secondary code for Schmidt: " + doubleMetaphone.doubleMetaphone("Schmidt", true)); } Using the Metaphone class is similar to using Soundex and DoubleMetaphone. Listing 10.5 shows an example of how to run the URLCodec class. You’ll notice the similarities in the working of this class and the rest of the classes. Listing 10.5 URLCodec example private void test5() { String urlData1 = "This#is^a&String with reserved @/characters"; URLCodec encoder = new URLCodec(); try { String result = encoder.encode(urlData1); } catch (EncoderException e) { e.printStackTrace(); } System.err.println("URL Encoding result: " + result); try { System.err.println("URL Decoding result: " + encoder.decode(result)); } catch (DecoderException e1) { e1.printStackTrace(); Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF } } Running this listing gives the original String as the result. The charset used in this example is the default charset of US-ASCII. Finally, listing 10.6 gives a few examples of using the DigestUtils class for calculating the hash of input data. Listing 10.6 Using DigestUtils private void test6() { String hashData = "Hello World!!"; System.err.println("Hello World!! as MD5 16 element hash: " + new String(DigestUtils.md5(hashData))); System.err.println("Hello World!! as MD5 Hex hash: " + DigestUtils.md5Hex(hashData)); System.err.println("Hello World!! as SHA byte array hash: " + new String(DigestUtils.sha(hashData))); System.err.println("Hello World!! as SHA Hex hash: " + DigestUtils.shaHex(hashData)); } When you run this listing, the first and the third methods output unintelligible data. This is because these methods output byte arrays, which contain the hash of the original data and are binary in nature; see figure 10.6. Figure 10.6 The result of running the DigestUtils class 10.5 Summary We have covered a lot of algorithms in this module. Some of the algorithms, like Base64, www-form- urlencoded, and Hex, are everyday algorithms that you as a computer programmer have encountered before. Others, like Soundex and Metaphone, are more unique; only programmers with exposure to linguistics or phonetics may have come across them. The Codec component provides you with easy implementations of these algorithms. As you saw in this module, Codec is extremely easy to use; once you know how to use one algorithm, you can extend it to others as well. You also saw that Codec provides a utility class for creating hashes of arbitrary data using well known MD5 and SHA algorithms. In the next module, we’ll look at the Modeler component, which is useful for configuring and instantiating Model MBeans (used in the Java Management Extensions API). Licensed to Tricia Fu MODULE 10: CODEC: ENCODERS AND DECODERS 19 Index 7-bit data, 4 8-bit data, 4 American Standard Code for Information Interchange. See ASCII ASCII, 2 Base64 encoding, 2 binary data, 4 Codec API binary encoding classes, 11 digest utilities, 13 interface definitions, 10 phonetic encoding classes, 12 URL Encoding class, 13 API, 9 Base64 encoding an image file, 16 using Binary classes, 15 using DigestUtils, 18 using phonetic encoders, 17 using URLCodec, 17 Content-Transfer-Encoding header, 1 decoding, 1 difference between 7-bit, 8-bit, and binary data, 4 Double Metaphone, 8 encoded messages, 1 encoding, 1 grouping similar sounding names, 5 hexadecimal notation, 4 MD5 hashing, 9 Metaphone algorithm, 6 Metaphone coding exceptions, 8 Metaphone conversion chart, 7 National Institute of Standards and Technology. See NIST NIST, 9 phonetic encoding, 5 RefinedSoundex, 12 rfc2045. See Base64 encoding SHA, 9 Soundex algorithm, 5 Soundex coding guide, 6 StringEncoderComparator, 11 Unicode, 2 URL encoding, 8 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Managing components with Modeler Vikram Goyal MODULE MANNING 11 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 11 Managing components with Modeler 11.1 Understanding JMX................................................................................................................................................ 1 11.2 Say hello to Modeler............................................................................................................................................. 20 11.3 Summary............................................................................................................................................................... 31 Index ............................................................................................................................................................................. 32 Imagine that you’ve just built the next killer Java application, and it’s selling like hotcakes. However, even though (being a good developer) you built in enough debug and tracing information to monitor the state of your live application units, you’re missing the ability to monitor each and every class and their attributes. Although you could build monitoring services as part of your application, it takes the focus away from your core business, the application itself. It would be nice if you could monitor your application for the state of its classes, attributes, and operations. Java Management Extensions (JMX) let you do just that. However, this chapter isn’t about JMX. It’s about how to easily configure JMX using the Modeler component. Specifically, it’s about how to use the Modeler component to create Model MBeans (a special kind of management bean), which are used to monitor resources in your application. It would be difficult to understand Modeler without first JMX. Therefore, this chapter starts with a simple tutorial of the JMX API and its components. We’ll pay special attention to Model MBeans and how they interact with their environment. We’ll then introduce Modeler using several examples. We’ll round out the chapter with examples of how to integrate Modeler with the Ant build tool. 11.1 Understanding JMX JMX is a toolset that provides guidelines to manage and monitor applications in a distributed environment. But what does that mean? It means that using JMX, you can view the state of your application, make changes to variables, invoke operations on your objects, and gather any other information that is exposed by your application, without making wholesale changes to your application’s main functions or architecture. Let’s try to understand this in the context of an application with diagrams and code. In this process, we’ll introduce the various layers of a JMX-managed application. The first layer is called the Instrumentation layer. 11.1.1 Introducing the Instrumentation layer To begin, consider figure 11.1. It shows an application as a standalone component that’s deployed and functional in a production environment. How do you introduce JMX in this picture? You start by introducing another component called a managed bean (MBean). An MBean is a Java object that acts as a wrapper for your application components and exposes them for management. Figure 11.2 shows an MBean introduced into figure 11.1. Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF Figure 11.1 An application in a production environment. How do you manage this application? Figure 11.2 Introducing an MBean to expose your application for management The MBeans and your application’s components together form what is called the Instrumentation layer for JMX-managed applications. The Instrumentation layer interfaces with the outside world to expose these components for management. But how do the MBeans in this layer expose these components? Well, MBeans are interface descriptions of your components. This means that the MBeans closely resemble the components they expose and that the components must implement these interfaces themselves. These interface definitions are read by the outside components using Java Reflection. Let’s look at some code to clarify things before proceeding further. Listing 11.1 shows a simple application that prints the current date and time after a specified delay. Consider this listing an application component that will be managed using JMX. Listing 11.1 Simple application that prints the current date and time after a delay package com.manning.commons.chapter11; import java.util.Date; import java.util.Timer; import java.util.TimerTask; public class TimerApplication extends TimerTask { private long delay; Licensed to Tricia Fu 3MODULE 11: MANAGING COMPONENTS WITH MODELER 3 private String message; public TimerApplication(long delay, String message) { this.delay = delay; this.message = message; Timer timer = new Timer(); timer.schedule(this, new Date(), delay); } public long getDelay() { return this.delay; } public void setDelay(long delay) { this.delay = delay; } public String getMessage() { return this.message; } public void setMessage(String message) { this.message = message; } public void run() { System.err.println(getMessage() + new Date()); } public static void main(String[] args) { TimerApplication timerApp = new TimerApplication(15000, "The time is now: "); } } When this application runs, it prints the following after every 15 seconds (the default value) until it’s terminated: The time is now: Sat Jul 31 16:08:44 EST 2004 The time is now: Sat Jul 31 16:08:59 EST 2004 The time is now: Sat Jul 31 16:09:14 EST 2004 The first step in managing this application component is to write an MBean interface, shown in listing 11.2. Listing 11.2 MBean interface to manage the application in listing 11.1 package com.manning.commons.chapter11; public interface TimerApplicationMBean { public long getDelay(); public void setDelay(long delay); public String getMessage(); public void setMessage(String message); } Notice how all the properties of the application in listing 11.1 have been extracted in this interface for exposure. If we wanted to, we could extract other information in terms of operations; but no other operations Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF in this application need to be exposed, so we’ll leave this interface as is. Listing 11.1 now needs to be changed to make it a manageable component; we do so by making it implement this MBean interface. The relevant code snippet is shown here, with the change in bold: public class TimerApplication extends TimerTask implements TimerApplicationMBean { There are no other changes in this class, because it was used as the basis for making the MBean interface; so, it already implements all the methods. Now our Instrumentation layer consists of the class TimerApplication; our application component; and the interface TimerApplicationMBean, the MBean interface that manages this component. The MBean now needs to talk to the Agent layer, which contains these two classes and exposes them for use to the outside world. 11.1.2 The Agent layer In this section, we’ll discuss the Agent layer of JMX-managed applications. This layer is responsible for managing the MBeans by putting them in a server called MBeanServer and registering them with this server. The registered MBeans (and the associated application components) are exposed to the outside world through this agent. Figure 11.3 shows the Agent layer introduced in figure 11.2. Figure 11.3 The Agent layer provides agent services and creates the MBeanServer. In addition to acting as a container for the MBeans, the Agent layer provides agent services. These services include the following, at minimum: ƒ A timer service for sending notifications at predefined intervals ƒ A dynamic loading service for loading MBeans over a network Licensed to Tricia Fu 5MODULE 11: MANAGING COMPONENTS WITH MODELER 5 ƒ A monitoring service for monitoring the state of MBeans registered in the MBeanServer ƒ A relation service that manages relationships between MBeans Before we go any further, you need to download the reference implementation (RI) for JMX from Sun’s web site at http://java.sun.com/products/JavaManagement/index.jsp. The RI contains two libraries: jmxri.jar and jmxtools.jar. Both these libraries must be in your CLASSPATH. Now, let’s write code for the agent that will create the MBeanServer and register our TimerApplicationMBean with it. Listing 11.3 shows the code for the TimerApplicationAgent class. Listing 11.3 TimerApplicationAgent class creates MBeanServer and registers MBeans package com.manning.commons.chapter11; import javax.management.ObjectName; import javax.management.MBeanServer; import javax.management.MBeanServerFactory; import com.sun.jdmk.comm.HtmlAdaptorServer; public class TimerApplicationAgent { public TimerApplicationAgent() { MBeanServer server = MBeanServerFactory.createMBeanServer("TimerDomain"); HtmlAdaptorServer adaptor = new HtmlAdaptorServer(); TimerApplication timerApp = new TimerApplication(15000, "The time is now: "); ObjectName adaptorName = null; ObjectName timerAppName = null; try { timerAppName = new ObjectName("TimerDomain:name=timerApp"); server.registerMBean(timerApp, timerAppName); adaptorName = new ObjectName("TimerDomain:name=adaptor, port=8082"); adaptor.setPort(8082); server.registerMBean(adaptor, adaptorName); adaptor.start(); } catch(Exception e) { System.err.println(e); } } Create MBeanServer using MBeanServerFactory Create HtmlAdaptor to interface agent over HTTP Register MBeans in MBeanServer using ObjectNames Create manageable MBean Start HtmlAdaptor to accept connections over port 8082 Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF public static void main(String[] args) { TimerApplicationAgent agent = new TimerApplicationAgent(); } } The steps to create an agent are simple. First an MBeanServer is created with the domain name TimerDomain. Next, MBeans are created and registered with this server. This code registers two MBeans: the TimerApplication MBean and a protocol adaptor MBean that lets us connect to this agent and manage the registered MBeans over HTTP. This code listing also introduces two new concepts, ObjectNames and Adaptors. Let’s talk about them next. ObjectNames An ObjectName is a unique identifier for each MBean. External monitoring applications use this ObjectName to identify the application components on which they need to operate. In listing 11.3, the TimerApplication is registered in the MBeanServer with the ObjectName TimerDomain:name= timerApp. As you can see, the ObjectName consists of a simple String object, which is divided into two parts as shown in figure 11.4. Figure 11.4 Separating the parts of an ObjectName The first part of the ObjectName is the domain of the MBean (everything until the colon). This implies that a single MBeanServer can contain MBeans for several different domains, which is correct. Thus, you can register several MBeans under one domain to help segregate your MBeans. If you don’t specify a domain for an MBean, it’s registered in the default domain, which is created when the server is created (in listing 11.3, the default domain is TimerDomain). Note that if you don’t specify a default domain, one is created for you. The second part of the ObjectName is a comma-separated key/value pair of properties that further qualify each MBean. This part gives more information about the MBean and helps differentiate between similar MBeans registered in the same domain. For example, a key/value pair of name=timerApp qualifies the TimerApplication in the example. If we were to add another TimerApplication for management, we could perhaps qualify it with name=timerApp1. Adaptors Although an agent application can start an MBeanServer and register MBeans with it, it can’t expose them to the outside world without help. This is where the Adaptors come in. An Adaptor is a component that attaches itself to the Agent layer of JMX application and formats the view of the registered MBeans for the target protocol over which they’re to be viewed and maintained. For example, Sun’s reference implementation Licensed to Tricia Fu 7MODULE 11: MANAGING COMPONENTS WITH MODELER 7 contains a protocol adaptor for the HTTP protocol. It’s called HtmlAdaptorServer, and it’s present in the com.sun.jdmk.comm package. The classes in this package aren’t considered part of the official reference implementation and are provided as is in the jmxtools.jar library. Therefore, if you find that this protocol adaptor didn’t serve your needs, you can roll your own. The Adaptors aren’t considered part of the Agent layer of the JMX architecture. They’re in a separate layer, called the Distributed layer. The JMX specification doesn’t provide any information about this layer, except a brief overview. The idea is to let management applications define their own protocol adaptors and connectors for accessing the agents of a JMX-managed application. However, Adaptors are themselves MBeans; that’s why, in listing 11.3, the HtmlProtocolAdaptor was registered with the MBeanServer as an MBean. For completeness, figure 11.5 shows this layer introduced to figure 11.3. Figure 11.5 The complete JMX architecture, showing all the layers Let’s now run the TimerApplicationAgent and see how we can manage its MBeans through the HtmlProtocolAdaptor. 11.1.3 Running TimerApplication Our goal in this section is to manage the application component shown in listing 11.1 by tweaking some of its properties through JMX. To do so, we’ve so far created an MBean interface in listing 11.2, modified listing 11.1 to implement this interface, and created an agent to create and register this MBean in listing 11.3. Listing 11.3 also created an HtmlAdaptor, which we’ll use as the interface into our application component. Run listing 11.3 via a command line, making sure you have the two libraries jmxri.jar and jmxtools.jar in your CLASSPATH. TimerApplication will start and will print the following on the command line, as expected: The time is now: Mon Aug 09 11:44:06 EST 2004 The time is now: Mon Aug 09 11:44:21 EST 2004 Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF Now, in a browser, navigate to the site localhost:8082. It will show the HtmlProtocolAdaptor view of the MBeans registered in the MBeanServer of the TimerApplicationAgent, as shown in figure 11.6. Figure 11.6 An agent view of the MBeans registered in the MBeanServer As you can see, this page lists the MBeans registered in the MBeanServer by domain names. Recall that we registered the TimerApplication and the HtmlProtocolAdaptor MBeans in the TimerApplicationAgent under the default domain name of TimerDomain. Thus, these two MBeans are listed under this domain. However, there is another domain, and its single MBean listed on this page. This domain is JMImplementation; it contains the MBean that represents the MBean server in which it’s running. This is interesting, because it lets you view the MBeanServer as an MBean itself and manage it using this Adaptor. Click the link for the TimerApplication MBean identified by its ObjectName of name=timerApp. You’ll see the details of the running TimerApplication MBean, as shown in figure 11.7. This page lists all the known information about the registered TimerApplication MBean. The top part lists the name (ObjectName) of this MBean and its class. You can unregister it by clicking the Unregister button. If you do so, the MBean will no longer be manageable in this MBeanServer. The lower part of this page lists the known attributes of this MBean. If any of these attributes are editable, as is the case with both the Delay and Message attributes in this case, they’re listed with a text field for entering the new attribute value; the current value is listed in this text field. Make a change to one of the attributes and click Apply, and you’ll see the result on the command line. Licensed to Tricia Fu 9MODULE 11: MANAGING COMPONENTS WITH MODELER 9 Figure 11.7 Viewing the details of the TimerApplication MBean Our TimerApplication is now a manageable application component, and we’re doing this via JMX, without making any big changes in our application. Let’s now look at the different kinds of MBeans that JMX allows for different application components. 11.1.4 Beans to an end The JMX architecture lets you create MBeans that cater to different scenarios. So far, we’ve encountered the easiest MBean of them all, the Standard MBean, which is useful for describing application components whose structure doesn’t change much. Standard MBeans A Standard MBean, by definition, is an MBean instrumentation that implements its own interface. Since the TimerApplication class in listing 11.1 implements its own MBean interface specified in listing 11.2, these two code listings represent an example of a Standard MBean. These types of MBeans are very quick and easy to create. All you need to do to make your application component an MBean is to define it as an interface. All attributes and operations listed in the interface can be managed by an agent using Java Reflection and, consequentially, by a management application via a protocol adaptor. This implies that if you didn’t want an attribute or operation to be manageable, you wouldn’t list it in the interface. Also, you can make sure attributes are read-only by not providing a setXXX method for the attribute. Although Standard MBeans are a quick solution to making your applications manageable via JMX, they impose a restriction by mandating an interface for each and every application component. This requires Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF knowing in advance the structure of the application component that is being instrumented. Dynamic MBeans, on the other hand, enforce no such restriction and can be instrumented at runtime. Dynamic MBeans Dynamic MBeans enable runtime instrumentation by a predefined interface. The interface is called DynamicMBean and is present in the javax.management package. It defines methods that allow the discovery of manageable attributes and operations at runtime. The application component that will be instrumented should implement this interface and define its methods, which are listed here: ƒ getMBeanInfo—Returns a description of the application component as an MBeanInfo object. An MBeanInfo object contains a list of attributes, constructors, operations, and notifications about an application component. ƒ getAttribute and getAttributes—Return a value for the attribute or attributes specified by name. The method should internally take care of mapping between the name specified and the actual attribute name whose value is returned. ƒ setAttribute and setAttributes—Set the value of an attribute or a set of attributes by name, again taking care to map between attribute names. ƒ invoke—Allows the execution of any operation exposed by the application component being instrumented. It expects the name of the operation, the attributes expected of the operation, and the type of each of those attributes. Listing 11.4 shows the TimerApplication code from listing 11.1 modified to act as a Dynamic MBean. Listing 11.4 TimerApplication modified to act as a Dynamic MBean package com.manning.commons.chapter11; import java.util.Date; import java.util.Timer; import java.util.Iterator; import java.util.TimerTask; import javax.management.MBeanInfo; import javax.management.Attribute; import javax.management.DynamicMBean; import javax.management.AttributeList; import javax.management.MBeanAttributeInfo; import javax.management.MBeanParameterInfo; import javax.management.MBeanConstructorInfo; import javax.management.AttributeNotFoundException; public class TimerApplication extends TimerTask implements DynamicMBean { private long delay; private String message; private MBeanInfo infoBean; Application extends DynamicMBean instead of its own interface Licensed to Tricia Fu 11MODULE 11: MANAGING COMPONENTS WITH MODELER 11 private MBeanAttributeInfo delayInfo; private MBeanAttributeInfo messageInfo; public TimerApplication(long delay, String message) { this.delay = delay; this.message = message; Timer timer = new Timer(); timer.schedule(this, new Date(), delay); } public Long getDelay() { return new Long(this.delay); } public void setDelay(Long delay) { this.delay = delay.longValue(); } public String getMessage() { return this.message; } public void setMessage(String message) { this.message = message; } public void run() { System.err.println(getMessage() + new Date()); } public static void main(String[] args) { TimerApplication timerApp = new TimerApplication(15000, "The time is now: "); } public MBeanInfo getMBeanInfo() { if(infoBean == null) { delayInfo = new MBeanAttributeInfo("delay", "java.lang.Long", "Delay Attribute", true, true, false); messageInfo = new MBeanAttributeInfo("message", "java.lang.String", "Message Attribute", true, true, false); MBeanAttributeInfo[] attributes = new MBeanAttributeInfo[2]; attributes[0] = delayInfo; attributes[1] = messageInfo; MBeanConstructorInfo[] constructors = new MBeanConstructorInfo[1]; MBeanParameterInfo[] constructorParams = new MBeanParameterInfo[2]; constructorParams[0] = new MBeanParameterInfo("delay", "lang", "Delay Param"); Primitive attributes can’t be used; use corresponding objects MBeanInfo contains complete information about Application component Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF constructorParams[1] = new MBeanParameterInfo("message", "java.lang.String", "Message Param"); constructors[0] = new MBeanConstructorInfo("TimerApplication", "TimerApp Constructor", constructorParams); infoBean = new MBeanInfo("com.manning.commons.chapter11.TimerApplication", "Timer Application prints time after a delay", attributes, constructors, null, null); } return infoBean; } public Object getAttribute(String attribute) throws AttributeNotFoundException { if(attribute == null) throw new RuntimeException("Attribute name cannot be null"); if(attribute.equals("delay")) return getDelay(); else if(attribute.equals("message")) return getMessage(); else throw new AttributeNotFoundException(attribute + " not found!"); } public AttributeList getAttributes(String[] attributeNames) { if(attributeNames == null) throw new RuntimeException("Attribute name cannot be null"); AttributeList resultList = new AttributeList(); if (attributeNames.length == 0) return resultList; for(int i=0 ; i < attributeNames.length ; i++){ try { Object value = getAttribute((String) attributeNames[i]); resultList.add(new Attribute(attributeNames[i],value)); } catch(Exception e) { e.printStackTrace(); } } return resultList; } public Object invoke(String actionName, Object[] params, String[] signature) Return attribute values individually or several at once Method doesn’t do anything: application has no invokable operations Licensed to Tricia Fu 13MODULE 11: MANAGING COMPONENTS WITH MODELER 13 { return null; } public void setAttribute(Attribute attribute) throws AttributeNotFoundException { if(attribute == null) throw new RuntimeException("Attribute name cannot be null"); String attributeName = attribute.getName(); Object attributeValue = attribute.getValue(); if(attributeName.equals("delay")) setDelay((Long)attributeValue); else if(attributeName.equals("message")) setMessage((String)attributeValue); else throw new AttributeNotFoundException(attribute + " not found!"); } public AttributeList setAttributes(AttributeList attributes) { if(attributes == null) throw new RuntimeException("Attributes cannot be null"); AttributeList resultList = new AttributeList(); if(attributes.isEmpty()) return resultList; for(Iterator i = attributes.iterator(); i.hasNext();) { Attribute attr = (Attribute)i.next(); try { setAttribute(attr); String name = attr.getName(); Object value = getAttribute(name); resultList.add(new Attribute(name, value)); } catch(Exception e) { e.printStackTrace(); } } return resultList; } } This code listing makes managing your application component decidedly more complex. There is a heavy penalty to be paid, in terms of code complexity, for the flexibility that a DynamicMBean provides. For example, in c, we have complete control over the way attribute values are set using JMX. We could do bounds checks, implement our own conversion patterns, or restrict the type of values that are set in our code. We couldn’t do the same with a Standard MBean implementation. The JMX agent discovers the operations and attributes that are allowed on our MBean at runtime and displays this data using an agent/protocol. To see this in action, run the TimerApplicationAgent from listing 11.3 again, using the code from listing 11.4 as the application component. c Set attribute values individually or several at once Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF Dynamic MBeans can be made portable by sticking to the primitive data types (int, long, boolean, short, and so on) and using Java and JMX-defined data types. A Dynamic MBean that follows these rules is called an Open MBean. Open MBeans An Open MBean is “open” to as many management applications as possible. By sticking to the primitive data types and Java and JMX-defined types, this is indeed true, because management interfaces know how to deal with these types, as opposed to data types whose class definitions may not available at runtime. An Open MBean is an extension of the Dynamic MBean discussed in the previous section. The only difference lies in the MBeanInfo object that is returned by the getMBeanInfo method. Open MBeans return OpenMBeanInfo instances instead of MBeanInfo. JMX provides an implementation of this interface, OpenMBeanInfoSupport, which extends the MBeanInfo class. Therefore, application components that want to identify themselves as Open MBeans can create instances of this class to indicate that they’re indeed open. By creating and setting the management information within our application component, we’ve strayed away from the path of keeping our manageable components simple and without any major modifications. Model MBeans, which we discuss in the next section, let you outsource the creation of this information so that your application components can continue to do only application-related activities. Model MBeans Model MBeans are also Dynamic MBeans. They differ from Dynamic and Open MBeans because they let you leave the application component they’re managing alone while creating information about the component in a separate interface. They differ from Standard MBeans as well in this regard, because even with Standard MBeans, you have to make the application component implement its own interface. However, unlike Standard MBeans, the definition of components within Model MBeans is dynamic. Listing 11.5 shows how to use Model MBeans for our timer application component. The code is a modified version of the TimerApplicationAgent class from listing 11.3. This listing also uses the code from listing 11.1, but it has been modified slightly: The primitive type of the delay attribute has been converted from long to the Long wrapper class, and the class has been renamed TimerApplicationOriginal. Other than that, we haven’t touched the application component with regard to JMX. Listing 11.5 Adding Model MBean support to TimerApplicationAgent package com.manning.commons.chapter11; import javax.management.ObjectName; import javax.management.MBeanInfo; import javax.management.MBeanServer; import javax.management.MBeanServerFactory; import javax.management.MBeanOperationInfo; import javax.management.MBeanParameterInfo; import javax.management.modelmbean.*; import com.sun.jdmk.comm.HtmlAdaptorServer; public class TimerApplicationModelMBeanAgent { Licensed to Tricia Fu 15MODULE 11: MANAGING COMPONENTS WITH MODELER 15 public TimerApplicationModelMBeanAgent() { MBeanServer server = MBeanServerFactory.createMBeanServer("TimerDomain"); HtmlAdaptorServer adaptor = new HtmlAdaptorServer(); TimerApplicationOriginal timerApp = new TimerApplicationOriginal(15000, "The time is now: "); ObjectName adaptorName = null; ObjectName timerAppName = null; try { timerAppName = new ObjectName("TimerDomain:name=timerApp"); RequiredModelMBean modelMBean = new RequiredModelMBean(buildModelMBeanInfo()); modelMBean.setManagedResource(timerApp, "objectReference"); server.registerMBean(modelMBean, timerAppName); adaptorName = new ObjectName("TimerDomain:name=adaptor, port=8082"); adaptor.setPort(8082); server.registerMBean(adaptor, adaptorName); adaptor.start(); } catch(Exception e) { System.err.println(e); } } private ModelMBeanInfo buildModelMBeanInfo() { DescriptorSupport delayDesc = new DescriptorSupport(); delayDesc.setField("name","Delay"); delayDesc.setField("descriptorType","attribute"); delayDesc.setField("displayName","Delay"); delayDesc.setField("getMethod","getDelay"); delayDesc.setField("setMethod","setDelay"); DescriptorSupport messageDesc = new DescriptorSupport(); messageDesc.setField("name","Message"); messageDesc.setField("descriptorType","attribute"); messageDesc.setField("displayName","Message"); messageDesc.setField("getMethod","getMessage"); messageDesc.setField("setMethod","setMessage"); k RequiredModelMBean c Create DescriptorSupport instances Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF ModelMBeanAttributeInfo delayInfo = new ModelMBeanAttributeInfo("delay", "java.lang.Long", "Delay Attribute", true, true, false, delayDesc); ModelMBeanAttributeInfo messageInfo = new ModelMBeanAttributeInfo("message", "java.lang.String", "Message Attribute", true, true, false, messageDesc); ModelMBeanAttributeInfo[] attributes = new ModelMBeanAttributeInfo[] {delayInfo, messageInfo }; ModelMBeanOperationInfo[] operations = new ModelMBeanOperationInfo[4]; MBeanParameterInfo[] params = null; MBeanParameterInfo[] setDelayParams = new MBeanParameterInfo[] { (new MBeanParameterInfo("delay", "java.lang.Lang", "new Delay value") )} ; MBeanParameterInfo[] setMessageParams = new MBeanParameterInfo[] { (new MBeanParameterInfo("message", "java.lang.String", "new Message value") )} ; DescriptorSupport getDelayDesc = new DescriptorSupport(new String[] {"name=getDelay", "descriptorType=operation", "class=com.manning.commons.chapter11.TimerApplicationOriginal", "role=getter"} ); DescriptorSupport setDelayDesc = new DescriptorSupport(new String[] {"name=setDelay", "descriptorType=operation", "class=com.manning.commons.chapter11.TimerApplicationOriginal", "role=setter"} ); DescriptorSupport getMessageDesc = new DescriptorSupport(new String[] {"name=getMessage", "descriptorType=operation", "class=com.manning.commons.chapter11.TimerApplicationOriginal", "role=getter"} ); d Create attributes e Collect attributes in array f Define array of ModelMBeanOperationInfo g MBeanParameterInfo objects h DescriptorSupport objects Licensed to Tricia Fu 17MODULE 11: MANAGING COMPONENTS WITH MODELER 17 DescriptorSupport setMessageDesc = new DescriptorSupport(new String[] {"name=setMessage", "descriptorType=operation", "class=com.manning.commons.chapter11.TimerApplicationOriginal", "role=setter"} ); operations[0] = new ModelMBeanOperationInfo("getDelay", "get value of delay attribute", params, "java.lang.Lang", MBeanOperationInfo.INFO, getDelayDesc); operations[1] = new ModelMBeanOperationInfo("setDelay", "set value of delay attribute", setDelayParams, "void", MBeanOperationInfo.ACTION, setDelayDesc); operations[2] = new ModelMBeanOperationInfo("getMessage", "get value of message attribute", params, "java.lang.String", MBeanOperationInfo.INFO, getMessageDesc); operations[3] = new ModelMBeanOperationInfo("setMessage", "set value of message attribute", setMessageParams, "void", MBeanOperationInfo.ACTION, setMessageDesc); DescriptorSupport beanDesc = new DescriptorSupport(new String[] { "name=timerApp", "descriptorType=MBean"}); ModelMBeanInfoSupport infoBean = new ModelMBeanInfoSupport( "com.manning.commons.chapter11.TimerApplicationOriginal", "Timer Application prints time after a delay", attributes, null, operations, null, beanDesc); return infoBean; } public static void main(String[] args) { TimerApplicationModelMBeanAgent agent = i Create operations and set in ModelMBeanOperationInfo array j Create description for ModelMBeanInfo object Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF new TimerApplicationModelMBeanAgent(); } } There are several steps in the creation of a Model MBean, all of which take place outside the application component that is being managed. In listing 11.5, we’ve made these changes in the TimerApplicationModelMBeanAgent agent class. These steps are as follows (figure 11.8 shows the steps sequentially): cde Here we create the attribute information for our application component. This is done by first creating a DescriptorSupport instance for each of the attributes that will be exposed, as shown in c. A DescriptorSupport class is a helper class for describing the details of an attribute, operation, constructor, parameter to an operation or constructor, or the component itself. The DescriptorSupport class is available in the javax.management.modelmbean package and consists of a collection of fields, which are in the format of fieldname=fieldValue. For example, the delay attribute’s descriptor is created by setting the value of five fields: name, descriptionType, displayName, getMethod, and setMethod. These are some of the predefined fields for attributes of Model MBeans; you can see a full list by looking at the Java Doc description for the ModelMBeanAttributeInfo class. Once the DescriptorSupport has been created for an attribute, the attribute itself is created by using the ModelMBeanAttributeInfo’s constructor, which takes the DescriptorSupport as a parameter, as shown in d. The rest of the parameters for this constructor are similar to those created for the MBeanAttributeInfo class in listing 11.4 All the attributes are then collected together in an array of type ModelMBeanAttributeInfo, as shown in e. fghi Once the attributes of our application that we want to expose have been created, we can perform the same process for the operations that we want to expose. In this case, we want to expose the get and set methods for the setting of the attributes defined in the previous steps. We do so by first defining an array of ModelMBeanOperationInfo to hold all the defined operations, in f. Again, a ModelMBeanOperationInfo object is similar to an MBeanOperationInfo object from listing 11.4, with the key difference that ModelMBeanOperationInfo contains a DescriptorSupport instance. These DescriptorSupport objects are created in h for the get/set methods of the delay and message attributes. However, the operations can’t be completely defined without the presence of the MBeanParameterInfo objects. These objects provide detailed information about the parameters accepted by each of the operations. For example, the get methods accept no parameter. Therefore, these are set as null in g. The set method for the delay attribute accepts the java.lang.Long parameter, and therefore an MBeanParameterInfo object is created for it in g. Finally, in i, the four operations are created with their descriptors and set in the ModelMBeanOperationInfo array. j We’ve created the attributes, and we’ve created the operations; all that is now left is to create is the description for the ModelMBeanInfo object. This is done by first creating the DescriptorSupport object and then creating the ModelMBeanInfoSupport class with this descriptor. k If you’ve been wondering about the need to define a DescriptorSupport object for every manageable attribute/operation of an application component, then it should be clear after examining this code snippet. Let’s examine this section in detail. In the first line of this snippet, a RequiredModelMBean object is created using the ModelMBeanInfoSupport class we just created as a parameter. As the name suggests, the Licensed to Tricia Fu 19MODULE 11: MANAGING COMPONENTS WITH MODELER 19 RequiredModelMBean class is a requirement with every implementation of JMX. Technically, a Model MBean is a generic class that can be used to manage any application component. Rather, an application component that wants to be managed should be able to associate itself with a Model MBean. JMX agents know how to manage a Model MBean, but they don’t know how to manage your application component. Therefore, by associating themselves with a Model MBean, application components allow themselves to be managed, but that still leaves the problem of association between application components values and methods and the Model MBean’s attributes and operations. For example, how does a JMX agent know that to set the value of your message attribute, it needs to call the setMessage method of the TimerApplication class and not some method called setMsg? This is where the DescriptorSupport classes come in. In h, we set the DescriptorSupport class to define the setMessage operation. In the second line of this code snippet, the RequiredModelMBean is told to manage the TimerApplication instance that has already been created. The second parameter of objectReference suggests that this instance is a type of object reference and not an RMIReference or an EJBHandle, among other things. Finally, the last line of this code snippet registers the RequiredModelMBean in the MBeanServer using the TimerApplication’s object name created previously in the code. Figure 11.8 Steps required to create a Model MBean Even though we’ve managed to completely separate the management of our application component from the component itself, and made the management generic in the process by using Model MBeans, we’ve introduced far too much complexity in the code. We have too much code for a two-attribute component, and writing this amount of code for larger and more complex components would be a dreary job. Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF Is there a way to take this complexity out of our code, even though it’s in the agent code? Yes, of course. This is where the Modeler component comes in. Let’s look at the Modeler component next to see how we can achieve this. 11.2 Say hello to Modeler The Modeler component was created from source code taken from Tomcat 4 development. Recognizing that creating metadata information about managed resources in code is a tedious process, the Tomcat developers extracted this information from code to an external XML-based file. They further realized that this extraction could easily be made useful for other managed components—not only server-based projects like Tomcat but any place that needed to use Model MBeans. Therefore, they created the Modeler package to make this service available across the board. As an application developer, it’s nice to know at a glance the services that Modeler provides. The following list shows these services: ƒ It provides a service for reading and parsing XML-based configuration information (the information contained in the buildModelMBeanInfo method in listing 11.5) for Model MBeans. ƒ It provides a central registry for storing this configuration information, which also forms the basis for easily creating ModelMBeanInfo objects from this information. ƒ It provides an alternate to the RequiredModelMBean object, which is a default implementation of the ModelMBean interface in JMX. This alternate is called BaseModelMBean. However, this alternate only supports management of objects that are of type objectReference (as opposed to RequiredModelMBean, which supports other types such as EJBHandle and RMIReference). ƒ It provides a Java Naming and Directory Interface (JNDI) context for storing information about MBeans. This is, in effect, an alternate view of the registry and provides a logging mechanism for changes made to Managed Beans. ƒ It provides a set of custom Ant tasks, which allow the creation of MBeans and reading of descriptor files from within an Ant build script. Let’s see these services in action with the help of some examples. We’ll start with the transfer of ModelMBeanInfo information from code to XML file. 11.2.1 Creating metadata in XML Modeler makes it easy to manage application components by combining the flexibility of Model MBeans with the ease of writing XML configuration information about modeled components. The information about modeled components, which we also call metadata, describes these components for the Model MBeans that are registered in the MBeanServer. This allows management of these components via JMX. Information from JMX agents passes to the Model MBeans and on to the actual managed components through the mapping provided by this metadata information. As you saw in the previous section, this information can get very complex if done in code. Before we discuss how to create this information via an external file, let’s look at the registry provided by Modeler. The registry is the central component of Modeler, and it manages interaction with the MBeanServer for management components. Licensed to Tricia Fu 21MODULE 11: MANAGING COMPONENTS WITH MODELER 21 The central registry A registry is a central place for managing components in Modeler. It’s used to load, register, and manage MBeans. Its primary function is to load metadata information from an XML file, which by default is called mbeans-descriptors.xml, and then transfer this information from the XML syntax into ModelMBeanInfo for the Model MBeans and application components that it will be mapped to. The registry relies on classes in the org.apache.commons.modeler.modules package for reading this information. To use the registry, you use the factory method getRegistry(Object key, Object guard). It creates a default registry if one doesn’t exist for the key that you specify. This implies that there can be several registries in Modeler, differentiated by a key. This is true, and it gives you greater flexibility in using the same environment for separating multiple application components based on this key. However, before you can use this feature, you must enable it by calling the static method setUseContextClassLoader(boolean enable) and passing in a true value for the enable parameter. Doing so creates a HashMap for storing registries based on keys. If you haven’t enabled multiple registries and wish to use the single default registry, you need to pass a value of null to the key parameter of the getRegistry method. The second parameter for the getRegistry method accepts an object called guard and is used to prevent unauthorized access to the registry. If you want to restrict access to a registry of your application components, use the following procedure: 1. Create the registry for the first time by calling Registry registry = getRegistry(null, null) or Registry registry = getRegistry(yourKey, null), as the case may be. 2. Decide on an object that you wish to become the guard for your registry. It can be as simple as a String passphrase, or a more complex object. For the purposes of this example, let’s use the String passphrase “Jupiter”. 3. Set this passphrase to be the guard for your registry by calling registry.setGuard("Jupiter");. 4. The next time you want to access your registry, you’ll need to use getRegistry(null, "Jupiter") or getRegistry(yourKey, "Jupiter"). 5. Any component trying to call your registry without supplying the correct value of the guard will get a null value. Once a registry has been created, it can be used to load metadata information and associate it with Model MBeans. Let’s look at how this is done in the next section. Loading and registering metadata The registry provides several methods for loading the information from the mbeans-descriptors file. The default, and the most convenient method, is to read the data from an XML file. As we said before, the default XML file is called mbeans-descriptors.xml. You can load this file either as a URL object, as a File object, or as an InputStream, and then call loadMetadata(Object source) to pass any of these three objects in as a parameter. The method figures out the type of the object and tries to retrieve information from it accordingly. It isn’t necessary that this file be an XML file (or that it be called mbeans-descriptors). You can use serialized versions of your metadata information, but the file loaded via a URL, File, or InputStream must end with a .ser extension. Listing 11.6 shows an XML file that contains the metadata information for our TimerApplication from listing 11.1. It contains only basic information about the attributes delay and message. Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF Listing 11.6 XML descriptor for TimerApplication The root element, , clearly states that this is a file containing MBeans information. Next, information about individual MBeans is encapsulated in individual elements. The only MBean defined in the above XML file is TimerApplication, and it contains information about it in three attributes. The allowed attributes for the element are listed here: ƒ name—Class name of the MBean, without package information, or a unique name for it within the MBeanServer. ƒ description—Human- readable description of the MBean. ƒ type—Fully qualified type of the Mbean-managed component. ƒ domain—Domain in which this MBean should be registered in the MBeanServer. ƒ group—Name of the group this MBean belongs to. ƒ className—Fully qualified class name of the Model MBean implementation to use. The default is RequiredModelMBean, supplied by JMX. Note that none of the attributes are essential. This might seem strange, but it’s plausible because Modeler uses reflection on the managed component to try to figure out the best values for the attributes, if they aren’t provided in the XML descriptor file. We define two attribute elements, which correspond to the delay and message attributes of TimerApplication. Each attribute element contains three attributes, like the attributes for the mbean element, giving more information about each attribute. However, you can define several more attributes for the attribute element, as follows: Root element Define TimerApplication MBean Define attributes Licensed to Tricia Fu 23MODULE 11: MANAGING COMPONENTS WITH MODELER 23 ƒ name—Name of the property in your application component. ƒ description—Human-readable description of the attribute. ƒ type—Fully qualified type of the attribute. Primitive types can be used as is; Modeler converts them into corresponding wrapper classes. ƒ displayName—Name to use for this attribute if you want it to be different from the name attribute. ƒ getMethod—Name of the method used to retrieve the value of this property. However, it’s used only if the property doesn’t follow JavaBeans get/set conventions. ƒ setMethod—Name of the method used to set the value of this property. Again, use it only if the property doesn’t follow JavaBeans get/set conventions. ƒ is—Specifies whether this property represents a boolean value and, if it does, whether it uses isXXX as the method for getting the value of this property. You can use this attribute and indicate a true condition by setting it to true or yes. ƒ readable—Indicates that this property is readable by JMX agents if set to true or yes. ƒ writable—Indicates that this property is writable by JMX Agent agents if set to true or yes. The two element-attributes and and the surrounding element information are sufficient to describe our TimerApplication. Now we need to know how to register this information with the Modeler registry. Listing 11.7 shows a modified version of the code in listing 11.5. In listing 11.5, we used the buildModelMBeanInfo method to build the information for the TimerApplication. In listing 11.7, we’ll use the Modeler registry to load this information from an XML file. Before you run this code, make sure that you have commons-modeler.jar and commons-logging.jar libraries in your CLASSPATH in addition to jmxri.jar and jmxtools.jar. Listing 11.7 Using Modeler to build ModelMBeanInfo for TimerApplication package com.manning.commons.chapter11; import java.net.URL; import javax.management.ObjectName; import org.apache.commons.modeler.Registry; import com.sun.jdmk.comm.HtmlAdaptorServer; public class TimerApplicationModelerAgent { public TimerApplicationModelerAgent() { URL url= this.getClass().getResource("/timer_app_11.6.xml"); HtmlAdaptorServer adaptor = new HtmlAdaptorServer(); URL to file containing metadata information Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF TimerApplicationOriginal timerApp = new TimerApplicationOriginal(15000, "The time is now: "); ObjectName adaptorName = null; ObjectName timerAppName = null; try { Registry registry = Registry.getRegistry(null, null); registry.loadMetadata(url); timerAppName = new ObjectName("TimerDomain:name=timerApp"); registry.registerComponent( timerApp, timerAppName, "com.manning.commons.chapter11.TimerApplicationOriginal"); adaptorName = new ObjectName("TimerDomain:name=adaptor, port=8082"); adaptor.setPort(8082); registry.registerComponent( adaptor, adaptorName, "com.sun.jdmk.comm.HtmlAdaptorServer"); adaptor.start(); } catch(Exception e) { System.err.println(e); } } public static void main(String[] args) { TimerApplicationModelerAgent agent = new TimerApplicationModelerAgent(); } } Using Modeler transforms the code required to create ModelMBeanInfo dramatically. Once the registry has been created and loaded with basic metadata information about application components (using information from a file we’ve called timer_app_11.6.xml instead of the default mbeans-descriptors.xml), it’s a breeze to register components on it by calling the registerComponent method. Note that we also tried to register the HtmlAdaptorServer component with the registry. This doesn’t work completely because the timer_app_11.6.xml doesn’t contain metadata information about the HtmlAdaptorServer. At runtime, Modeler tries its best by guessing information about the component using Reflection; as you can see in figure 11.9, you only get a list of attributes and other reflexive operations when you look at its details from the browser by navigating to localhost:8082. Information about the current state of the attributes is unavailable because Modeler doesn’t know how to extract it. Try to register HtmlAdaptorServer (partially works) Register TimerApplication application component in MBeanServer Create registry without key or guard Load information from file in registry Licensed to Tricia Fu 25MODULE 11: MANAGING COMPONENTS WITH MODELER 25 Figure 11.9 HtmlAdaptorServer view throws up exceptions because it isn’t defined in the XML file One important element that we didn’t include in listing 11.6 is . Recall that we used a DescriptorSupport class in listing 11.5 to include detailed information about an attribute, a constructor, an operation, or an application component. A element can be used similarly in the mbeans-descriptors file to provide information about a surrounding element. The following code snippet shows how this can be done for the message attribute from listing 11.6: It should be easy to see how we would add more information in our mbeans-descriptors file now. For example, if we wanted to include information about the constructors for the TimerApplication, we only need to add the following element in this file, anywhere inside the element tag: 26 JAKARTA COMMONS ONLINE BOOKSHELF type="int" /> You can similarly add tags for the operations supported by your application component. Although TimerApplication doesn’t contain any specific operations, the following code snippet shows how you can add information about the set method of the delay attribute: Listing 11.8 puts all this information together in a final XML file for our TimerApplication. Listing 11.8 Final XML file containing metadata about TimerApplication 27MODULE 11: MANAGING COMPONENTS WITH MODELER 27 description="Message to print" type="java.lang.String" /> In addition to services for loading and registering MBeans from XML files, Modeler provides Ant tasks for integration within a build process. These tasks include loading, creating, and modifying MBeans. Let’s discuss these tasks next. 11.2.2 Modeler and Ant The Ant tasks for Modeler reside in the package org.apache.commons.modeler.ant. These tasks make it convenient to integrate lifecycle management of MBeans from within an Ant build file. Before you run any of these tasks, make sure that commons-modeler.jar, commons-logging.jar, jmxri.jar, and jmxtools.jar are set to be in the CLASSPATH for Ant. The easiest way to do this is to keep these libraries in the ANT_HOME\lib directory. In the next few sections, we’ll use Ant to load a Modeler registry, create an MBean and manipulate it, all from within an Ant build file. We’ll start with a rudimentary build file and add targets and tasks to it as we know more about them. Let’s start with the Registry task. Registry (mbeans-descriptors) task The Registry task, also called the mbeans-descriptors task, lets you load the registry information from a file or a URL. It also lets you store this information in a serialized copy. Listing 11.9 shows the first target in our build file for Ant. This target includes a task for loading the registry. Listing 11.9 First draft of the Modeler build file: loading and saving the registry Define class for mbeans-descriptors task Load and serialize registry Licensed to Tricia Fu 28 JAKARTA COMMONS ONLINE BOOKSHELF As you can see, this build file contains two targets. The target buildRegistry contains a single task called mbeans-descriptors, and the target all is the default target that will be used to run all the other targets. The mbeans-descriptors task accepts two attributes. The file attribute points to the location of the file from which the registry is to be loaded, and the out attribute points to the location of the serialized file where the registry is to be dumped. You can ignore the out attribute if you don’t wish to get a serialized copy of your registry, but the file attribute is required. Instead of the file attribute, however, you can use the attribute resource, which attempts to load the registry information independently using the current classloader’s getResource method. Other than loading the registry and dumping a serialized version of it, there isn’t much else you can do with the Registry task. The MBean and JMX-Operation tasks, discussed next, are much more useful. MBean and JMX-Operation tasks The MBean task, defined by the org.apache.commons.modeler.ant.MLETTask class, lets you create Managed Beans. Once an MBean has been created, you can invoke operations on it by using the JMX-Operation task, which is defined by the org.apache.commons.modeler. ant.JmxInvoke class. Listing 11.10 continues listing 11.9 and adds a new target called createMBean. Listing 11.10 Creating and managing MBeans via the Modeler build file Run all targets Define tasks createMBean target creates two MBeans Create HtmlAdaptorServer MBean d Start HtmlAdaptorServer MBean c Create TimerApplication MBean Licensed to Tricia Fu 29MODULE 11: MANAGING COMPONENTS WITH MODELER 29 Press Return key to continue... In this listing, we create two MBeans: HtmlAdaptorServer, which is created so we can view the status of the MBeans through a browser; and TimerApplication: c An MBean is created and registered in the MBeanServer using the MBean task. This task requires, at the very least, values for name and code attributes. The name attribute should be set to the desired ObjectName for this MBean, and the code attribute should be set to the fully qualified class name of the MBean. Thus, here we set name to DefaultDomain:name=HtmlAdaptorServer for the ObjectName of the HtmlAdaptorServer and code to com.sun.jdmk.comm. HtmlAdaptorServer. We also supply arguments for the TimerApplication class. If no arguments are supplied using the element tags, the no-argument constructor of the class specified by the code attribute is invoked. In this case, we do need to supply these arguments, because our TimerApplication class doesn’t contain a no-argument constructor. These arguments are supplied using the element tag, which takes two attributes: type specifies the data type of the argument; and value is, well, the value to be passed for this argument. Arguments are passed to the constructor in the order in which they’re specified. Warning: bug alert! Unfortunately, the org.apache.commons.modeler.ant. MLETTask class only works for String argument types. This means that when you run the code in listing 11.10 as a build file using Ant, the long argument type raises an error. Nothing can be done until the Commons Modeler team releases a fix that addresses this issue. In the meantime, you can use a patch, which is available from this book’s web site. d This shows how easy it is to invoke an operation on an MBean that has been registered in the MBeanServer using the jmx-operation task. To use this task, you need to identify the MBean on which the operation is to be performed, by specifying it as a value to the objectName attribute—in this case, the HtmlAdaptorServer. The actual method/operation that is to be invoked is specified by the operation attribute—in this case, the start method. You can supply arguments for this operation, similar to supplying arguments for the MBean constructors. Warning: The jmx-operation task also suffers from the String-values-only bug. A patch is available on this book’s web site. Using Ant, run this build file on a command line. When the build pauses waiting for your input, start a browser and navigate to localhost:8082. You’ll be able to see the MBeans that we’ve registered using this build file, and you can manage them via the browser interface. In the next section, we’ll look at our final Ant task, jmx-attribute, which lets you set the value of MBean attributes. Wait for user to continue Add new target createMBean Licensed to Tricia Fu 30 JAKARTA COMMONS ONLINE BOOKSHELF JMXSet (jmx-attribute) task The jmx-attribute task is specified using the org.apache.commons.modeler.ant.JmxSet class in the Modeler package. It lets you set the value of attributes via the Ant build file on MBeans that have been previously registered in an MBeanServer using the mbean task. Listing 11.11 completes our build file for Modeler by including an example of the jmx-attribute task. Listing 11.11 The complete Modeler build file showing all the tasks Press Return key to continue... Press Return key to continue... Define jmx- attribute task Set value of message attribute Set value of MBean attribute Licensed to Tricia Fu 31MODULE 11: MANAGING COMPONENTS WITH MODELER 31 The jmx-attribute task sets the value of an MBean attribute identified by the objectName and attribute attributes. Here, we’re changing the value of the message that is printed. Warning: Again, there is no type conversion for non-String types in the jmx-attribute task. A patch for this task is also available on the book’s web site. 11.3 Summary Modeler is a Commons component that simplifies building Model MBeans for JMX integration. It does this by providing a registry, which can load information about Model MBeans from a variety of sources. In this chapter, we reviewed the basic concepts of JMX and created a simple timing application. You should now understand the differences between Standard, Dynamic, Open, and Model MBeans. The chapter then introduced Modeler using a variety of examples. We explained how the Modeler registry works and gave examples of how to write XML files that contain metadata information about Model MBeans. We ended the chapter by showing examples of Ant and Modeler integration. In the next chapter, we’ll look at the CLI component, which is used to parse command-line options for Java applications. Licensed to Tricia Fu 32 JAKARTA COMMONS ONLINE BOOKSHELF Index HtmlAdapterServer, 7 Java Management Extensions. See JMX JMImplementation domain, 8 JMX, 1 Adapters, 6 Agent layer, 4 Agent Services, 4 creating a Dynamic MBean, 10 creating an agent, 5 creating Model MBean support, 14 Distributed layer, 7 Dynamic MBeans, 10 Instrumentation layer, 1 managing the application component, 7 MBean attributes, 8 Model MBeans, 14 ObjectName, 6 Open MBeans, 14 Standard MBeans, 9 steps to creating Model MBeans, 18 Managed Bean. See MBean managing applications with JMX, 1 MBean, 1 MBeanServer, 4 Model MBean creation, 18 Modeler ANT JMX-Operation task, 28 JMXSet task, 30 MBean task, 28 registry task, 27 creating metadata in XML, 20 creating registry, 21 introduction, 20 loading and registering metadata, 21 registry, 21 restricting access to registry, 21 services, 20 using Modeler with ANT, 27 RequiredModelMBean, 20 Writing a simple MBean interface, 3 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Command-line processing with CLI Vikram Goyal MODULE MANNING 12 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 12 Command-line processing with CLI 12.1 The process of command-line interface.................................................................................................................. 1 12.2 Introducing CLI ...................................................................................................................................................... 5 12.3 Summary............................................................................................................................................................... 25 Index ............................................................................................................................................................................. 26 Applications don’t work in isolation. A simple enough statement, which means that any application interacts with its environment, affecting the way it behaves. This interface to an application lets you modify application parameters that affect the application’s behavior, either at start-time or at runtime. In this module, we’ll look at the Command Line Interface (CLI) component from the Commons stable, which enables optimal processing of application parameters at an application’s start-time. (The previous module on Modeler discussed how you can manage application parameters at runtime.) We’ll first look at command-line processing. It might seem that the process is simple, but in reality it involves three stages, as used by CLI. We’ll then take a brief look at the CLI API. This will set us up for the CLI examples that follow. 12.1 Command-line processing Command-line processing is the means by which an application understands its initial environment. It sets the stage for subsequent processing. The process that creates the application supplies the parameters—or, as they’re known in CLI, the options—that define this environment. The application then parses these options and executes accordingly. As defined by CLI, there are three stages in the processing of options on a command line for an application: definition, parsing, and interrogation. Figure 12.1 shows these three stages. Figure 12.1 The three stages of command-line processing Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF We’ll introduce each of these stages with some examples. 12.l.1 Defining the options Option processing starts with a definition of each option for the application. This implies that you’re required to know in advance all the options possible for an application. This makes inherent sense, because the application itself needs to be aware of the environment and the factors that affect it. You can define options as part of the main application code or as a helper class that creates these options for you. By choosing the latter approach, you can separate application logic from option-creation tasks and let the main application code query the helper classes for the presence or absence of options. Listing 12.1 shows a very simple application that uses a helper class to define and parse an option. The application code requires that the user be welcomed with a personalized greeting. The application uses a helper class not only to define the user’s name as an option, but also to validate and return it. Listing 12.1 Option definition using a helper class package com.manning.commons.chapter12; public class MainApplicationV1 { public static void main(String args[]) { System.err.println("Hello " + HelperV1.processArgs(args)); } } class HelperV1 { static String processArgs(String args[]) { if(args.length != 2 || !args[0].equals("-n")) { usage(); System.exit(-1); } return args[1]; } static void usage() { System.err.println("java MainApplicationV1 -n [Your Name]"); } } As you may notice, the helper class defines options at the same time it’s validating them. For simple applications like this, option definition is almost always done with option validation. Here, the option definition declares that two arguments are required in order to successfully run the MainApplicationV1 application and that one of them must be –n. As you’ll see later, CLI makes the process of option definition more formal by using the Option and Options classes. Next, we’ll look at how to parse options and make them available for use. 12.1.2 Parsing options Option parsing is the step that accepts, separates, and prepares the options on a command line as defined in the previous stage, for processing by the application code. In listing 12.1, options are parsed in the Define options Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 3 processArgs() method (the options are also defined). Listing 12.2 shows a similar code snippet, where the options are processed further before being made available to the main application code. Listing 12.2 Parsing the options using the helper class package com.manning.commons.chapter12; import java.io.File; public class MainApplicationV2 { public static void main(String args[]) { String[] options = HelperV2.processArgs(args); System.err.println("Hello " + options[0] + ", your file size is " + new File(options[1]).length() + " bytes"); } } class HelperV2 { static String[] processArgs(String args[]) { if(args.length != 4 || !args[0].equals("-n") || !args[2].equals("-f") || !(new File(args[3]).isFile())) { usage(); System.exit(-1); } return new String[] { args[1], args[3] }; } static void usage() { System.err.println( "java MainApplicationV2 -n [Your Name] -f [Your File]"); } } As you can see in the processArgs method, options are accepted from the command line, segregated into the name and file options, and prepared for use (in this case, only the file option needs preparing) by the main application code. This is what parsing the options is all about: accepting, segregating, and preparing. CLI provides the Parser class (discussed later), which provides methods to parse the command-line options. Once the options have been parsed, the main application code needs to interrogate the helper class for the availability or nonavailability of the options and the value of those options. In listings 12.1 and 12.2, the main application accepts these options via return values for the processArgs method. In the next section, you’ll see a more advanced approach to interrogation. 12.1.3 Option interrogation By the time this stage of option processing is reached, the options have been defined and parsed. The main application code now needs to interrogate the process that parses these options via the command line and use these options accordingly. Guarantee that options have been parsed Only make option values available to main application Parse File argument to verify it exists as a file Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF Advanced interrogation requires the helper classes to provide a means to access meta-level information about the options. This meta-level information can vary based on application requirements, but at the very least, it should include information about the presence or absence of a particular option. In listing 12.3, the helper class from listing 12.2 has been modified to provide this information. It also includes methods that allow the main application to interrogate the option for its value, if it’s present. Listing 12.3 Helper class modified to provide interrogation methods package com.manning.commons.chapter12; import java.io.File; import java.util.HashMap; public class MainApplicationV3 { public static void main(String args[]) { HelperV3.processArgs(args); System.err.println( "Hello " + HelperV3.getOptionValue("n") + ", your file size is " + new File(HelperV3.getOptionValue("f")).length() + " bytes"); } } class HelperV3 { private static HashMap arguments = new HashMap(); static void processArgs(String args[]) { if(args.length != 4 || !args[0].equals("-n") || !args[2].equals("-f") || !(new File(args[3]).isFile())) { usage(); System.exit(-1); } arguments.put("n", args[1]); arguments.put("f", args[3]); } static void usage() { System.err.println( "java MainApplicationV3 -n [Your Name] -f [Your File]"); } static boolean hasOption(String optionName) { return arguments.containsKey(optionName); } static String getOptionValue(String optionName) { return (String)arguments.get(optionName); } } Process options, but don’t return them Store option values in HashMap Did command line contain an option? Return option value Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 5 The helper class has been modified significantly. It now processes the options using the processArgs() method, but it doesn’t return them to the calling application. Instead, it stores them in a HashMap and provides methods that allow the calling application class to interrogate the HashMapfor the availability of these options. Note that the hasOption() method is superfluous in this case, because all options are required; therefore, this method will always return true by the time it’s called. An optional option would make more sense than this method. The getOptionValue() method returns the user-provided values for these options from the HashMap. This method makes option value retrieval a generic process; you don’t need to provide separate methods for each option. Now that you know the different stages of processing a command-line interface, it’s time to look at the CLI component and introduce its API. We’ll take a brief look at this API and then move to some concrete CLI examples. 12.2 Introducing CLI The CLI component allows you to access command-line options by providing a consistent interface. Using CLI is simple and straightforward; the well-designed API separates the tasks of definition, parsing, and interrogation into their own packages. Let’s look at these packages more closely. Note: The CLI API has undergone major changes in the months prior to this book’s publication. The entire API has been completely rewritten, and it’s called CLI 2 to differentiate it from the previous version. Because the differences between CLI 1 and CLI 2 are so great, little work has been done to make sure CLI 2 is backward compatible, although CLI 1 is included in its own package in the CLI 2 API. CLI 2 classes are marked as such with package names that include CLI2. Newcomers to CLI are told to start with CLI 2, and older users of CLI are encouraged to switch to CLI 2. We hope that by the time you read this book, CLI 2 is stable enough that you won’t need to worry about CLI 1. This book concentrates on using CLI 2 only. 12.2.1 The CLI API The CLI API is divided into packages that roughly correspond to each of the stages of command-line processing. The definition stage requires several packages, whereas the parsing and interrogation stages are intermixed in the same package. Table 12.1 lists the packages and the stage they’re used in. Table 12.1 CLI API packages and corresponding stages Processing stage CLI API package(s) Definition org.apache.commons.cli2 org.apache.commons.cli2.builder org.apache.commons.cli2.option Parsing org.apache.commons.cli2.commandline Interrogation org.apache.commons.cli2 org.apache.commons.cli2.commandline In addition, the following three packages contain utility classes: org.apache.commons.cli2. resource, org.apache.commons.cli2.util, and org.apache. commons.cli2.validation. Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF The org.apache.commons.cli2 main package contains interface definitions for the definition and interface stages. The concrete classes for these interfaces are defined in org.apache.commons. cli2.option and org.apache.commons.cli2.commandline packages, respectively, whereas the package org.apache.commons.cli2.builder contains classes that help in the creation and definition of options. Typically, you’ll use classes in this builder package to create options for your application. Notice that the org.apache.commons.cli2.commandline package is shared between the parsing and interrogation stages. CLI 2 defines only one parsing class in this package: Parser. The rest of the classes in this package contain implementations of the CommandLine interface, used in the interrogation stage. The CommandLine interface represents parsed options and therefore provides methods to interrogate them. This interface is itself specified in the org.apache.commons.cli2 package. The utilities in CLI are divided into three packages based on functionality. The org.apache. commons.cli2.resource package contains a single class called ResourceHelper that loads messages from a resource bundle. This allows internationalized messages to be printed for error conditions while you’re using CLI. The org.apache.commons.cli2.util package contains several comparators. Comparators are classes that allow for strict ordering of elements in a list. CLI provides several comparators that enable the ordering of options when they’re parsed from the command line. Finally, the org.apache.commons.cli2.validation package contains classes that perform custom validation on options. The package contains an interface called Validator and several implementations that include FileValidator, UrlValidator, and so on. FileValidator, for example, validates a command-line option to find out whether the File object represented by the option exists, whether it’s a writeable file, and so on. Before we look at some examples of using CLI, we need to discuss the kinds of options that are available when you’re using CLI. Options can be passed to CLI in a variety of formats, as you’ll see in the next section. 12.2.2 CLI option formats Options come in a variety of formats. Some have a value associated with them, and these options are called arguments. Some are standalone and are called, well, options! Here are some examples of each CLI option format: ƒ Simple options—This is the basic option format, without any arguments or multiple values. For example, all boolean options are classified as simple options, because they represent command-line parameters that indicate a value by its presence or absence. In CLI, the Option interface and the OptionImpl abstract class together represent simple options. Note that simple options form the basis of more complex options. ƒ Argument options—These options represent an argument that is supplied on the command line. For example, in listings 12.1–12.3, all options are of this format, because each option requires a value to be passed in. In CLI, argument options are represented using the Argument interface and the ArgumentImpl class. A variation on this option lets you provide multiple arguments for a single option. This is implemented by allowing one or more than one child argument followed by a parent argument, as in the SourceDestArgument class. The copy command on most operating systems is an example where this class is useful; cp on Unix requires a single destination but can be provided with multiple source-file arguments. ƒ Group options—A collection of options is represented using the group option format. These options represent a choice that is available to the user for a conceptually similar task; only one may be used at a given time. Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 7 For example, the Tomcat Servlet Engine uses a group option to manage starting, stopping, and restarting the server. To start the server, you would normally use the command catalina –start. The same command is used with a different option, catalina –stop, to stop the server, and catalina –restart restarts a running server. In this case, the options start, stop, and restart are group options and are shown as [–start|-stop|-restart]. The Group interface and the GroupImpl class represent group options in CLI. However, these classes are also used to group options together on a command line. Thus a Group class may also represent a placeholder for all the options you define for your application. ƒ Parent options—A parent option is used to represent options that need to be qualified using a group option. Parent options can also have arguments; therefore a parent option is in essence a combination of a simple option, a group option, and an optional-argument option. For example, -log [-warn|-debug|-error|-fatal] represents a parent option. In this case, -log is the simple option, the values in the [] brackets represent the group options (also called the child options in this case), and the logFileName value represents the optional-argument option. As with the other options, the Parent interface and the ParentImpl class are used to define the parent options in CLI. ƒ Property options—This is a Java-specific option format, which handles the unique options of the format -Dproperty=value. There is no interface definition for these options, but the PropertyOption class that extends the OptionImpl class implements it in CLI. ƒ Command options—Command options represent options that are similar to giving commands on a Concurrent Versions System (CVS) Server. These options have the syntax command [command- options] [command-args]. Options can also have several aliases. The Command class in CLI implements this option and is a special case of the parent option. ƒ Switch options—A switch option can have either of two states. These are similar to simple options, but either state must be explicitly set. For example, suppose you have an application that can increase or decrease the brightness of your computer screen. If you run it from the command line, you can supply a switch option that can be set to either of the two states (+b or –b) that increases or decreases the brightness, respectively. In CLI, switch options are implemented as a special case of the parent options in the class Switch. This list sums up the available option formats in CLI. Keep these formats in mind when we show you how to use these formats in code in the next few sections. 12.2.3 CLI in action The following sections show examples of using CLI in a variety of scenarios. Before you run any of our examples, make sure commons-cli.jar is in your CLASSPATH. Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF CLI terminology We also need to be sure you understand the terminology associated with CLI. Figure 12.2 separates a typical option into its constituent parts and names each of these parts. Figure 12.2 The parts of an option As you can see, an option can consist of a short or long prefix, an argument, and the option itself, in either a short form or a long form. This terminology is important, because we’ll use it in most of our examples while constructing options. Using base classes We’ll start with the simplest possible case: a single option without any arguments, without a long form, and with default prefixes. Listing 12.4 shows the code. Listing 12.4 Simplest case of using CLI package com.manning.commons.chapter12; import java.util.List; import java.util.ArrayList; import org.apache.commons.cli2.Group; import org.apache.commons.cli2.Option; import org.apache.commons.cli2.CommandLine; import org.apache.commons.cli2.option.GroupImpl; import org.apache.commons.cli2.commandline.Parser; import org.apache.commons.cli2.option.DefaultOption; public class CLIApplicationV1 { public static void main(String args[]) throws Exception { DefaultOption optionName = new DefaultOption( "-", "--", false, "n", "Print standard greeting", null, null, c Define option Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 9 true, null, null, 'n'); List options = new ArrayList(); options.add(optionName); Group optionGroups = new GroupImpl( options, "Options", "All the Options", 1, 1); Parser parser = new Parser(); parser.setGroup(optionGroups); CommandLine commandLine = parser.parseAndHelp(args); if(commandLine != null && commandLine.hasOption(optionName)) System.err.println("Hello NoName"); } } c This example starts with the creation of the solitary option that we want to use in our application. By tracking whether the user supplied the option called n on the command line, we’ll print the greeting “Hello NoName”. To do this, an option is created using the DefaultOption class. This class requires several parameters, which specify the following: Short prefix Long prefix No bursting Option name Help No aliases No burst aliases Make it a mandatory option with no arguments, no child options, and a unique character ID Supplying so many parameters for a single option seems like an arduous task. CLI provides the Builder classes to make this task easier; we’ll discuss them in the next section. de Once the option has been created, it needs to be set in a group. Here we put the option we just created in an ArrayList, and then the list is used to create a group called Options. We specify a group description, the minimum number of options allowed, and the maximum number of options allowed. The GroupImpl class is used to create the group; but like options, we can use a Builder class to create this group (discussed in the next section). d Put option in a List c Define option e Create group to hold options f Create parser t g Set parser to look for group options h Parse arguments i Was optionName on the command Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF fg A parser is created next to parse the arguments on the application’s command line. It’s seeded with the group of options created; this is how the parser understands what options to look for on the command line. h The parser is called on to parse the arguments. The parseAndHelp method of the parser class prints a default help message on the screen if the expected options aren’t supplied. You can instead use the parse method, which throws an OptionException in the same case. i The CommandLine interface is used to represent the results of the parsing. Here we test whether the result of the parsing was successful by checking if it isn’t null. When you use parseAndHelp, if a required option is missing, the CommandLine evaluates to null (when you’re using parse it doesn’t matter, because it throws an OptionException). The CommandLine interface provides methods to evaluate the presence of options and the value of options that expect arguments. You can do this by calling the hasOption and getValue methods, respectively. These methods accept either the Option class or the option name as a parameter. In this case, we’ve used hasOption with the Option class; but using hasOption is superfluous here, because we have a single mandatory option and CommandLine will evaluate to null if it isn’t present. In the next section, you’ll see how to simplify the creation of options and groups by using the Builder classes. Using Builder classes In this section, we’ll rework listing 12.4 to use the Builder classes from the org.apache. commons.cli2.builder package. The classes in this package are useful for creating options and groups with only the desired parameters, leaving the rest of the parameters for CLI to fill in as defaults. Listing 12.5 shows how to use these classes. Listing 12.5 Using the Builder classes package com.manning.commons.chapter12; import java.util.List; import java.util.ArrayList; import org.apache.commons.cli2.Group; import org.apache.commons.cli2.Option; import org.apache.commons.cli2.CommandLine; import org.apache.commons.cli2.commandline.Parser; import org.apache.commons.cli2.builder.GroupBuilder; import org.apache.commons.cli2.builder.DefaultOptionBuilder; public class CLIApplicationV2 { public static void main(String args[]) throws Exception { DefaultOptionBuilder oBuilder = new DefaultOptionBuilder(); Option optionName = oBuilder .withShortName("n") .withDescription("Print standard greeting") .create(); c Create DefaultOptionBuilder Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 11 GroupBuilder gBuilder = new GroupBuilder(); Group optionGroups = gBuilder .withOption(optionName) .create(); Parser parser = new Parser(); parser.setGroup(optionGroups); CommandLine commandLine = parser.parseAndHelp(args); if(commandLine != null && commandLine.hasOption(optionName)) System.err.println("Hello NoName"); } } c Using the Builder classes, you only set the parameters that you want to set; the rest of the parameters are set to defaults. Of course, you need to provide minimum parameters; for example, while creating an option using DefaultOptionBuilder, you must supply, at the very least, either the short name or the long name. Here we supply the short name and the description using the methods withShortName and withDescription, respectively. d Similarly, the group is created using the GroupBuilder class. Since we needed to add only one option in this case, we used the method withOption only once to set up this group. The rest of the parameters, such as the minimum and maximum options allowed, are left as defaults of 0 and Integer.MAX_VALUE, respectively. The rest of the code remains unchanged from listing 12.4. The parser is set up with the group, and parseAndHelp is called to parse the arguments on the command line. The command line is then queried for the option n. Note: If you’re wondering about the slightly weird syntax used to create the option and the group using the Builders in listing 12.5, then you aren’t alone. Many developers are confused by the following syntax: oBuilder .withShortName("n") .withDescription("Print standard greeting") .create(); At first glance, it seems that the code should not compile. After all, in normal circumstances, you might think that a method call like withShortName, would return a void type (there’s no reason to expect anything else, because you’re setting a property, like a setter method); therefore, the second method call (withDescription on a void type) should fail compilation. It works because all the methods in the Builder classes that set a parameter return the Builder instance that is being worked on, after setting the parameter. This allows multiple method calls on the same line to set as many parameters as you like, because they’re being called on the most up-to- date Builder instance. d Create GroupBuilder Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF In the next section, we’ll look at how to create options with arguments, which lets you pass a value along with the option. Creating options with arguments To create arguments for options, we’ll use the ArgumentBuilder class. This class creates instances of ArgumentImpl, which is a special type of option. You can create instances of this class directly; but as with options and groups in the previous section, it’s better to use the Builder class, because then you only set the parameters you need. Listing 12.6 creates two options with arguments using this Builder class. Listing 12.6 Using ArgumentBuilder to create options with arguments package com.manning.commons.chapter12; import java.util.List; import java.util.ArrayList; import org.apache.commons.cli2.Group; import org.apache.commons.cli2.Option; import org.apache.commons.cli2.Argument; import org.apache.commons.cli2.CommandLine; import org.apache.commons.cli2.commandline.Parser; import org.apache.commons.cli2.builder.GroupBuilder; import org.apache.commons.cli2.builder.ArgumentBuilder; import org.apache.commons.cli2.builder.DefaultOptionBuilder; public class CLIApplicationV3 { public static void main(String args[]) throws Exception { GroupBuilder gBuilder = new GroupBuilder(); ArgumentBuilder aBuilder = new ArgumentBuilder(); DefaultOptionBuilder oBuilder = new DefaultOptionBuilder(); Argument aName = aBuilder .withDescription("Your Name") .withName("name") .withMinimum(1) .withMaximum(1) .withDefault("NoName") .create(); Option oName = oBuilder .withArgument(aName) .withShortName("n") .withDescription("Enter your name") .create(); Argument aAge = aBuilder .withDescription("Your Age") .withName("age") Create Builders c Create Name argument d Create Name option Create Age argument Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 13 .withMinimum(1) .withMaximum(1) .create(); Option oAge = oBuilder .withArgument(aAge) .withShortName("a") .withDescription("Enter your age") .create(); Group gOptions = gBuilder .withOption(oName) .withOption(oAge) .create(); Parser parser = new Parser(); parser.setGroup(gOptions); CommandLine commandLine = parser.parseAndHelp(args); if(commandLine != null) { int age; String name = (String)commandLine.getValue(oName, "Pretty Face"); if(commandLine.hasOption(oAge)) { try{ age = Integer.parseInt((String)commandLine.getValue(oAge)); } catch(Exception e) { age = 0; } if(age > 80) System.err.println(name + ", you are still young at heart!"); else System.err.println(name + ", you are still young!"); } } } } Creating options with arguments requires that you first create an argument instance, specifying its parameters, and then use this instance to create the option with its own parameters: c The name argument is created with at least one value and at most one value expected and a default value of NoName (as expected, this value will be used if we don’t supply a value on the command line for the option this argument is related to). d The option for this argument is created next. The argument is assigned to the name option by calling the withArgument method. e Once both the options have been created, they’re put together in a group in. Notice that multiple options are assigned to the group using the withOption method. f The command line is parsed as before, and the result is stored in the CommandLine interface. g Interrogate CommandLine f Parse CommandLine e Bundle options in group Create Age option Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF g This interface is interrogated to find out whether it contains values for options that we specified. If not, default values are used. Notice that when commandLine.getValue(oName, "Pretty Face") is invoked, a further default value of Pretty Face is used. But we already specified a default value for the name option in c. Which one will be used if no value is specified on the command line? CLI gives preference to the default value specified during interrogation over the default value specified at definition; therefore, Pretty Face is used. In the listing, each argument has only one value attached to it. What happens when we want more than one value for each argument? For example, suppose you’re creating an application that creates various geometrical shapes. For each shape, the user is supposed to provide measurement values. Thus, to create a rectangle, the user must supply the width and the height and specify that a rectangle should be created. An example command line would be java CLIApplicationV4 –rect -circle Both width and height are arguments to the rect argument option. Both must be provided, and both act as multiple arguments to a single option. This is different from specifying setMinimum(2) on the rect argument, because each value needs to be for a different subargument. To solve this issue of a command line with multiple arguments, the rect argument option must become a parent of the width and height arguments, as illustrated in listing 12.7. Listing 12.7 Creating an option with multiple arguments using a parent-child relationship package com.manning.commons.chapter12; import java.util.List; import java.util.ArrayList; import org.apache.commons.cli2.Group; import org.apache.commons.cli2.Option; import org.apache.commons.cli2.Argument; import org.apache.commons.cli2.CommandLine; import org.apache.commons.cli2.commandline.Parser; import org.apache.commons.cli2.builder.GroupBuilder; import org.apache.commons.cli2.builder.ArgumentBuilder; import org.apache.commons.cli2.builder.DefaultOptionBuilder; public class CLIApplicationV4 { public static void main(String args[]) throws Exception { GroupBuilder gBuilder = new GroupBuilder(); ArgumentBuilder aBuilder = new ArgumentBuilder(); DefaultOptionBuilder oBuilder = new DefaultOptionBuilder(); Argument aWidth = aBuilder .withName("width") .withMinimum(1) .withMaximum(1) .create(); Argument aHeight = Create Width child argument Create Height child argument Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 15 aBuilder .withName("height") .withMinimum(1) .withMaximum(1) .create(); Group gArgs = gBuilder .withOption(aWidth) .withOption(aHeight) .create(); Option oRectangle = oBuilder .withChildren(gArgs) .withShortName("rect") .create(); Group gOptions = gBuilder .withOption(oRectangle) .create(); Parser parser = new Parser(); parser.setGroup(gOptions); CommandLine commandLine = parser.parseAndHelp(args); if(commandLine != null) { // process these values } } } The child arguments are created as normal arguments. However, once created, they’re set in a group; then this group is used by the parent argument option to indicate that they’re child arguments, not normal arguments, using the withChildren method. By doing this, the rect argument option is established as the parent argument for the two child arguments width and height. Before we move on, notice that in listing 12.6, the Age option had to be parsed and validated before it could be used. CLI provides a way to validate values provided for options with arguments. In the next section, we’ll look at some common validators and how to use them. Using validators with arguments In this section, we’ll create a validator to validate alphanumeric values and use it, along with other predefined validators, to validate a variety of argument values. In the org.apache.commons.cli2.validation package, CLI provides several validators that can be used to validate argument values and convert them to the right type, if required. These predefined validators are listed in table 12.2. Create group to hold parent option Create group to hold child arguments Create parent option Set child group Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF Table 12.2 Predefined validators in CLI Validator class Validates… FileValidator Existing File and Directory instances and attributes such as readable, writeable, and hidden. If the value is valid, then the validator converts it to File instance from a String value. UrlValidator Valid allowed protocols. ClassValidator Valid class names. If required, the validator tries to load and/or instantiate the class. DateValidator Date formats and ranges specified by the user. If the range and/or format is valid, the validator converts the String value to a Date value. EnumValidator Permitted Strings from a range specified by the user. NumberValidator Number formats and ranges specified by the user. If the range and/or format is valid, the validator converts the String value to a Number value. As expected, nothing stops you from adding to this list of validators. You can create your own custom validators by implementing the Validator interface (in the org.apache.commons.cli2. validation package) and defining its lone method, validate. If the argument values fail to pass validation in this method, you need to throw the InvalidArgumentException, as the predefined validators do. Let’s see how to create a custom validator and then use it to define some arguments. We’ll also use some of the predefined validators: FileValidator, DateValidator, and EnumValidator. To begin, listing 12.8 shows a custom validator that is used to validate argument values to make sure they’re alphanumeric. Any special characters in an argument value will cause the InvalidArgumentException to be thrown. Listing 12.8 Creating a new validator for testing alphanumeric values package com.manning.commons.chapter12; import java.util.List; import java.util.ListIterator; import org.apache.commons.cli2.validation.Validator; import org.apache.commons.cli2.validation.InvalidArgumentException; public class AlphaNumericValidator implements Validator { public void validate(List values) throws InvalidArgumentException { for (final ListIterator i = values.listIterator(); i.hasNext();) { final String value = (String)i.next(); char charArray[] = value.toCharArray(); for(int k = 0; k < charArray.length; k++) { if(!Character.isLetterOrDigit(charArray[k])) Implement Validator interface Check whether character is valid Iterate over possible argument values Throw InvalidArgumentException Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 17 throw new InvalidArgumentException( charArray[k] + " is not allowed in " + value); } } } } The validate method receives a list of all possible values supplied on the command line for a particular argument. The method iterates through this list and breaks down each value into a character array. Each character in the array is examined to see whether it’s alphanumeric; if it isn’t, an InvalidArgumentException containing the value in the list is thrown. To use this validator in code, it must be attached to the argument, which needs to be validated by calling the withValidator method on the corresponding ArgumentBuilder. Listing 12.9 shows this in code along with some other predefined validators. Listing 12.9 Using predefined and custom validators package com.manning.commons.chapter12; import java.util.List; import java.util.HashSet; import java.util.ArrayList; import java.text.DateFormat; import org.apache.commons.cli2.Group; import org.apache.commons.cli2.Option; import org.apache.commons.cli2.Argument; import org.apache.commons.cli2.CommandLine; import org.apache.commons.cli2.commandline.Parser; import org.apache.commons.cli2.builder.GroupBuilder; import org.apache.commons.cli2.validation.Validator; import org.apache.commons.cli2.builder.ArgumentBuilder; import org.apache.commons.cli2.validation.FileValidator; import org.apache.commons.cli2.validation.EnumValidator; import org.apache.commons.cli2.validation.DateValidator; import org.apache.commons.cli2.builder.DefaultOptionBuilder; public class CLIApplicationV5 { public static void main(String args[]) throws Exception { GroupBuilder gBuilder = new GroupBuilder(); ArgumentBuilder aBuilder = new ArgumentBuilder(); DefaultOptionBuilder oBuilder = new DefaultOptionBuilder(); HashSet possibleEnumValues = new HashSet(); possibleEnumValues.add("Orange"); possibleEnumValues.add("Apple"); Validator vEnum = new EnumValidator(possibleEnumValues); c Create EnumValidator Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF ArrayList possbileDateFormats = new ArrayList(); possbileDateFormats.add( DateFormat.getDateInstance(DateFormat.SHORT)); possbileDateFormats.add( DateFormat.getDateInstance(DateFormat.MEDIUM)); DateValidator vDate = new DateValidator(possibleDateFormats); vDate.setMaximum(new Date()); vDate.setMinimum(new Date(0)); // 1/1/1970 FileValidator vFile = FileValidator.getExistingFileInstance(); vFile.setWritable(true); Validator vCustom = new AlphaNumericValidator(); Argument aCustom = aBuilder .withName("custom") .withMinimum(1) .withMaximum(1) .withValidator(vCustom) .create(); Option oCustom = oBuilder .withShortName("c") .withArgument(aCustom) .create(); Argument aDate = aBuilder .withName("date") .withMinimum(1) .withMaximum(1) .withValidator(vDate) .create(); Option oDate = oBuilder .withShortName("d") .withArgument(aDate) .create(); Argument aEnum = aBuilder .withName("enum") .withMinimum(1) .withMaximum(1) .withValidator(vEnum) .create(); Option oEnum = oBuilder .withShortName("e") d Create DateValidator with possible Date formats e Create FileValidator for existing writeable files f Create custom validator g Set correct validator on corresponding argument Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 19 .withArgument(aEnum) .create(); Argument aFile = aBuilder .withName("file") .withMinimum(1) .withMaximum(1) .withValidator(vFile) .create(); Option oFile = oBuilder .withShortName("f") .withArgument(aFile) .create(); Group gOptions = gBuilder .withOption(oCustom) .withOption(oDate) .withOption(oEnum) .withOption(oFile) .create(); Parser parser = new Parser(); parser.setGroup(gOptions); CommandLine commandLine = parser.parseAndHelp(args); if(commandLine != null) { // process the values } } } c The first validator that is created utilizes the EnumValidator to restrict the range of acceptable values for its corresponding argument. The EnumValidator’s sole constructor accepts a Set of values to which input must be restricted. d The second validator, DateValidator, defines two acceptable DateFormats and an acceptable range for input. The DateValidator constructor accepts a list of these valid formats. Once they’re created, we can use the methods setMaximum(Date date) and setMinimum(Date date) to set the range for acceptable dates. e The third validator is FileValidator. This class provides static methods to get instances of this validator for common file-validation tasks. For example, because validating the input for a valid existing file is such a common occurrence, the class provides a method called getExistingFileInstance(), which returns a FileValidator whose validation properties have been set to validate for an existing file. Similarly, you can use the static getExistingInstance and getExistingDirectoryInstance methods. Of course, you can create your own instances using the FileValidator constructor and set the required parameters yourself. Here, we’ve used the method g Set correct validator on corresponding argument Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF getExistingFileInstance, and added another requirement to make sure that the file instance is writable. f The final validator is the custom validator we created in listing 12.7, AlphaNumericValidator. It’s a simple validator that doesn’t require any other properties set for it, so it can be used as such. g The validators are attached to their arguments using the withValidator method. The rest of the code is the same as before. When you run this application, all validations are performed accordingly for each argument. If validation fails for any one argument, processing stops, and an error message from that validator is displayed on the screen. So far, we’ve been able to use the Builder classes associated with options to create them. However, in CLI, some of the options (like switch and property options) don’t have associated Builder classes. To use these properties, you must use the classes directly. In the next section, you’ll see examples of these options. Using switch and property options To handle Java style -DpropertyName=propertyValue options on the command line, CLI provides the PropertyOption class. Similar to handle switch options, where the option on the command line can have one of two discrete values, CLI provides the Switch class. Listing 12.10 shows how to use these classes. Listing 12.10 Using switch and property options package com.manning.commons.chapter12; import org.apache.commons.cli2.Group; import org.apache.commons.cli2.Option; import org.apache.commons.cli2.CommandLine; import org.apache.commons.cli2.option.Switch; import org.apache.commons.cli2.commandline.Parser; import org.apache.commons.cli2.builder.GroupBuilder; import org.apache.commons.cli2.option.PropertyOption; public class CLIApplicationV6 { public static void main(String args[]) throws Exception { GroupBuilder gBuilder = new GroupBuilder(); Switch sBright = new Switch( "+", "-", "b", null, "Increase or Decrease Brightness", true, null, null, 'b'); Option pOption = new PropertyOption(); c Create switch d Create PropertyOption Licensed to Tricia Fu MODULE 12: COMMAND-LINE PROCESSING WITH CLI 21 Group gOptions = gBuilder .withOption(sBright) .withOption(pOption) .create(); Parser parser = new Parser();