Node.js in Action version 16

MEAP Edition Manning Early Access Program Node.js in Action version 16 Copyright 2013 Manning Publications For more information on this and other Manning titles go to ©Manning Publications Co. Please post comments or corrections to the Author Online forum: to Roger Chen brief contents PART 1: NODE FUNDAMENTALS Chapter 1: Welcome to Node.js Chapter 2: Building a multi-room chat application Chapter 3: Node programming fundamentals PART 2: WEB APPLICATION DEVELOPMENT WITH NODE Chapter 4: Buiding Node web applications Chapter 5: Storing Node application data Chapter 6: Testing Node applications Chapter 7: Connect Chapter 8: Connect’s built-in middleware Chapter 9: Express Chapter 10: Web application templating Chapter 11: Deploying Node web applications PART 3: GOING FURTHER WITH NODE Chapter 12: Beyond web servers Chapter 13: The Node ecosystem APPENDIXES Appendix A: Installing Node and community add-ons Appendix B: Debugging Node ©Manning Publications Co. Please post comments or corrections to the Author Online forum: to Roger Chen 1 This chapter covers: What Node.js is JavaScript on the server Asyncronous and evented nature of Node Types of applications Node is designed for Sample Node programs So what is Node.js? It’s likely you have heard the term. Maybe you use Node. Maybe you are curious about it. At this point in time, Node is very popular and young (it debuted in 2009 ). It is the second most watched project on GitHub , has1 2 quite a following in its Google group and IRC channel and has more than 15,0003 4 community modules published in NPM (the package manager) . All this to say, 5 .there is considerable traction behind this platform Footnote 1 First talk on Node by creator Ryan Dahl - Footnote 2 Footnote 3 Footnote 4 Footnote 5 http://npmjs.orgm The official website ( defines Node as "a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. Node.js Welcome to Node.js ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 1 Licensed to Roger Chen uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices." In this chapter, we'll look at: Why JavaScript matters for server side development How the browser handles I/O using JavaScript How Node handles I/O on the server What are DIRTy applications and why they are a good fit for Node A sampling of a few basic Node programs Let's first turn our attention to JavaScript... For better or worse JavaScript is the world’s most popular programming language6 . If you have done any programming for the web, it is unavoidable. JavaScript, because of the sheer reach of the web, is the dream that"write once, run anywhere" Java had back in the 1990s. Footnote 6 "JavaScript: Your New Overlord” - Around the "Ajax revolution" in 2005, JavaScript went from being a "toy" language to something people write real and significant programs with. Some of the notable firsts were Google Maps and Gmail but today there are a host of web applications from Twitter to Facebook to Github. Since the release of Google Chrome in late 2008, JavaScript performance has been improving at an incredibly fast rate due to heavy competition between browser vendors (i.e. Mozilla, Microsoft, Apple, Opera, and Google). The performance of these modern JavaScript virtual machines are literally changing the types of applications we can build on the web . A compelling and frankly7 mind-blowing example of this is jslinux , a PC emulator running in JavaScript8 where you can load a Linux kernel, interact with the terminal session, and compile a C program all in your browser. Footnote 7 For some examples: Footnote 8 Node specifically uses V8, the virtual machine that powers Google Chrome, for server-side programming. V8 gives a huge boost in performance because it cuts out 1.1 Built on JavaScript ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 2 Licensed to Roger Chen the middleman, prefering straight compilation into native machine code over executing bytecode or using an interpreter. Because Node uses JavaScript on the server there are other benefits: Developers can write web applications in one language, which helps by: reducing the "context" switch between client and server development, and allowing for code sharing between client and server (e.g. reusing the same code for form validation or game logic). JSON is a very popular data interchange format today and it is native JavaScript. JavaScript is the language used in various NoSQL databases (e.g. CouchDB/MongoDB) so interfacing with them is a natural fit (e.g. MongoDB shell and query language is JS, CouchDB map/reduce is JS). JavaScript is a compilation target and there are a number of languages that compile to it already .9 F o o t n o t e 9 m Node uses one virtual machine (V8) that keeps up with the ECMAScript standard. In other words, you don’t have to wait for all the browsers to10 catch up to use new JavaScript language features in Node. Footnote 10 Who knew JavaScript would end up being a compelling language for writing server-side applications? Yet, due to its sheer reach, performance, and other characteristics mentioned previously, Node has gained a lot of traction. JavaScript is only one piece of puzzle though, the Node uses JavaScript is even moreway compelling. To understand the Node environment, let's dive into the JavaScript environment we are most familiar with: the browser. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 3 Licensed to Roger Chen Node provides an event-driven and asynchronous platform for server-side JavaScript. It brings JavaScript to the server much in the same way a browser brings JavaScript to the client. It is important to understand how the browser works in order to understand how Node works. Both are event-driven (i.e. use an event loop ) and non-blocking when handling I/O (i.e. use asynchronous I/O ). Let’s11 12 look an example to explain what that means. Footnote 11 Footnote 12 Take this common snippet of jQuery performing an Ajax request using XHR :13 Footnote 13 $.post('/resource.json', function (data) { console.log(data); }); // script execution continues In this program we perform an HTTP request for . When theresource.json response comes back, an anonymous function is called (a.k.a. the "callback" in this context) containing the argument , which is the data received from thatdata request. Notice that the code was written like this:not var data = $.post('/resource.json'); console.log(data); In this example, the assumption is that the response for resource.json would be stored in the variable and that the data when it is ready console.log function . The I/O operation (e.g. Ajax request) wouldwill not execute until then "block" script execution from continuing until ready. Since the browser is , if this request took 400ms to return, any other events happeningsingle-threaded on that page would wait until then before execution. You can imagine the poor user experience, for example, if an animation was paused or the user was trying to interact with the page somehow. I/O does not block execution I/O blocks execution until finished 1.2 Asynchronous and Evented: The Browser ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 4 Licensed to Roger Chen Thankfully that is not the case. When I/O happens in the browser, it happens outside of the event loop (i.e. main script execution) and then an "event" is emitted when the I/O is finished which is handled by a function (often called the14 "callback") as seen in figure 1.1: Footnote 14 For completeness, there are a few exceptions which "block" execution in the browser and usage ism typically discouraged: alert, prompt, confirm, and synchronous XHR. $.post('/resource.json', function (data) { console.log(data); }) EVENT LOOP Ajax request made for "resource.json" resource.json 4 1 2 user interaction - "onclick" another Ajax response User clicks, "onclick" event handled 1 2 ...waiting... 3 Finally, Ajax response for "resource.json" comes back and is handled in the callback4 Another Ajax response comes back3 Figure 1.1 An example of non-blocking I/O in the browser The I/O is happening asynchronously and not "blocking" the script execution allowing the event loop to respond to whatever other interactions or requests that are being performed on the page. This enables the browser to be responsive to the client and handle a lot of interactivity on the page. Make a note of that and let’s switch over to the server. For the most part, we are familiar with a conventional I/O model for server-side programming like the "blocking" jQuery example in section 1.2. Here's an example of how it looks in PHP: $result = mysql_query('SELECT * FROM myTable'); print_r($result); Execution stops until DB query completes 1.3 Asynchronous and Evented: The Server ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 5 Licensed to Roger Chen We are doing some I/O and the process is blocked from continuing until all the data has come back. For many applications this model is fine and is easy to follow. What may not be apparent is that the process has state/memory and is essentially doing "nothing" until the I/O is completed. That could take 10ms to minutes depending on the latency of the I/O operation. Latency can be unexpected as well, for example: The disk is performing a maintainence operation, pausing reads/writes A database query is slower because of increased load Pulling a resource from "" is sluggish today for some reason If a program blocks on I/O, what does the server do when there are more requests to handle? Typically this means that we use a multi-threaded approach. A common implementation is to use one thread per connection and setup a thread pool for those connections. You can think of threads as computational workspaces in which the processor works on one task. In many cases, a thread is contained inside a process and maintains its own working memory. Each thread handles one or more server connections. While this sounds like a natural way to delegate server labor, to developers who've been doing this a long time, managing threads within an application can be complex. Also, when a large number of threads is needed to handle many concurrent server connections, threading can tax operating system resources. Threads require CPU to perform context-switches as well as additional RAM. To illustrate this, let's look at a benchmark (shown in Figure 1.2) comparing NGINX and Apache. NGINX , if you aren’t familiar with it, is an HTTP server15 like Apache but instead of using the multi-threaded approach with blocking I/O, it uses an event loop with asynchronous I/O (like the browser and Node). Because of these design choices, NGINX is often able to handle more requests and connected clients, making it a more responsive solution.16 Footnote 15 Footnote 16 If you are more interested in this problem: ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 6 Licensed to Roger Chen Number of requests handled per second Programs utilizing an asynchronous and evented approach, like NGINX, are able to handle more communication between the client and the server Number of established client/server connections open 1 21 2 3 3 4 4 Such programs are also more responsive; in this example, at 3500 connections, each NGINX request is serviced roughly 3x faster Figure 1.2 WebFaction Apache / NGINX benchmark: In Node, I/O almost always is performed outside of the main event loop allowing the server to stay efficient and responsive like NGINX. This makes it much harder for a process to become I/O bound because I/O latency isn’t going to crash your server or use the resources it would if you were blocking. It allows the server to be lightweight on what are typically the slowest operations a server performs. .17 Footnote 17 More details at This mix of an event-driven and asynchronous model and the widely accessible JavaScript language helps open up a exciting world of data-intensive real-time applications. There actually is acronym for the types of applications Node is designed for: .DIRT It stands for applications. Since Node itself is verydata-intensive real-time lightweight on I/O, it is good at shuffling/proxying data from one pipe to another. It allows a server to hold open a number of connections while handling many requests and keeping a small memory footprint. It is designed to be responsive like the browser. Real-time applications are a new use-case of the web. Many web applications 1.4 DIRTy Applications ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 7 Licensed to Roger Chen now provide information virtually instantly, implementing things like: online whiteboard collaboration, realtime pinpointing of approaching public transit buses, and multiplayer games. Whether its existing applications being enhanced with real-time components or completely new types of applications, the web is moving towards more responsive and collaborative environments. These new types of web applications, however, call for a platform that can respond almost instantly to a large number of concurrent users. Node is good at this and not just for web, but also other I/O heavy applications. A good example of a DIRTy application written with Node is Browserling18 (shown in Figure 1.3). The site allows in-browser use of other browsers. This is extremely useful to front-end web developers as it frees them from having to install numerous browsers and operating systems solely for testing. Browserling leverages a Node-driven project called StackVM, which manages virtual machines (VMs), created using the QEMU ("Quick Emulator") emulator. QEMU emulates the CPU and peripherals needed to run the browser. Footnote 18 browserling.comm Figure 1.3 Browserling: Interactive cross-browser testing utilizing Node.js Browserling has VMs run test browsers and then relays the keyboard and mouse input data from the user's browser to the emulated browser which, in turn, streams the repainted regions of the emulated browser and redraws them on the canvas of the user's browser. This is illustrated in figure 1.4. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 8 Licensed to Roger Chen Emulated Browser (QEMU) Node.jsWebSockets HTML5 Canvas users mouse and keyboard events updated images in the form of data uris 1 2 1 In the browser, user's mouse and keyboard events are passed over WebSockets in real time to Node.js which in turn passes those to the emulator 2 The repainted regions of the emulated browser affected by the user interaction are streamed back though Node and WebSockets and drawn on the canvas in the browser Figure 1.4 Browserling Workflow Browserling also provides a complimentary project using Node called Testling allowing you to run a tests suites against multiple browsers in parallel from the19 command line. Footnote 19 testling.comm Browserling and Testling are good examples of a DIRTy applications and the infrustructure for building a scalable network applications like it are at play when you sit down to write your first Node application. Let's take a look at how Node's API provides this tooling right out of the box. Node was built from the ground up to have an event-driven and asynchronous model. JavaScript has never had standard I/O libraries, which are common to server-side languages, the "host" environment has always determined this for JavaScript. The most common "host" environment for JavaScript, the one most developers are used to, is the browser which is event-driven and asynchronous. Node tries to keep consistency between the browser by reimplementing common "host" objects, for example: Timer API (e.g. )setTimeout 1.5 DIRTy by Default ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 9 Licensed to Roger Chen Console API (e.g. )console.log Node also includes a "core" set of modules for many types of network and file I/O. These include modules for: HTTP, TLS, HTTPS, File System (POSIX), Datagram (UDP), and NET (TCP). The "core" is intentially small, low-level, and uncomplicated including just the building blocks for I/O based applications. 3rd-party modules build upon these blocks to offer greater abstractions for common problems. SIDEBAR Platform vs. Framework Node is a platform for JavaScript applications which is not to be confused with a framework. It is a common misconception to think of Node as "Rails" or "Django" for JavaScript when it is much lower level. Although if you are interested in frameworks for web applications, we will talk about a popular one for Node called Express later on in this book. After all this discussion you probably are wondering what does Node code look like? Let's cover a few simple examples: A simple asyncronous example A "hello world" web server An example of streams In section 1.2, we looked at this Ajax example using jQuery: $.post('/resource.json', function (data) { console.log(data); }); Let's do something similar in Node but instead using the file system ( )fs module to load from disk. Notice how similar the program isresource.json compared to the previous jQuery example. var fs = require('fs'); fs.readFile('./resource.json', function (er, data) { console.log(data); 1.5.1 Simple Async Example ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 10 Licensed to Roger Chen }) In this program, we read the file from disk. When all theresource.json data is read, an anonymous function is called (a.k.a. the "callback") containing the arguments , if any error occured, and , which is the file data The process "loops" behind the scenes able to handle any other operations that may come its way until the data is ready. All the evented/async benefits we talked about earlier are in play automatically. The difference here is that instead of making an Ajax request from the browser using jQuery, we are accessing the file system in Node to grab . This latter action is illustrated inresource.json figure 1.5. EVENT LOOP File request made for "resource.json" resource.json 4 1 2 another I/O operation finishes an event is triggered An event is triggered 1 2 ...waiting... 3 Finally, file data for "resource.json" comes back and is handled in the callback4 Another I/O operation finishes3 var fs = require('fs'); fs.readFile('./resource.json', function (err, data) { console.log(data); }); Figure 1.5 An example of non-blocking I/O in Node A very common use case for Node is building servers. Node makes it very simple to create different types of servers. This can feel odd if you are used to having a server 'host' your application (e.g. a PHP application hosted on Apache HTTP server). In Node, the server and the application . Here is an example ofare the same an HTTP server that simply responds to any request with 'Hello World': 1.5.2 Hello World HTTP Server ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 11 Licensed to Roger Chen var http = require('http'); http.createServer(function (req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello World\n'); }).listen(3000); console.log('Server running at http://localhost:3000/'); Whenever a request happens, the "callback" isfunction (req, res) fired and "Hello World" is written out as the response. This event model is akin to listening to an event in the browser. A 'click' could happen at any pointonclick so you setup a function to perform some logic whenever that happens. Here, Node provides a function for a 'request' whenever that happens. Another way to write this same server to make the 'request' event even more explicit is: var http = require('http'); var server = http.createServer(); server.on('request', function (req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello World\n'); }) server.listen(3000); console.log('Server running at http://localhost:3000/'); Node is also huge on streams and streaming. You can think of streams like arrays but instead of having data distributed over space, streams can be thought of as data . By bringing data in chunk by chunk, the developer is givendistributed over time the ability to handle that data as it comes in instead of waiting for it all to arrive before acting. Here's how we would stream :resource.json var stream = fs.createReadStream('./resource.json') stream.on('data', function (chunk) { console.log(chunk) }) stream.on('end', function () { console.log('finished') }) data events are fired whenever a new chunk of data is ready and is firedend Setting up an event listener for 'request' 'data' event fires when new chunk is ready 1.5.3 Streaming Data ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 12 Licensed to Roger Chen when all the chunks have been loaded. A 'chunk' can vary in size depending on the type of data. This low level access to the read stream allows you efficiently deal with data as it is read instead of waiting for it all to buffer in memory. Node also provides writable streams that you can write chunks of data to. One of those is the response ( ) object when a 'request' happens on an HTTP server.res Readable and writeable streams can be connected to make pipes much like the operator in shell scripting. This provides an efficient way to write out| (pipe) data as soon as it's ready without waiting for the complete resource to be read and then written out. Let's use our HTTP server from before to illustrate streaming an image to a client: var http = require('http'); var fs = require('fs'); http.createServer(function (req, res) { res.writeHead(200, {'Content-Type': 'image/png'}); fs.createReadStream('./image.png').pipe(res); }).listen(3000); console.log('Server running at http://localhost:3000/'); In this one-liner, the data is read in from the file ( )fs.createReadStream and being sent out ( ) to the client ( ) as it comes in. The event loop is.pipe res able to handle other events while data is being streamed. Node provides this "DIRTy by default" approach across multiple platforms including various UNIXes and Windows. The underlying asynchronous I/O library (libuv) was built specifically to provide a unified experience regardless of the parent operating system which allows programs to be more easily ported across devices and run on multiple devices if needed. Like any technology, Node is not a silver bullet, it just tries to tackle certain problems and open new possibilities. One of the interesting things about Node is it brings people from all aspects of the system together. Many come to Node being JavaScript client-side programmers, others are server-side programmers, and others are systems level programmers. Wherever you fit, we hope you have an understanding of where Node may fit in your stack. To review: Node is: Built on JavaScript Piping from a readable stream to a writeable stream. 1.6 In Summary ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 13 Licensed to Roger Chen Evented and asynchronous For data-intensive real-time applications In Chapter 2, we will build a simple DIRTy web application so you can see how a Node application works. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 14 Licensed to Roger Chen 2 This chapter covers: A first look at various Node components A sample real-time application using Node Server and client-side interaction In chapter 1, you learned how asynchronous development using Node differs from conventional/synchronous development. In this chapter, we'll take a practical look at Node by creating a small event-driven chat application. Don't worry if the details in this chapter seem over your head: the intent is to demystify Node development and give you a preview of what you'll be able to do once you've completed the book. This chapter assumes you: have experience with web application development, have a basic understanding of HTTP, and are familiar with jQuery. As you move through the chapter, you'll: Tour the application to see how it will work Review technology requirements and perform the intial application setup Serve the application's HTML, CSS, and client-side JavaScript Handle chat-related messaging using Use client-side JavaScript for the application's UI Let's start with an application overview--what it will look like and how it will behave when it's completed. Building a multi-room chat application ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 15 Licensed to Roger Chen The application allows users to chat online with each other by entering messages into a simple form as shown in figure 2.1. A message, once entered, is sent to all other users in the same chat room. Figure 2.1 Entering a message into chat When starting the application, a user is automatically assigned a guest name, but can change it by entering a command, as shown in figure 2.2. Chat commands are prefaced with a "/". 2.1 Application overview ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 16 Licensed to Roger Chen Figure 2.2 Changing one's chat name Similarly, a user can enter a command to create a new room (or change to it if it already exists), as shown in figure 2.3. When joining or creating a room, the new room name will be shown in the horizontal bar at the top of the chat application. The room will also be included in the list of available rooms to the right of the chat message area. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 17 Licensed to Roger Chen Figure 2.3 Changing rooms After changing to a new room the system will confirm it, as shown in figure 2.4. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 18 Licensed to Roger Chen Figure 2.4 The results of changing to a new room While the functionality of this application is deliberately barebones, it showcases important components and fundamental techniques needed to create a real-time web application. The application shows how Node can simultanously serve conventional HTTP data (like static files) and real-time data (chat messages). It also shows how Node applications are organized and how dependencies are managed. Let's now look at the technologies needed to implement this application. The chat application you'll create needs to: serve static files (such as HTML, CSS, and client-side JavaScript), handle chat-related messaging on the server, and handle chat-related messaging in the user's web browser To serve static files, we'll use Node's built-in "http" module. However, when serving files via HTTP, it's usually not enough to just send the contents of a file; you also should include the type of file being sent. This is done by setting the "Content-Type" HTTP header with the proper MIME type for the file. To look up1 these MIME types we will use a 3rd-party module called "mime". Footnote 1 To handle chat-related messaging, we could poll the server with Ajax. However, to make this application as responsive as possible, it will avoid using traditional Ajax means to send messages. Ajax uses HTTP as a transport mechanism and HTTP wasn't designed for real-time communication. When a message is sent using HTTP, a new TCP/IP connection must be used. Opening and closing connections takes time and the size of the data transfer is larger since HTTP headers are sent on every request. Instead of employing a solution reliant on HTTP, the application will prefer WebSocket , which was designed as a2 bidirectional lightweight communications protocol to support real-time communication. Footnote 2 As only HTML5 compliant browsers, for the most part, support WebSocket, the application will leverage the popular library which provides a number3 of fallbacks, including the use of Flash, should using WebSocket not be possible. 2.2 Application requirements and initial setup ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 19 Licensed to Roger Chen handles fallback functionality transparently, requiring no additional code or configuration. is covered more deeply in chapter 12: Beyond Web Servers. Footnote 3 Before we get in and actually do the preliminary work of setting up the application's file structure and dependencies, let’s talk more about how Node lets you simultaneously handle HTTP and WebSocket – one of the reasons why it’s so good for real-timey applications. Although we're avoiding the use of Ajax for sending and receiving chat messages, the application will still use HTTP to deliver the necessary HTML, CSS, and client-side JavaScript needed to set things up in the user's browser. Node can handle simultaneously serving HTTP and WebSocket easily, using a single TCP/IP port, as figure 2.5 shows visually. Node comes with a module that provides HTTP serving functionality. Then are a number of third-party Node modules, such as Express, which build upon Node's built-in functionality to make web serving even easier. We'll go into depth about how to use Express to build web application in chapter 9. In this chapter's application, however, we'll stick to the basics. Figure 2.5 Handling HTTP and WebSocket within a single application Now that you have a rough idea of the core technologies the application will use, let's start fleshing it out. SIDEBAR Need to install Node? If you haven't already installed Node, please head to appendix A now for instructions for doing so. 2.2.1 Serving HTTP and WebSocket ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 20 Licensed to Roger Chen To start constructing the tutorial application, create a project directory for it. The main application file will go directly in this directory. You'll need to add a "lib" subdirectory, within which some server-side logic will be placed. You'll need to create a "public" subdirectory where client-side files will be placed. Within the "public" subdirectory create a "javascripts" subdirectory and a "stylesheets"directory. Your directory structure should now look like figure 2.6. Note that while we've chosen to organize the files in a particular way in this chapter, Node doesn't require you to maintain any particular directory structure: application files can be organized in any way that makes sense to you. Figure 2.6 The skeletal project directory for the chat application. Now that we've established a directory structure, you'll want to specify the application's dependencies. An application dependency, in this context, is a module that needs to be installed to provide functionality needed by the application. Let's say, for example, that you were creating an application that needed to access data stored using a MySQL database. Node doesn't come with a built-in module that allows access to MySQL so you'd have to install a 3rd-party one and this would be considered a dependency. 2.2.2 Creating the application file structure ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 21 Licensed to Roger Chen Although you can create Node applications without formally specifying dependencies, it's a good habit to take the time to specify them. This way if you want others to use your application, or you plan on running it in more than one place, it becomes more straightforward to set up. Application dependencies are specified using a file. This filepackage.json is always placed in an application's root directory. A file consistspackage.json of a JSON expression that follows the CommonJS package descriptor standard4 and describes your application. In a file you can specify manypackage.json things, but the most important are the name of your application, the version, a description of what the application does, and the application's dependencies. Footnote 4m Listing 2.1 contains a package descriptor file that describes the functionality and dependencies of the tutorial application. Save this file as "package.json" in the root directory of the tutorial application. Listing 2.1 package.json: A package descriptor file Name of package Package dependencies If the content of this file seems a bit confusing, don't worry... you'll learn about files in depth in chapter 13: The Node Ecosystem.package.json 2.2.3 Specifying dependencies { "name": "chatrooms", "version": "0.0.1", "description": "Minimalist multi-room chat server", "dependencies": { "": "~0.9.6", "mime": "~1.2.7" } } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 22 Licensed to Roger Chen With a file defined, installing your application's dependenciespackage.json becomes trivial. The Node Package Manager (npm) is a utility that comes bundled5 with Node. It offers a great deal of functionality, allowing you to easily install third-party Node modules and globally publish any Node modules you yourself create. Another thing it can do is read dependencies from filespackage.json and install each of them with a single command. Footnote 5 Enter the following command in the root of your tutorial directory. If you look in the tutorial directory now there should be a newly created directory, as shown in figure 2.7. This directory contains yournode_modules application's dependencies. Figure 2.7 When using npm to install dependencies a "node_modules" directory is created. With the directory structure established and dependencies installed, you're ready to start fleshing out the application logic. As outlined earlier, the chat application needs to be capable of three basic things: 2.2.4 Installing dependencies npm install 2.3 Serving the application's HTML, CSS, and client-side JavaScript ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 23 Licensed to Roger Chen serving static files to the user's web browser handling chat-related messaging on the server handling chat-related messaging in the user's web browser Application logic will be handled by a number of files, some run on the server and some run on the client, as shown in figure 2.8. The JavaScript files run on the client need to be served as static assets, rather than executed by Node. Figure 2.8 When using npm to install dependencies a "node_modules" directory is created. In this section, tackling the first of those requirements, we'll define the logic needed to serve static files. We'll then add the static HTML and CSS files themselves. To create a static file server, we'll leverage some of Node's built-in functionality as well as the third-party "mime" add-on for determining a file MIME type. To start the main application file, create a file named in the rootserver.js of your project directory and put the listing 2.2's variable declarations in it. These 2.3.1 Creating a basic static file server ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 24 Licensed to Roger Chen declarations will give you access to Node's HTTP-related functionality, the ability to interact with the filesystem, functionality related to file paths, and the ability to determine a file's MIME type. The variable will be used to cache file data.cache Listing 2.2 server.js: variable declarations The built-in "http" module provides HTTP server and client functionality The built-in "fs" module provides filesystem-related functionality The built-in "path" module provides filesystem path-related functionality The add-on "mime" module provides the ability to derive a MIME type based on a filename extension The cache object is where the contents of cached files are stored Next you'll add three helper functions used for serving static HTTP files. The first will handle the sending of 404 errors for when a file is requested that doesn't exist. Add the following helper function to .server.js The second helper function handles serving file data. The function first writes the appropriate HTTP headers then sends the contents of the file. Add the following code to .server.js var http = require('http'); var fs = require('fs'); var path = require('path'); var mime = require('mime'); var cache = {}; SENDING FILE DATA AND ERROR RESPONSES function send404(response) { response.writeHead(404, {'Content-Type': 'text/plain'}); response.write('Error 404: resource not found.'); response.end(); } function sendFile(response, filePath, fileContents) { response.writeHead( 200, {"content-type": mime.lookup(path.basename(filePath))} ); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 25 Licensed to Roger Chen Accessing memory storage (RAM) is faster than accessing the filesystem. Because of this, it's common for Node applications to cache frequently used data in memory. Our chat application will cache static files to memory, only reading them from disk the first time they are accessed. The next helper, detailed in listing 2.3, determines whether or not a file is cached and, if so, serves it. If a file isn't cached, it is read from disk and served. If the file doesn't exist, an HTTP 404 error is returned as a response. Add this helper function, as well, to .server.js Listing 2.3 server.js: Serving static files Check if file is cached in memory Serve file from memory Check if file exists Read file from disk Serve file read from disk Send HTTP 404 response response.end(fileContents); } function serveStatic(response, cache, absPath) { if (cache[absPath]) { sendFile(response, absPath, cache[absPath]); } else { fs.exists(absPath, function(exists) { if (exists) { fs.readFile(absPath, function(err, data) { if (err) { send404(response); } else { cache[absPath] = data; sendFile(response, absPath, data); } }); } else { send404(response); } }); } } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 26 Licensed to Roger Chen For the HTTP server, an anonymous function is provided as an argument to , acting as a callback that defines how each HTTP request shouldcreateServer be handled. The callback function accepts two arguments: and request . When the callback executes, the HTTP server will populate theresponse request and response arguments with objects that, respectively, allow you to work out the details of the request and send back a response. You'll learn about Node's HTTP module in detail in chapter 4, Building Node web applications. Now, let's add the logic in listing 2.4 to create the HTTP server. Listing 2.4 server.js: Logic to create an HTTP server Create HTTP server, using anonymous function to define per-request behavior Determine HTML file to be served by default Translate URL path to relative file path Serve the static file While we've created the HTTP server in the code, we haven't actually added the logic needed to start it. Add the following lines which start the server, requesting that it listen on TCP/IP port 3000. Port 3000 is an arbitrary choice: any unused port above 1024 would work as well (a port under 1024 may also work if you're running Windows or, if in Linux or OS X, start your application using a privileged user such as "root"). CREATING THE HTTP SERVER var server = http.createServer(function(request, response) { var filePath = false; if (request.url == '/') { filePath = 'public/index.html'; } else { filePath = 'public' + request.url; } var absPath = './' + filePath; serveStatic(response, cache, absPath); }); STARTING THE HTTP SERVER server.listen(3000, function() { ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 27 Licensed to Roger Chen If you'd like to see what the application can do at this point you can start the server by entering the following into your command-line prompt: With the server running, visiting in your webhttp:// browser will simply result in the triggering of the 404 error helper and "Error 404: resource not found." will be displayed. While you've added static file handling logic, you haven't added the static files themselves. A running server can be stopped by using on the command-line.CTRL+C Next, let's move on to adding the static files necessary to get the chat application more functional. The first static file you'll add is the base HTML. Create a file in the "public" directory named "index.html" and place the HTML in listing 2.5 in it. The HTML will include a CSS file, set up some HTML DIV elements in which application content will be displayed, and will load a number of client-side JavaScript files. The JavaScript files provide client-side functionality, jQuery (for easy DOM manipulation), and a couple of application-specific files providing chat functionality. Listing 2.5 public/index.html: The HTML for the chat application console.log("Server listening on port 3000."); }); node server.js 2.3.2 Adding the HTML and CSS files Chat
©Manning Publications Co. Please post comments or corrections to the Author Online forum: 28 Licensed to Roger Chen DIV in which the current room name will be displayed DIV in which a list of available rooms will be displayed DIV in which chat messages will be displayed Form input element in which the user will enter commands and messages The next file we'll add defines the application's CSS styling. In the "public/stylesheets" directory create a file named "style.css" and put the CSS code in listing 2.6 in it. Listing 2.6 public/stylesheets/style.css: Application CSS
Chat commands:
  • Change nickname: /nick [username]
  • Join/create room: /join [room name]
body { padding: 50px; font: 14px "Lucida Grande", Helvetica, Arial, sans-serif; } a { color: #00B7FF; } #content { width: 800px; margin-left: auto; margin-right: auto; } #room { background-color: #ddd; ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 29 Licensed to Roger Chen The application will be 800 pixels wide and horizontally centered CSS rules for area in which current room name is displayed The message display area will be 690 pixels wide and 300 pixels high Allow DIV in which messages are displayed to scroll when it's filled up with content With the HTML and CSS roughed out, run the application and take a look using your web browser. The application should look like figure 2.9. Figure 2.9 The application-in-progress. The application isn't yet functional, but static files are being served and the margin-bottom: 1em; } #messages { width: 690px; height: 300px; overflow: auto; background-color: #eee; margin-bottom: 1em; margin-right: 10px; } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 30 Licensed to Roger Chen basic visual layout is established. With that taken care of, let's move on to defining the server-side chat message dispatching. Of the three things we said that the app had to do, we’ve already covered the first one, serving static files - and now we’re going to tackle the second – handling communication between the browser and server. Modern browsers are capable of6 using WebSocket to handle communication between the browser and the server. Footnote 6 provides a layer of abstraction over WebSocket and other transports for both Node and client-side JavaScript. It will fallback transparently to other WebSocket alternatives if it's not implemented in a web browser while keeping the same API. In this section, we’ll: briefly introduce you to and define the functionality you’ll need on the server-side add code that sets up a server add code to handle various chat application events, out of the box, provides virtual "channels" so instead of broadcasting every message to every connected user you can broadcast only to those who are "subscribed" to a specific channel. This functionality makes implementing chat rooms in our application quite simple, as you'll see later. is also a good example of the usefulness of what are called "event emitters" in Node. Event emitters are, in essence, a handy design pattern for organizing asynchronous logic. While you'll see some event emitter code at work in this chapter, we'll go into more detail in the next chapter. SIDEBAR Event emitters An event emitter is associated with a conceptual resource of some kind and can send and receive messages to and from the resource. Some examples of what the "resource" could be include a connection to a remote server or something more abstract, like a game character. The Johhny-Five project , in fact, leverages Node for robotics7 applications, using event emitters to control Arduino microcontrollers. Footnote 7m First, we'll start the server functionality and establish connection logic. Then, 2.4 Handling chat-related messaging using ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 31 Licensed to Roger Chen we'll define the functionality you need on the server side. To begin, append the following two lines to . The first line loadsserver.js functionality from a custom Node module that we'll define next. This module supplies logic to handle server-side chat functionality. The next line starts the server functionality, providing it with an already defined HTTP server so it can share the same TCP/IP port. You now need to create a new file, , inside the chat_server.js lib directory. Start this file by adding the following variable declarations. These declarations allow the use of and initialize a number of variables that define chat state. Next, add the logic in listing 2.7 to define the chat server function . Thislisten function is invoked in . It starts the server, limits theserver.js verbosity of's logging to console, and establishes how each incoming connection should be handled. The connection handling logic, you'll notice, calls a number of helper functions that we'll add next to .chat_server.js Listing 2.7 lib/char_server.js: Starting up a server 2.4.1 Setting up the server var chatServer = require('./lib/chat_server'); chatServer.listen(server); var socketio = require(''); var io; var guestNumber = 1; var nickNames = {}; var namesUsed = []; var currentRoom = {}; ESTABLISHING CONNECTION LOGIC exports.listen = function(server) { io = socketio.listen(server); io.set('log level', 1); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 32 Licensed to Roger Chen Start the server, allowing it to piggyback on the existing HTTP server Define how each user connection will be handled Assign user a guest name when they connect Place user in the "Lobby" room when they connect Handle user messages, name change attempts, and room creation/changes. Provide user with a list of occupied rooms on request. Define "cleanup" logic for when a user disconnects With the connection handling established, you'll now add the individual helper functions that will handle the application's needs. The chat application needs to handle the following types of scenarios and events: guest name assignment name change requests chat messages room creation user disconnection The first helper function you need to add is , which handlesassignGuestName the naming of new users. When a user first connects to the chat server, the user is placed in a chat room named "Lobby" and is called toassignGuestName assign them a name to distinguish them from other users. Each guest name is essentially the word "Guest" followed by a number that io.sockets.on('connection', function (socket) { guestNumber = assignGuestName(socket, guestNumber, nickNames, namesUsed); joinRoom(socket, 'Lobby'); handleMessageBroadcasting(socket, nickNames); handleNameChangeAttempts(socket, nickNames, namesUsed); handleRoomJoining(socket); socket.on('rooms', function() { socket.emit('rooms', io.sockets.manager.rooms); }); handleClientDisconnection(socket, nickNames, namesUsed); }); }; 2.4.2 Handling application scenarios and events ASSIGNING GUEST NAMES ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 33 Licensed to Roger Chen increments each time a new user connects. The guest name is stored in the variable for reference, associated with the internal socket ID. ThenickNames guest name is also added to , a variable in which names that are beingnamesUsed used are stored. Add the code in listing 2.8 to tolib/chat_server.js implement this functionality. Listing 2.8 lib/chat_server.js: Assigning a guest name Generate new guest name Associate guest name with client connection ID Let user know their guest name Note that guest name is now used Increment counter used to generate guest names The second helper function you'll need to add is . This function, shownjoinRoom in listing 2.9, handles logic related to a user joining a chat room. While having a user join a room is simple, requiring only a call to the method of a socket object, the application communicates related details tojoin the user and other users in the same room. The application lets the user know what other users are in the room and lets these other users know that the user is now present. Listing 2.9 lib/chat_server.js: Logic related to joining a room function assignGuestName(socket, guestNumber, nickNames, namesUsed) { var name = 'Guest' + guestNumber; nickNames[] = name; socket.emit('nameResult', { success: true, name: name }); namesUsed.push(name); return guestNumber + 1; } JOINING ROOMS function joinRoom(socket, room) { socket.join(room); currentRoom[] = room; ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 34 Licensed to Roger Chen Make user join room Note that user is now in this room Let user know they're now in a new room Let other users in room know that a user has joined Determine what other users are in the same room as the user If other users exist, summarize who they are Send the summary of other users in the room to the user If every user just kept their guest name, if would be hard to remember who's who. For this reason the chat application allows the user to request a name change. As figure 2.10 shows, a name change involves the user's web browser making a request via then receiving a response indicating success or failure. socket.emit('joinResult', {room: room});'message', { text: nickNames[] + ' has joined ' + room + '.' }); var usersInRoom = io.sockets.clients(room); if (usersInRoom.length > 1) { var usersInRoomSummary = 'Users currently in ' + room + ': '; for (var index in usersInRoom) { var userSocketId = usersInRoom[index].id; if (userSocketId != { if (index > 0) { usersInRoomSummary += ', '; } usersInRoomSummary += nickNames[userSocketId]; } } usersInRoomSummary += '.'; socket.emit('message', {text: usersInRoomSummary}); } } HANDLING NAME CHANGE REQUESTS ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 35 Licensed to Roger Chen Figure 2.10 A name change request and response Add the code in listing 2.10 to to define a functionlib/chat_server.js that handles requests by users to change their names. From the application's perspective, the users aren't allowed to change their name to anything beginning with "Guest" or use a name that's already in use. Listing 2.10 lib/chat_server.js: Logic to handle name request attempts Added listener for nameAttempt events Don't allow nicknames to begin with "Guest" If the name isn't already registered, register it Remove previous name to make available to other clients. Send an error to the client if the name's already registered function handleNameChangeAttempts(socket, nickNames, namesUsed) { socket.on('nameAttempt', function(name) { if (name.indexOf('Guest') == 0) { socket.emit('nameResult', { success: false, message: 'Names cannot begin with "Guest".' }); } else { if (namesUsed.indexOf(name) == -1) { var previousName = nickNames[]; var previousNameIndex = namesUsed.indexOf(previousName); namesUsed.push(name); nickNames[] = name; delete namesUsed[previousNameIndex]; socket.emit('nameResult', { success: true, name: name });[]).emit('message', { text: previousName + ' is now known as ' + name + '.' }); } else { socket.emit('nameResult', { success: false, message: 'That name is already in use.' }); } } }); } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 36 Licensed to Roger Chen Now that user nicknames are taken care of, you'll add a function that defines how a chat message sent from a user is handled. Figure 2.11 shows the basic process: the user emits an event indicating the room where the message is to be sent and the message text. The server then relays the message to all other users in the same room. Figure 2.11 Sending a chat message Add the following code to .'s lib/chat_server.js broadcast function is used to relay the message. Next, you'll add functionality that allows a user to join an existing room or, if it doesn't yet exist, create it. Figure 2.12 shows visually the interaction between the user and the server. SENDING CHAT MESSAGES function handleMessageBroadcasting(socket) { socket.on('message', function (message) {'message', { text: nickNames[] + ': ' + message.text }); }); } CREATING ROOMS ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 37 Licensed to Roger Chen Figure 2.12 Changing to a different chat room Add the following code to to enable room changing.lib/chat_server.js Note the use of's method.leave Finally add the logic below to to remove a user'slib/chat_server.js nickname from and when the user leaves the chatnickNames namesUsed application. With the server-side components fully defined, you're now ready to further develop the client-side logic. function handleRoomJoining(socket) { socket.on('join', function(room) { socket.leave(currentRoom[]); joinRoom(socket, room.newRoom); }); } HANDLING USER DISCONNECTIONS function handleClientDisconnection(socket) { socket.on('disconnect', function() { var nameIndex = namesUsed.indexOf(nickNames[]); delete namesUsed[nameIndex]; delete nickNames[]; }); } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 38 Licensed to Roger Chen Now that you've added server-side logic to dispatch messages sent from the browser, let's add the client-side JavaScript needed to communicate with the server. Client-side Javascript is needed to handle the following functionality: determine whether user input is a message or a chat command process chat commands and send information to the server Let's start with the first piece of functionality. The first bit of client-side JavaScript you'll add is a JavaScript prototype object that will process chat commands, send messages, and request room and nickname changes. In the "public/javascripts" directory create a file named "chat.js" and put the following code in it. This code starts JavaScript's equivalent of a "class" that takes a single argument, a socket, when instantiated. Next, add the following function to send chat messages. Add the following function to change rooms. 2.5 Using client-side JavaScript for the application's user interface 2.5.1 Relaying messages and name/room changes to the server var Chat = function(socket) { this.socket = socket; }; Chat.prototype.sendMessage = function(room, text) { var message = { room: room, text: text }; this.socket.emit('message', message); }; Chat.prototype.changeRoom = function(room) { this.socket.emit('join', { ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 39 Licensed to Roger Chen Finally, add the function defined in listing 2.11 for processing a chat command. Two chat commands are recognized: "join" for joining/creation a room and "nick" for changing one's nickname. Listing 2.11 public/javascripts/chat.js: Processing chat commands Parse command from first word Handle room changing/creation Handle name change attempts Return an error message if the command isn't recognized newRoom: room }); }; Chat.prototype.processCommand = function(command) { var words = command.split(' '); var command = words[0] .substring(1, words[0].length) .toLowerCase(); var message = false; switch(command) { case 'join': words.shift(); var room = words.join(' '); this.changeRoom(room); break; case 'nick': words.shift(); var name = words.join(' '); this.socket.emit('nameAttempt', name); break; default: message = 'Unrecognized command.'; break; } return message; }; ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 40 Licensed to Roger Chen You'll now start adding logic that interacts directly with the browser-based user interface using jQuery. The first functionality you'll add will be to display text data. In web applications there are, from a security perspective, two types of text data. There is "trusted" text data, which consists of text supplied by the application, and "untrusted" text data, which is text created by, or derived from, users of the application. Text data from users is considered untrusted because malicious users may intentionally submit text data that includes JavaScript logic contained in tags. This text data, if displayed unaltered to other users, can cause"}; console.log(ejs.render(template, context)); <script>alert('XSS attack!');</script> ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 297 Licensed to Roger Chen If you trust data being used in your template and don’t want to escape a context value in an EJS template you can use instead of in your template tag, as<%- <%= the following code demonstrates: Note that if you don’t like the characters used by EJS to specify tags, you can customize this, as the following example shows: Now that you know the basics of EJS, let’s look at things you can do with it that’ll make managing presentation of data easier. EJS provides support for “filters”—a feature that allows you to easily do lightweight data transformations without code. When using filters add a “:” to the opening characters of your EJS tag. For example: <%=: would be used for escaped EJS output <%-: for unescaped output Filters can also be “chained,” meaning you can put multiple filters in a single EJS tag and display the cumulative effect of all filters (similar to the "pipe" concept on *NIX systems). In the next few sections we’ll run through a number of var ejs = require('ejs'); var template = '<%- message %>'; var context = { message: ""}; } console.log(ejs.render(template, context)); var ejs = require('ejs'); = '{{' ejs.close = '}}' var template = '{{= message }}'; var context = {message: 'Hello template!'}; console.log(ejs.render(template, context)); 10.2.2 Manipulating template data using EJS filters ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 298 Licensed to Roger Chen filters that are useful in common scenarios. EJS filters are put into EJS tags. To give you an example of the usefulness of filters, imagine an application that allows users to let people know what movies they’ve watched. One bit of important information might be the most recent movie they’ve watched. The EJS tag in the template in the following example displays the last movie in an array of movies by using a filter to display only the last itemlast in an array: Note that is also a valid filter. If you want to get a specific item in thefirst list you could use the filter. The EJS tag get <%=: movies | get:1 %> would display the second item in the movies array (with item 0 being the first item). You can also use the filter to show properties if the context value is anget object rather than an array. EJS filters can also be used to change case. The EJS tag in the following template includes a filter that’ll capitalize the first letter in a context value, in this case changing the displayed value from “bob” to “Bob”: If you want to display a context value entirely to upper case you could use the filter. Conversely, using the filter would display the value inupcase downcase FILTERS THAT HANDLE SELECTION var ejs = require('ejs'); var template = '<%=: movies | last %>'; var context = {'movies': [ 'Bambi', 'Babe: Pig in the City', 'Enter the Void' ]}; console.log(ejs.render(template, context)); FILTERS FOR CASE MANIPULATION var ejs = require('ejs'); var template = '<%=: name | capitalize %>'; var context = {name: 'bob'}; console.log(ejs.render(template, context)); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 299 Licensed to Roger Chen lower case. Text can be sliced and diced by EJS filters. You can truncate text, append to prepended text, and even replace parts of your text. Truncating text to a certain character count allows you to eliminate the problem of long strings of text causing problems with HTML layouts. The following code, for example, will truncate the title text to 20 characters, displaying “The Hills are Alive”: If you want to truncate text to a certain number of words, an EJS filter supports that, too. In the previous example you could replace the EJS tag with <%=: to truncate the context value to twotitle | truncate_words:2 %> words. The output would then be “The Hills”. The "replace" filter uses String.prototype.replace(pattern) behind the scenes, so it accepts either a string or a RegExp. The following code shows an example of automatically abbreviating a word using an EJS filter: You can append text by adding a filter like .append:'some text' Similarly, you can prepend text using a filter like .prepend:'some text' EJS filters can also sort. Returning to the previously cited movie title example, you could use EJS filters to sort the movies by title and display the first item in alphabetical order, as illustrated by figure 10.3: FILTERS FOR TEXT MANIPULATION var ejs = require('ejs'); var template = '<%=: title | truncate:20 %>'; var context = {title: 'The Hills are Alive With the Sound of Critters'}; console.log(ejs.render(template, context)); var ejs = require('ejs'); var template = "<%=: weight | replace:'kilogram','kg' %>"; var context = {weight: '40 kilogram'}; console.log(ejs.render(template, context)); FILTERS THAT DO SORTING ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 300 Licensed to Roger Chen Figure 10.3 Visualizing the use of EJS filters to process arrays of text The following code would implements this: If you want to sort an array composed of objects, but you’d like to sort by comparing object properties, you can do this using filters, as the following code shows: Note the use of at the end of the filter chain. You use it becauseget:'name' what the sort returns is an object and you need to select what property of the object to display. var ejs = require('ejs'); var template = '<%=: movies | sort | first %>'; var context = {'movies': [ 'Bambi', 'Babe: Pig in the City', 'Enter the Void' ]}; console.log(ejs.render(template, context)); var ejs = require('ejs'); var template = "<%=: movies | sort_by:'name' | first | get:'name' %>"; var context = {'movies': [ {name: 'Babe: Pig in the City'}, {name: 'Bambi'}, {name: 'Enter the Void'} ]}; console.log(ejs.render(template, context)); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 301 Licensed to Roger Chen The EJS filter allows you to specify the property of an object that you wantmap subsequent filters to operate on. In the previous example we could use the filter chain using . As an alternative to having to specify the property using the map filter and having to specify the property to display using the filter,sort_by get you would use the filter to create an array from object properties. Themap resulting EJS would be <%=: movies | map:'name' | sort | first .%> Although EJS comes with filters for most common needs, you may want something beyond what EJS offers. If you want a filter that could, for example, round to an arbitrary decimal place, you’d find there’s no built-in filter to do this. Luckily, with EJS it’s easy to add your own custom filters, as listing 10.5 shows: Listing 10.5 custom_ejs.js: Defining your own custom EJS filters Simply define a function on the ejs.filters object The first argument is the input value the context or previous filter result As you can see, filters in EJS provide a great way to lessen the amount of logic you need to prepare data for display. Rather than doing these transformations to your data manually before rendering the template, EJS provides nice built-in mechanisms for doing that for you. THE "MAP" FILTER CREATING CUSTOM FILTERS var ejs = require('ejs'); var template = '<%=: price * 1.14 | round:2 %>'; var context = {price: 21}; ejs.filters.round = function(number, decimalPlaces) { number = isNaN(number) ? 0 : number; decimalPlaces = !decimalPlaces ? 0 : decimalPlaces; var multiple = Math.pow(10, decimalPlaces); return Math.round(number * multiple) / multiple; }; console.log(ejs.render(template, context)); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 302 Licensed to Roger Chen Because it’s awkward to store templates in files along with application code and doing this clutters up your code, we’ll show you how to use Node’s filesystem API to read them from separate files. Move to a working directory and create a file named containing theapp.js code in listing 10.6: Listing 10.6 app.js: Storing template code in files Note location of template file Data to pass to template engine Create HTTP server Read template from file Render template Send HTTP response 10.2.3 Integrating EJS in your application var ejs = require('ejs'); var fs = require('fs'); var http = require('http'); var filename = './template/students.ejs'; var students = [ {name: 'Rick LaRue', age: 23}, {name: 'Sarah Cathands', age: 25}, {name: 'Bob Dobbs', arg: 37} ]; var server = http.createServer(function(req, res) { if (req.url == '/') { fs.readFile(filename, function(err, data) { var template = data.toString(); var context = {students: students}; var output = ejs.render(template, context); res.setHeader('Content-type', 'text/html'); res.end(output); }); } else { res.statusCode = 404; res.end('Not found'); } }); server.listen(8000); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 303 Licensed to Roger Chen Next, create a child directory in it called . In this directory you’lltemplate keep your templates. Create a file named in the template/students.ejs directory and enter the code in listing 10.7 into it:template Listing 10.7 students.ejs: EJS template that renders an array of "students" EJS supports optional, in-memory caching of template functions. What this means is that EJS, after parsing your template file once, will store the function that’s created by the parsing. Rendering a cached template will be faster because the parsing step can be skipped. If you’re doing initial development of a Node web application, and want to see any changes you make to your template files reflected immediately, you won’t want to enable caching. But if you’re deploying an application to production, enabling caching is a quick, easy win. Notice how caching is conditionally enabled via the environment variable.NODE_ENV To try out caching, change the call to EJS’s function in the previousrender example to the following: Note that the option doesn’t necessarily have to be a file—you canfilename use a unique value that identifies whichever template you’re rendering. Now that you’ve learned how to integrate EJS with your Node applications, let’s look at how EJS can be used in a different way: in web browsers. <% if (students.length) { %>
    <% students.forEach(function(student) { %>
  • <%= %> (<%= student.age %>)
  • <% }) %>
<% } %>; CACHING EJS TEMPLATES var cache = process.env.NODE_ENV === 'production'; var output = ejs.render( template, {students: students, cache: cache, filename: filename} ); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 304 Licensed to Roger Chen We’ve shown an example that uses EJS with Node; now let’s now take a quick look at using EJS in the browser. To use EJS on the client-side you’ll first want to download the EJS engine to your working directory, as shown by the following commands: If you don't have by now then you can download the file at curl in your web browser. Once you download the file, then you can use EJS in yourejs.js client-side code. The following listing 10.8 shows a simple client-side application of EJS: Listing 10.8 browser_ejs.html: Using EJS to add templating capabilities to the client-side 10.2.4 Using EJS for client-side applications cd /your/working/directory curl -o ejs.js EJS example
©Manning Publications Co. Please post comments or corrections to the Author Online forum: 305 Licensed to Roger Chen Include jQuery library for DOM manipulation Placeholder for rendered template output Template to use to render content Data to use with template Wait until browser loads page Render template to DIV with ID ‘output’ You’ve learned how to use a fully featured Node template engine, so it’s time to look at the Hogan template engine, which deliberately limits the functionality available to templating code. Hogan.js is a template engine that was created by Twitter for its templating needs.3 Hogan is an implementation of the popular Mustache template language standard,4 which was created by GitHub’s Chris Wanstrath. Mustache takes a minimalist approach to templating. Unlike EJS, the Mustache standard deliberately doesn’t include conditional logic, nor any built-in content filtering capabilities other than escaping content to prevent XSS attacks. Mustache advocates that template code should be kept as simple as possible. Footnote 3 m Footnote 4 m In this section you’ll learn: How to create and implement Mustache templates in your application. The different template tags available in the Mustache standard. How to organize your templates using “partials.” How to fine-tune Hogan with your own delimiters and other options. To use Hogan in an application, or to try out the demos in this section, you’ll need to install Hogan in your application directory. You can do this by entering the following into the command line: 10.3 Using the Mustache templating languange with Hogan 10.3.1 Creating a template npm install hogan.js ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 306 Licensed to Roger Chen The following is a bare-bones example of Node using Hogan to render a simple template using a context. Running it will output the text “Hello template!”: Now that you know how to process Mustache templates with Hogan, let’s look at what tags Mustache supports. Mustache tags are conceptually similar to EJS’s tags. Mustache tags serve as placeholders for variable values, indicate where iteration is needed, and allow you to augment Mustache’s functionality and add comments to your templates. To display a context value in a Mustache template, include the name of the value in double braces. Braces, in the Mustache community, are known as “mustaches.” If you want to display the value for context item “name,” for example, you’d use the Hogan tag .{{name}} Like most template engines, Hogan escapes content by default to prevent XSS attacks. But to display an unescaped value in Hogan, you can either add a third mustache or prepend the name of the context item with an ampersand. Using the previous “name” example, you could display the context value unescaped by either using the or tag formats.{{{name}}} {{&name}} If you want to add a comment in a Mustache template, you can use this format: .{{! This is a comment }} Although Hogan doesn’t allow the inclusion of logic in templates, it does include an elegant way to iterate through multiple values in a context item using Mustache “sections.” The following context, for example, contains an item with an array of values: var hogan = require('hogan.js'); var template = '{{message}}'; var context = {message: 'Hello template!'}; var template = hogan.compile(template); console.log(template.render(context)); 10.3.2 Mustache tags DISPLAYING SIMPLE VALUES SECTIONS: ITERATING THROUGH MULTIPLE VALUES ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 307 Licensed to Roger Chen If you want to create a template that would display each student in a separate HTML paragraph, with output similar to the following, it would be a straightforward task using a Hogan template: The following template would produce the desired HTML: What if the value of the “students” item in the context data wasn’t an array? If the value was a single object, for example, the template would display it. But sections won’t display if the corresponding item’s value is undefined, false, or is an empty array. If you want your template to output a message indicating that values don’t exist for a section, Hogan supports what Mustache calls “inverted sections.” The following template code, if added to the previous student display template, would display a message when no student data exists in the context: var context = { students: [ { name: 'Jane Narwhal', age: 21 }, { name: 'Rick LaRue', age: 26 } ] };

Name: Jane Narwhal, Age: 21 years old

Name: Rick LaRue, Age: 26 years old


Name: {{name}}, Age: {{age}} years old


No students found.

{{/students}} ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 308 Licensed to Roger Chen In order to allow developers to augment Mustache’s functionality, the Mustache standard lets you define section tags that process template content rather than iterate through arrays. As an example use of a section lambda, listing 10.9 shows how you’d use a lambda to add Markdown support when rendering a template. Note that the example uses the module, which you’ll havegithub-flavored-markdown to install by entering intonpm install github-flavored-markdown your command line. In listing 10.9, the in the template gets rendered to **Name** when passing through the Markdown parser calledName by the lambda logic: Listing 10.9 hogan_lambda.js: Using a lambda in Hogan Require Markdown parser Mustache template also contains Markdown formatting The template context includes a lambda to parse Markdown in the template Lambdas allow you to easily implement things like caching and translation mechanisms into your templates. SECTION LAMBDAS: CUSTOM FUNCTIONALITY IN SECTION BLOCKS var hogan = require('hogan.js'); var md = require('github-flavored-markdown'); var template = '{{#markdown}}' + '**Name**: {{name}}' + '{{/markdown}}'; var context = { name: 'Rick LaRue', markdown: function() { return function(text) { return md.parse(text); }; } }; var template = hogan.compile(template); console.log(template.render(context)); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 309 Licensed to Roger Chen When writing templates you want to avoid unnecessarily repeating the same code in multiple templates. One way to avoid this is to create “partials.” Partials are templates used as building blocks and included in other templates. Another use of partials is to break up complicated templates into simpler templates. Listing 10.10, for example, uses a partial to separate the template code used to display student data from the main template: Listing 10.10 hogan_partials: The use of partials in Hogan Template code used for partial Main template code Compiling the main and partial templates Rendering the main template and partial PARTIALS: REUSING TEMPLATES WITHIN OTHER TEMPLATES var hogan = require('hogan.js'); var studentTemplate = '

Name: {{name}}, ' + 'Age: {{age}} years old

'; var mainTemplate = '{{#students}}' + '{{>student}}' + '{{/students}}'; var context = { students: [{ name: 'Jane Narwhal', age: 21 },{ name: 'Rick LaRue', age: 26 }] }; var template = hogan.compile(mainTemplate); var partial = hogan.compile(studentTemplate); var html = template.render(context, {student: partial}); console.log(html); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 310 Licensed to Roger Chen Hogan is fairly simple to use—once you’ve learned its vocabulary of tags you should be off and running. You may need to tweak only a couple of things as you use it. If you don’t like Mustache-style braces, you can change the delimiters Hogan uses by passing the method an option to override them. The followingcompile example shows compiling in Hogan using EJS-style delimiters. If you’d like to use section tags that don’t begin with the character ‘#’ after the opening mustaches, you can do that with another method option: compile . You might, for example, want to use a different tag format forsectionTags section tags in which lambdas are employed. The following listing 10.11 alters the earlier example in listing 10.9 to use an underscore prefix to differentiate the section tag from subsequent section tags that iterate rather thanmarkdown employ lambdas: Listing 10.11 hogan_custom: Using custom section tags in Hogan 10.3.3 Fine-tuning Hogan hogan.compile(text, {delimiters: '<% %>'}); var hogan = require('hogan.js'); var md = require('github-flavored-markdown'); var template = '{{_markdown}}' + '**Name**: {{name}}' + '{{/markdown}}'; var context = { name: 'Rick LaRue', _markdown: function(text) { return md.parse(text); } }; var template = hogan.compile( template, {sectionTags: [{o: '_markdown', c: 'markdown'}]} ); console.log(template.render(context)); Require Markdown parser Custom tag used in template Lambda for custom tag Custom opening and closing tags defined ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 311 Licensed to Roger Chen When using Hogan, you won’t have to change any options to enable caching. Caching is built into the function and enabled by default.compile Now that you’ve learned two fairly straightforward Node template engines, let’s look at the Jade template engine, which approaches the problem of dealing with presentation markup differently than EJS and Hogan. Jade offers an alternative way to specify HTML. The key difference between Jade5 and the majority of other templating systems is the use of meaningful whitespace. When creating a template in Jade, you use indentation to indicate HTML tag nesting. Indentation tags don’t have to be explicitly closed, which eliminates the problem of accidentally closing tags prematurely, or not at all. Using indentation also results in templates that are less visually dense and easier to maintain. Footnote 5m For a quick example of this at work, let’s look at how you’d represent the following snippet of HTML: This HTML could be represented using the following Jade template: Jade, like EJS, allows you to embed JavaScript, and you can use it on the server- or client-side. But Jade offers additional features such as support for template inheritance and “mixins.” Mixins allow you to define easily reusable mini-templates to represent the HTML used for commonly occurring visual elements, such as item lists and boxes. Mixins are very similar in concept to the Hogan.js “partials” which you learned about in the previous section. Template inheritance makes it easy to organize the Jade templates needed to render a single HTML page into multiple files. You’ll learn about these features in detail later in this section. 10.4 Templating with Jade
"Hello world!"
div.content#main strong "Hello world!" ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 312 Licensed to Roger Chen To install Jade in a Node application directory, enter the following into the command line: Installing Jade with the global flag is useful as well as it gives you access to-g a command-line tool that will allow you to quickly render a template tojade HTML. The following command-line use would result in the file being rendered to in the template/sidebar.jade sidebar.html directory. The Jade command-line tool gives you an easy way totemplate experiment with Jade syntax. In this section you'll learn: Jade basics such as specifying class names, attributes, and block expansion. How to add logic to your Jade templates using built-in keywords. How to organize your templates using inheritance, blocks and mixins. To get started, let's look at the basics of Jade usage and syntax. Jade uses the same tag names as HTML, but lets you lose the opening and closing and characters and uses indentation to express tag nesting.< > A tag can have one or more CSS classes associated with it by adding “..” A element with the “content” and “sidebar” classes applieddiv to it would be represented using the following: CSS IDs are assigned by adding “#” to the tag. You’d add a CSS ID of “featured_content” to the previous example using the following Jade npm install jade jade template/sidebar.jade 10.4.1 Jade basics div.content.sidebar ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 313 Licensed to Roger Chen representation. SIDEBAR DIV tag shorthand Because the tag is commonly used in HTML, Jade offers adiv shorthand way of specifying it. The following example will render to the same HTML as the previous example: Now that you know how to specify HTML tags and their CSS classes and IDs, let’s look at how to specify HTML tag attributes. Specifying tag attributes is done by enclosing the attributes in parentheses, separating the specification of each attribute with a comma. You can specify a hyperlink that’ll open in a different tab by using the following Jade representation: As the specification of tag attributes can lead to long lines of Jade, the template engine provides you with some flexibility. The following Jade is valid and equivalent to the previous example: You can also specify attributes that don’t require a value. The following Jade example shows the specification of an HTML form that includes a select element with an option preselected: div.content.sidebar#featured_content .content.sidebar#featured_content SPECIFYING TAG ATTRIBUTES a(href='', target='_blank') a(href='', target='_blank') ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 314 Licensed to Roger Chen In the previous code snippet, we also showed examples of tag content: “Select your favorite food:” after the tag; “Cheese” after the first tag; and thestrong option tag content “Tofu” after the second tag.option This is the normal way to specify tag content in Jade, but not the only way. Although this style is great for short bits of content, it can result in Jade templates with overly long lines if a tag’s content is lengthy. Luckily, as the following example shows, Jade will allow you to specify tag content using the character:| If the HTML tag, like the and tags, only ever accepts text ( script does not allow nested HTML elements) then the characters can be left out| entirely as the following example shows: Having two separate ways to express long tag content and short tag content helps you keep your Jade templates elegant. Jade also supports an alternate way to express nesting called “block expansion.” strong Select your favorite food: form select option(value='Cheese') Cheese option(value='Tofu', selected) Tofu SPECIFYING TAG CONTENT textarea | This is some default text | that the user should be | provided with. style h1 { font-size: 6em; color: #9DFF0C; } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 315 Licensed to Roger Chen Jade normally expresses nesting through indentation but sometimes indentation can lead to excess whitespace. For example, here is a Jade template which uses indentation to define a simple list of links: A more compact way to express the previous example is by using a Jade feature called “block expansion.” With block expansion, you add a colon after your tag to indicate nesting. The following code generates the same output as the previous listing in four lines instead of seven: Now that you’ve had a good look at how to represent markup using Jade, let’s look at how to integrate Jade with your web application. Data is relayed to the Jade engine in the same basic way as EJS. The template is first compiled into a function that is then called with a context in order to render the HTML output. The following is an example of this: KEEPING IT ORGANIZED WITH "BLOCK EXPANSION" ul li a(href='') Node.js homepage li a(href='') NPM homepage li a(href='') Nodebits blog ul li: a(href='') Node.js homepage li: a(href='') NPM homepage li: a(href='') Nodebits blog INCORPORATING DATA IN JADE TEMPLATES var jade = require('jade'); var template = 'strong #{message}'; var context = {message: 'Hello template!'}; ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 316 Licensed to Roger Chen In the previous example, the in the template specified a#{message} placeholder to be replaced by a context value. Context values can also be used to supply values for attributes. The following example would render . Now that you’ve learned how HTML is represented using Jade and how you can provide Jade templates with application data, let’s look at how you can incorporate logic in Jade. Once you supply Jade templates with application data, you need logic to deal with that data. Jade allows you to directly embed lines of JavaScript code into your templates which is how you define logic in your templates. Code like if statements, loops and declarations are common.for var Let's first look at the different ways Jade handles output when embedding JavaScript code. Prefixing a line of JavaScript logic with will execute the JavaScript without- including any value returned from the code in the template’s output. Prefixing JavaScript logic with will include a value returned from code, escaped to prevent= XSS attacks. But if your JavaScript generates code that shouldn’t be escaped, you can prefix it with . Table 10.1 summarizes these prefixes:!= var fn = jade.compile(template); console.log(fn(context)); var jade = require('jade'); var template = 'a(href = url)'; var context = {url: ''}; var fn = jade.compile(template); console.log(fn(context)); 10.4.2 Logic in Jade templates USING JAVASCRIPT IN JADE TEMPLATES ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 317 Licensed to Roger Chen Jade includes a number of commonly used conditional and iterative statements that can be written without prefixes. They are: if else if else case when default until while each unless Jade also allows you to define variables. The following shows two ways of assignment that are equivalent in Jade: The unprefixed statements have no output just like the prefix above.- Values passed in a context are accessible to JavaScript in Jade. In the following example we’ll read a Jade template from a file and pass the Jade template a context containing a couple of messages that we intend to display in an Array: Table 10.1 Prefixes used to embed JavaScript in Jadem Prefix Output = Escaped output (for untrusted or unpredictable values, XSS safe) != Output without escaping (for trusted or predictable values) - No output - var count = 0 count = 0 ITERATING THROUGH OBJECTS AND ARRAYS ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 318 Licensed to Roger Chen The Jade template would contain the following: The final HTML output would be: Jade also supports an non-JavaScript form of iteration: the statement. each statements allow you to cycle through arrays and object properties with ease.each The following is equilavent to the previous example but using instead:each You can perform cycling through object properties using a slight variation, as the following example shows: var jade = require('jade'); var fs = require('fs'); var template = fs.readFileSync('./template.jade'); var context = { messages: [ 'You have logged in successfully.', 'Welcome back!' ]}; var fn = jade.compile(template); console.log(fn(context)); - messages.forEach(function(message) { p= message - })

You have logged in successfully.

Welcome back!

each message in messages p= message each value, key in post div ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 319 Licensed to Roger Chen Sometimes templates need to make decisions on how data is displayed depending on the value of data. The following example illustrates a conditional in which, roughly 50 percent of the time, the script tag is outputted as HTML: Conditionals can also be written in Jade using the following cleaner, alternate form: If you’re writing a negated conditional, for example , you couldif (n != 1) use Jade’s keyword:unless Jade also supports a non-JavaScript form of conditional similar to a : the switch statement. statements allow you specify an outcome based on acase case number of template scenarios. strong #{key} p value CONDITIONALLY RENDERING TEMPLATE CODE - var n = Math.round(Math.random() * 1) + 1 - if (n == 1) { script alert('You win!'); - } - var n = Math.round(Math.random() * 1) + 1 if n == 1 script alert('You win!'); - var n = Math.round(Math.random() * 1) + 1 unless n == 1 script alert('You win!'); USING CASE STATEMENTS IN JADE ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 320 Licensed to Roger Chen The following example template shows how the case statement can be used to display results from the search of a blog in three different ways. If the search finds nothing, a message is shown indicating that. If a single blog post is found, it’s displayed in detail. If multiple blog posts are found, an statement is used toeach iterate through the posts, displaying their titles. With your templates defined, you next need to know how to organize them. As with application logic, you don’t want to make your template files overly large. A single template file should correspond to a conceptual building block: a page, a sidebar, or blog post content for example. In this subsection you’ll learn a few mechanisms that allow different template files to work together to render content: Structuring multiple templates with template inheritance Implementing layouts using block prepending/appending Template including Reusing template logic with mixins Template inheritance is one means of structuring multiple templates. The concept treats templates, conceptually, like classes in the object-oriented programming paradigm. One template can extend another, which can in turn extend another. You can use as many levels of inheritance as makes sense. As a simple example, let’s look at using template inheritance to provide a basic HTML wrapper with which to wrap page content. In a working directory, create a folder called in which you’ll put the example’s Jade file. For a pagetemplate template you’ll create a file called containing the following Jade:layout.jade case results.length when 0 p No results found. when 1 p= results[0].content default each result in results p= result.title 10.4.3 Organizing Jade templates STRUCTURING MULTIPLE TEMPLATES WITH TEMPLATE INHERITANCE ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 321 Licensed to Roger Chen The template contains the bare-bones definition of an HTMLlayout.jade page as well as two “blocks.” Blocks are used in template inheritance to define where a descendant template can provide content. In there is alayout.jade “title” block, allowing a descendent template to set the title, and a “content” block, allowing a descendent template to set what is to be displayed on the page. Next, in your working directory’s directory, create a file named template . This template file will populate the “title” and “content” blocks:page.jade Finally, add the logic in listing 10.12 (a modification of an earlier example in this section), which will display the template results, showing inheritance in action: Listing 10.12 inheritance.js: Template inheritance in action html head block title body block content extends layout block title title Messages block content each message in messages p= message var jade = require('jade'); var fs = require('fs'); var templateFile = './template/page.jade'; var iterTemplate = fs.readFileSync(templateFile); var context = {messages: [ 'You have logged in successfully.', 'Welcome back!' ]}; var iterFn = jade.compile( iterTemplate, {filename: templateFile} ); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 322 Licensed to Roger Chen Now that you’ve seen template inheritance in action, let’s look at another template inheritance feature: block prepending and appending. In the previous example the blocks in contained no content, whichlayout.jade made setting the content in the template straightforward. But if apage.jade block in an inherited template does contain content, this content can be built upon, rather than replaced, by descendent templates using block prepending and appending. This allows you to define common content and add to it, rather than replace it. The following template contains an additional block, “scripts,”layout.jade which contains content: a tag that’ll load the jQuery JavaScript library:script If you want the template to additionally load the jQuery UIpage.jade library, you could do this by using the template in the following listing 10.13: Listing 10.13 block_append.js: Using block append to load an additional JavaScript file console.log(iterFn(context)); IMPLEMENTING LAYOUTS USING BLOCK PREPENDING/APPENDING html head block title block scripts script(src='//') body block content extends layout baseUrl = "" block title title Messages block style link(rel="stylesheet", href= baseUrl+"themes/flick/jquery-ui.css") block append scripts ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 323 Licensed to Roger Chen This template extends the layout template Define the style block Append this scripts block to the one defined in layout But template inheritance isn’t the only way to integrate multiple templates. You also can use the Jade command.include Another tool for organizing templates is Jade’s command. Thisinclude command incorporates the contents of another template. If you added the line to the template from the earlier exampleinclude footer layout.jade you’d end up with the following template: This template would include the contents of a template named footer.jade in the rendered output of layout.jade, as illustrated in figure 10.4: script(src= baseUrl+"jquery-ui.js') block content count = 0 each message in messages - count = count + 1 script $(function() { $("#message_#{count}").dialog({ height: 140, modal: true }); }); != '
' + message + '
' TEMPLATE INCLUDING html head block title block style block scripts script(src='//') body block content include footer ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 324 Licensed to Roger Chen Figure 10.4 Jade’s include mechanism provides a simple way to include the contents of one template in another template during rendering This could be used, for example, to add information about the site, or design elements, to . You can also include non-Jade files by specifyinglayout.jade the file extension (e.g. ).include twitter_widget.html Although Jade’s command is useful for bringing in previously createdinclude chunks of code, it’s not ideal for creating a library of reusable functionality that you can share between pages and applications. For this, Jade provides the mixin command which lets you define reuseable Jade snippets. A Jade mixin is analogous to a JavaScript function. A mixin can, like a function, take arguments. These arguments can be used to generate Jade code. Let’s say, for example, your application handles a data structure similar to the following: If you want to define a way to output an HTML list derived from a given property of each object you could define a mixin like the following one to REUSING TEMPLATE LOGIC WITH MIXINS var students = [ {name: 'Rick LaRue', age: 23}, {name: 'Sarah Cathands', age: 25}, {name: 'Bob Dobbs', age: 37} ]; ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 325 Licensed to Roger Chen accomplish this: You could then use the mixin to display the data using the following line of Jade: By using template inheritance, statements, and mixins you can easilyinclude reuse presentation markup and can prevent your template files from becoming larger than they need to be. Now that you’ve learned how three popular HTML template engines work, you can use the technique of templating to keep your application logic and presentation organized. The Node community has created many template engines, which means if there’s something you don’t like about the three you’ve tried in this chapter you may want to check out other engines .6 Footnote 6m The Handlebars.js template engine, for example, extends the Mustache7 templating language, adding additional features such as conditional tags and globally available lambdas. Dustjs prioritizes performance and features such as8 streaming. For a list of Node template engines, check out the consolidate.js project , which provides an API that abstracts the use of template engines, making it easy9 to use multiple engines in your applications. But if the idea of having to learn any kind of template language at all seems distasteful, an engine called Plates allows10 you to stick to HTML, using its engine’s logic to map application data to CSS IDs and classes within your markup. mixin list_object_property(objects, property) ul each object in objects li= object[property] mixin list_object_property(students, 'name') 10.5 Summary ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 326 Licensed to Roger Chen Footnote 7 m Footnote 8 m Footnote 9 m Footnote 10 If you find Jade’s way of dealing with the separation of presentation and application logic appealing, you might also want to look at Stylus , a project that11 takes a similar approach to dealing with the creation of CSS. Footnote 11 m You now have the final piece you need to create professional web applications. In the next chapter we’ll look at deployment: how to make your application available to the rest of the world. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 327 Licensed to Roger Chen 11 This chapter covers: Choosing where to host your Node application Deploying a typical application Maintaining uptime and maximizing performance Developing a web application is one thing, but putting it into production is another. Every web platform requires knowledge of tips and tricks that increase stability and maximize performance, and Node is no different. When you’re faced with deploying a web application you’ll find yourself considering where to host it. You'll want to consider how to monitor your application and keep it running. You may also wonder what you can do to make it as fast as possible. In this chapter you’ll learn how to address these concerns for your Node web application. You'll learn how to: get your Node application up and running in public, how to deal with unexpected application termination, and how to get respectable performance. To start, let’s look at where you might choose to host your application. Deploying Node applications and maintaining uptime ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 328 Licensed to Roger Chen Most web application developers are familiar with PHP-based applications. When an Apache server with PHP support gets an HTTP request it’ll map the path portion of the requested URL to a specific file and PHP will execute the contents of the file. This functionality makes it easy to deploy PHP applications: you upload PHP files to a certain location of the filesystem and they become accessible via web browsers. In addition to being easy to deploy, PHP applications can also be hosted cheaply as servers are often shared between a number of users. But currently, deploying Node applications isn’t that simple or cheap. Whereas Apache is usually provided by hosting companies to handle HTTP, Node must handle HTTP itself. You either need to set up and maintain a Linux server or use Node-specific cloud hosting services offered by companies like Joyent, Heroku, Nodejitsu, VMware, and Microsoft. Node-specific cloud hosting services are worth looking into if you want to avoid the trouble of administering your own server or want to benefit from Node-specific diagnostics, such as Joyent SmartOS’s ability to measure what logic in a Node application performs slowest. The Cloud9 website, itself built using Node.js, offers a browser-based integrated development environment (IDE) in which you can clone projects from GitHub, work on them via the browser, then deploy them to a number of Node-specific cloud hosting services, as shown in the following table 11.1: 11.1 Hosting Node applications Table 11.1 Node-specific cloud hosting and IDE servicesm Name Website Heroku Nodejitsu VMware’s Cloud Foundry Microsoft Azure SDK for Node.js Cloud9 IDE ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 329 Licensed to Roger Chen Currently, running your own Linux server offers the most flexibility because you can easily install any related applications you need, such as database servers. Node-specific cloud hosting only offers you the choice of a small selection of related applications. Linux server administration is its own realm of expertise and, if you choose to handle your own deployment, it’s advisable to read up on your chosen Linux variant to be sure you’re familiar with setup and maintenance procedures. TIP VirtualBox If you’re new to server administration, you can experiment by running software like VirtualBox , which allows you to run a virtual1 Linux computer on your workstation, no matter what operating system your workstation runs. Footnote 1 m If you’re familiar with Linux server options, you may want to skim until you get to section 11.2, which is where we start to talk about cloud hosting. Right now though, let's talk about the options available to you: dedicated servers virtual private servers generate-purpose cloud servers Your server may either be a physical one, commonly known as a “dedicated,” server, or a virtual one. Virtual servers run inside physical servers and are assigned a share of the physical server’s RAM, processing power, and disk space. Virtual servers emulate a physical server and you can administer them in the same way. More than one virtual server can run inside a physical server. Dedicated servers are usually more expensive than virtual servers and often require more setup time as components may have to be ordered, assembled, and configured. Virtual private servers (VPSs), on the other hand, can be set up quickly as they are created inside preexisting physical servers. VPSs are a good hosting solution for web applications if you don't anticipate a quick growth in usage. VPSs are cheap and can be easily allocated additional 11.1.1 Dedicated and virtual private servers ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 330 Licensed to Roger Chen resources, when needed, such as disk space and RAM. The technology is established and many companies, such as Linode and Prgmr make it easy to get2 3 up and running. Footnote 2 m Footnote 3 m VPSs, like dedicated servers, can’t usually be created on-demand. Being able to handle quick growth in usage requires the ability to quickly add more servers, without relying on human intervention. For the ability to handle this you’ll need to use “cloud” hosting. Cloud servers are similar to VPSs in that they are virtual emulations of dedicated servers. But they have an advantage over dedicated servers and VPSs in that their management can be fully automated. Cloud servers can be created, stopped, started, and destroyed using a remote interface or API. Why would you need this? Let’s say, for example, that you’ve founded a company that has created Node-based corporate intranet software. You’d like clients to be able to sign up for your service and, shortly after sign up, receive access to their own server running your software. You could hire technical staff to set up and deploy servers for these clients around the clock, but unless you maintained your own data center they’d still have to coordinate with dedicated or VPS server providers to provide the needed resources in a timely manner. By using cloud servers you could have a management server send instructions via an API to your cloud hosting provider to give you access to new servers as needed. This level of automation enables you to deliver service to the customer quickly and without human intervention. Figure 11.1 visually represents how you can use cloud hosting to automate creating and destroying an application’s servers. 11.1.2 Cloud hosting ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 331 Licensed to Roger Chen Figure 11.1 Creating, starting, stopping, and destroying cloud servers can be fully automated The downside to using cloud servers is they tend to be more expensive than VPSs and can require some knowledge specific to the cloud platform. The oldest and most popular cloud platform is Amazon Web Services (AWS) .4 AWS itself consists of a range of different hosting-related services, like email delivery, content-delivery networks and lots more. Amazon’s Elastic Cloud Computing (EC2), one of AWS's central services, allows you to create servers in the cloud whenever you need them. Footnote 4 m EC2 virtual servers are called “instances” and can be managed using either the command line or a web-based control console, shown in figure 11.2. As command-line use of AWS takes some time to get used to, the web-based console is recommended for first time users. AMAZON WEB SERVICES ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 332 Licensed to Roger Chen Figure 11.2 The Amazon Web Services web console provides an easier way to manage Amazon cloud servers for new AWS users than the command line Luckily, because AWS’s are ubiquitous, it’s easy to get help online and find related tutorials such as Amazon’s “Get Started with EC2” .5 Footnote 5 m A more basic, easier-to-use cloud platform is Rackspace Cloud . Although a6 gentler learning curve may be appealing, Rackspace Cloud offers a smaller range of cloud-related products and functionality than does AWS and has a somewhat clunkier web interface. Rackspace Cloud servers either can be managed using a web interface, or with community-created command-line tools. Table 11.2 summarizes the hosting options: Footnote 6 m RACKSPACE CLOUD ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 333 Licensed to Roger Chen Now that you’ve had an overview of where you can host your Node application, let’s look at exactly how you’d get your Node application running on a server. Suppose you’ve created a web application that you want to show off or maybe you’ve created a commercial application and need to test it before putting it into full production. You’ll likely start with a simple deployment, and then do some work later to maximize uptime and performance. In this section, we’ll walk you through a simple, temporary Git deployment, as well as keeping the application up and running with Forever. Temporary deploys don’t persist beyond reboots, but they have the advantage of being quick to set up. Let’s quickly go through a basic deployment using a Git repository to give you a feel for the fundamental steps. Deployment is most commonly done by: Connecting to a server using SSH Installing Node and version control tools, like Git or Subversion, on the server (if needed) Downloading application files, including Node scripts and static assets like images and CSS stylesheets, from a version control repository to the server Starting the application To follow is an example of an application start after downloading the application files using Git: Table 11.2 Summary of hosting optionsm Suitable traffic growth Hosting option Cost Slow Dedicated $$ Linear Virtual private server $ Unpredictable Cloud $$$ 11.2 Deployment basics 11.2.1 Deploying from a Git repository ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 334 Licensed to Roger Chen Like PHP, Node doesn’t run as a background task. Because of this, the basic deployment we outlined would require keeping the SSH connection open. As soon as the SSH connection closes, the application will terminate. Luckily, it’s fairly easy to keep your application running using a simple tool. NOTE Automating deployment If you’d like to automate deployment of your Node application you can take a number of approaches. One way is to use a tool like Fleet , which allows you to deploy, using , to one or7 git push more servers. A more traditional way is by using Capistrano, as detailed in this blog post .8 Footnote 7 m Footnote 8 m Let’s say you’ve created a personal blog, using the Cloud 9 "Nog" blogging application , and want to deploy it, making sure that it stays running even if you9 disconnect from SSH or it crashes. Footnote 9 m The most popular tool in the Node community for dealing with this is Nodejitsu’s Forever . It keeps your application running after you disconnect from10 SSH and, additionally, restarts it if it crashes. Figure 11.3 shows, conceptually, how Forever works. Footnote 10m git clone cd hellonode node server.js 11.2.2 Keeping Node running ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 335 Licensed to Roger Chen Figure 11.3 The Forever tool helps you keep your application running, even if it crashes You can install Forever globally using the command.sudo NOTE The "sudo" command Often when installing an module (with the flag), younpm globally -g will need to prefix the command with the command in ordersudo11 to run the command with superuser privileges. The first time you use the command you will be prompted to enter yoursudo password, and after that then the command specifed after it will be run. Footnote 11m If you're following along, install it now using the following command: Once you’ve installed Forever, you could use it to start your blog, and keep it running, using the following command: If you wanted to stop your blog for some reason, you’d then use Forever’s stop command: sudo npm install -g forever forever start server.js forever stop server.js ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 336 Licensed to Roger Chen When using Forever, you can get a list of what applications the tool is managing by using its “list” command: Another useful capability of Forever is the optional ability to have your application restarted when any source files have changed. This frees you from having to manually restart each time you add a feature or fix a bug. To start Forever is this mode, use the flag:-w Although Forever is an extremely useful tool for deploying applications, you may want to use something more fully featured for long-term deploys. In the next section we’ll look at more “industrial strength” monitoring solutions and how to maximize application performance. Once a Node application is release-worthy, you’ll want to make sure it starts and stops when the server starts and stops as well as automatically restarts when the server crashes. It’s easy to forget to stop an application before a reboot or to forget to start an application after a reboot. You’ll also want to make sure you’re taking steps to maximize performance. Modern servers, for example, are generally multi-core and it makes sense when you’re running your application on a server with a quad-core CPU that you’re not using only a single core. If you’re using only a single core and your web application’s traffic increases to the extent that the single core doesn’t have spare processing capability to handle the traffic, your web application won’t be able to consistently respond. In addition to using all CPU cores, you’ll want to avoid using Node to host static files for high-volume production sites. Node can’t serve static files as efficiently as software optimized to do only this; instead, you’ll want to use forever list forever -w start server.js 11.3 Maximizing uptime and performance ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 337 Licensed to Roger Chen technologies like Nginx , which specializes in serving static files. You could also12 simply upload all your static files to a Content Delivery Network (CDN) , like13 Amazon S3, and reference those files in your application. Footnote 12m Footnote 13m In this section, we'll cover some server uptime and performance tips: Using Upstart to keep your application up and running through restarts and crashes Using Node's "cluster" API to utilize multi-core processors Serving Node application static files using NGINX Let’s say you’re happy with an application and want to market it to the world. You want to make dead sure that if you restart a server you don’t then forget to restart your application. You also want to make sure that if your application crashes it’s not only automatically restarted, but the crash is logged and you’re notified, which allows you to diagnose any underlying issues. In Ubuntu and CentOS, the two most popular Linux distributions for serving web applications, services are provided for application management and monitoring to help keep a tight rein on your applications. These services are mature, robust, and well-supported, and a proper deployment using them will maximize your uptime. Let’s start by looking at Upstart: Ubuntu’s solution to maintaining uptime. Upstart is a project that provides an elegant way to manage the starting and stopping of any Linux application, including Node applications. Modern versions of Ubuntu and CentOS 6 support the use of Upstart. You can install Upstart on Ubuntu, if it’s not already installed, using the following command: 11.3.1 Maintaining uptime USING UPSTART TO START AND STOP YOUR APPLICATION sudo apt-get install upstart ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 338 Licensed to Roger Chen You can install Upstart on CentOS, if it’s not already installed, using the following command: Once you’ve installed Upstart you’ll need to add an Upstart configuration file for each of your applications. These files are created in the directory/etc/init and are named something like . Themy_application_name.conf configuration files do not need to be marked as executable. The following steps will create an empty Upstart configuration file for the previous example application: Now, add the contents of listing 11.1 to your config file. This setup specifies to run the application upon startup of the server, and stop it upon shutdown. The section is what gets executed by Upstart:exec Listing 11.1 /etc/init/hellonode.conf: A typical Upstart configuration file sudo yum install upstart sudo touch /etc/init/hellonode.conf #!upstart author "mcantelon" description "hellonode" setuid "nonrootuser" start on (local-filesystems and net-device-up IFACE=eth0) stop on shutdown respawn console log env NODE_ENV=production exec /usr/bin/node /path/to/server.js ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 339 Licensed to Roger Chen Application author name Application name and/or description Run the application using as user "nonrootuser" Start application on startup after the file system and network is available Stop application on shutdown Restart application when it crashes Log STDIN and STDERR to /var/log/upstart/yourapp.log Set any environmental variables necessary to the application Command to execute your application This configuration will keep your process up and running, during server restarts and even if it crashes unexpectantly. All the application generated output will be sent to /var/log/upstart/hellonode.log and Upstart will manage the log rotation for you. Now that you’ve created an Upstart configuration file you can start your application using the following command: If your application was started successfully you will see: Upstart is highly configurable. Checkout the online cookbook for all the14 available options. Footnote 14 sudo start hellonode hellonode start/running, process 6770 ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 340 Licensed to Roger Chen SIDEBAR Upstart and respawning Upstart, by default when using the option, will continuallyrespawn reload your application on crashes unless the application is restarted 10 . You can change this limit using the times within 5 seconds respawn option where COUNT is the number oflimit COUNT INTERVAL times within the INTERVAL, which is in seconds. For example, 20 times in 5 seconds would be: If your application is reloaded 10 times within 5 seconds (the default), typically there is something wrong in the code or configuration, and it will never start successfully. Upstart will not try to restart after reaching the limit in order to save resources for other processes. It is a good idea to do health checks outside of Upstart that ideally provide alerts through email or some other means of quick communication back to the development team. A health check, for a web application, can simply be hitting the website and seeing if you get a valid response. You could roll your own or use tools such as Monit or Zabbix for this.15 16 Footnote 15 Footnote 16 With the ability to keep your application running regardless of crashes and server reboots, the next logical concern is performance, which Node’s cluster API can help with. Most modern computer CPUs have multiple cores. But a Node process uses only one of them when running. If you were hosting a Node application on a server and wanted to maximize the server’s usage you could manually start multiple instances of your application on different TCP/IP ports, and use a load balancer to distribute web traffic to these different instances, but that’s laborious to set up. To make it easier to use multiple cores for a single application, the cluster API was added. This API makes it easy for your application to run multiple “workers” that each do the same thing, and respond to the same TCP/IP port, but can be run respawn respawn limit 20 5 11.3.2 The cluster API: Taking advantage of multiple cores ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 341 Licensed to Roger Chen simultaneously using multiple cores. Figure 11.4 visually represents how an application’s processing would be organized, using the cluster API on a four-core processor. Figure 11.4 A visualization of a master spawning three workers on a four-core processor Listing 11.2 automatically spawns a master process and a worker for each additional core. Listing 11.2 cluster_demo.js: A demonstration of Node’s cluster API Determine number of cores server has Create a fork for each core Define work to be done by each worker var cluster = require('cluster'); var http = require('http'); var numCPUs = require('os').cpus().length; if (cluster.isMaster) { for (var i = 0; i < numCPUs; i++) { cluster.fork(); } cluster.on('death', function(worker) { console.log('Worker ' + + ' died.'); }); } else { http.Server(function(req, res) { res.writeHead(200); res.end('I am worker ID ' + process.env.NODE_CLUSTER_ID); }).listen(8000); } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 342 Licensed to Roger Chen Because masters and workers run in separate operating system processes, which is necessary if they’re to run on separate cores, they can’t share state through global variables. But the cluster API does provide a means of communicating between the master and the workers. Listing 11.3 shows an example use of the cluster API where messages are passed between the master and the workers. A count of all requests is kept by the master and whenever a worker reports handling a request, it is relayed to each worker. Listing 11.3 cluster_messaging.js: A demonstration of Node’s cluster API var cluster = require('cluster'); var http = require('http'); var numCPUs = require('os').cpus().length; var workers = {}; var requests = 0; if (cluster.isMaster) { for (var i = 0; i < numCPUs; i++) { workers[i] = cluster.fork(); (function (i) { workers[i].on('message', function(message) { if (message.cmd == 'incrementRequestTotal') { requests++; for (var j = 0; j < numCPUs; j++) { workers[j].send({ cmd: 'updateOfRequestTotal', requests: requests }); } } }); })(i); } cluster.on('death', function(worker) { console.log('Worker ' + + ' died.'); }); } else { process.on('message', function(message) { if (message.cmd == 'updateOfRequestTotal') { requests = message.requests; } }); http.Server(function(req, res) { ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 343 Licensed to Roger Chen Listen for messages from the worker Increase request total Send new request total to each worker Use a closure to preserve the value of worker Listen for messages from the master Update request count using master’s message Let master know request total should increase Node’s cluster API is a simple way of creating applications that take advantage of modern hardware. Although Node is an effective solution for serving dynamic web content, it’s not the most efficient way to serve static files such as images, CSS stylesheets, or client-side JavaScript. Serving static files over HTTP is a specific task for which specific software projects are optimized because they’ve focused primarily on this task for many years. Fortunately nginx , an open source web server optimized for serving static17 files, is easy to set up alongside Node to serve those files. In a typical nginx/Node configuration, nginx initially handles each web request, relaying requests that aren’t for static files back to Node. This configuration is represented visually in figure 11.5: Footnote 17 m res.writeHead(200); res.end('Worker ID ' + process.env.NODE_WORKER_ID + ' says cluster has responded to ' + requests + ' requests.'); process.send({cmd: 'incrementRequestTotal'}); }).listen(8000); } 11.3.3 Hosting static files and proxying ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 344 Licensed to Roger Chen Figure 11.5 You can use nginx as a proxy to relay static assets quickly back to web clients The configuration in the following listing 11.4, which would be put in the nginx configuration file’s section, implements this setup. The configuration filehttp conventionally stores in a Linux server’s directory at /etc ./etc/nginx/nginx.conf Listing 11.4 nginx_proxy.conf: An example configuration file that uses nginx to proxy Node.js and serve static files http { upstream my_node_app { server; } server { listen 80; server_name localhost; access_log /var/log/nginx/my_node_app.log; location ~ /static/ { root /home/node/my_node_app; if (!-f $request_filename) { return 404; } } location / { proxy_pass http://my_node_app; proxy_redirect off; ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 345 Licensed to Roger Chen IP and port of Node application Port on which proxy will receive requests Handle file requests for URL paths starting with “/static/” Define URL path proxy will respond to By using nginx to handle your static web assets, you’ll ensure that Node is dedicated to doing what it’s best at. In this chapter, we’ve introduced you to a number of Node hosting options, including Node-specific hosting, dedicated, virtual private server hosting, and cloud hosting. Each option suits different use cases. Once you’re ready to deploy a Node application to a limited audience, you can get up and running quickly, without having to worry about your application crashing, by using the Forever tool to supervise your application. But for long-term deployment you may want to automate your application’s starts and stops using Upstart. To get the most of your server resources, you can take advantage of Node’s cluster API to run application instances simultaneously on multiple cores. If your web application requires serving static assets such as images and PDF documents, you may also want to run the Nginx server and proxy your Node application through it. Now that you have a good handle on the in’s and out’s of Node web applications, including deployment, it’s a good time to look at all the other things Node can do. In the next chapter we’ll look at Node’s other applications, everything from building command-line tools to “scraping” data from websites. proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-NginX-Proxy true; } } } 11.4 Summary ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 346 Licensed to Roger Chen 12 In this chapter: Socket.IO TCP/IP networking in-depth Tools for interacting with the operating system environment Developing command-line tools Node's asynchronous nature makes it possible to do I/O intensive tasks that may otherwise be difficult or inefficient to do in a synchronous environment. We have mostly covered HTTP applications in this book but what about other kinds of applications? What else is Node useful for? The truth is that Node is tailored not only to HTTP, but to all kinds of general purpose I/O. In practice, this means you can build practically any type of application using Node, for example command-line programs, system administrative scripts, and realtime web applications. In this chapter you will learn how Node is capable of creating web servers that go beyond a traditional web server by communicating in realtime. You will also learn about some of the other APIs that Node provides that you can use to create other kinds of applications, like TCP servers or command-line programs. Specifically, we will cover: Socket.IO, which brings cross-browser realtime communication to web browsers TCP/IP networking in-depth, so you can implement all kinds of network applications with Node The APIs that Node provides to interact with the operating system directly Command line tools, including how to develop and work with them Beyond Web Servers ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 347 Licensed to Roger Chen Socket.IO is arguably the most well known module in the Node community.1 People who are interested in creating "realtime web applications", but have never heard of Node, usually hear about Socket.IO sooner or later, which then brings them to Node itself. Socket.IO allows you to write realtime web applications using a bi-directional communication channel between the server and client. At its simplest, Socket.IO has an API very similar to the WebSocket API , but has23 built-in fallbacks for older browsers where such features did not yet exist. Socket.IO also provides convenient APIs for broadcasting, volatile messages, and a lot more. These features have made Socket.IO very popular for web-based browser games, chat type apps, and streaming applications. Footnote 1m Footnote 2m Footnote 3m HTTP is a stateless protocol, meaning that the client is only able to make single, short-lived requests to the server, and the server has no real notion of connected or disconnected users. This limitation has initiated the standardization of the WebSocket protocol, which specifies a way for browsers to maintain a full-duplex connection to the server, allowing both ends to send and receive data simultaneously. These APIs allow for a whole new breed of web applications utilizing realtime communication between the client and server. The problem with the WebSocket protocol is it is not yet finalized, and while some browsers have begun shipping versions with WebSockets, there's still all the older versions out there, especially Internet Explorer. Socket.IO solves this by utilizing WebSockets when available in the browser, otherwise falling back to other browser-specific tricks to simulate the behavior that WebSockets provide, even in older browsers. In this section, you will build up two sample applications using Socket.IO: A minimal Socket.IO application that pushes the server's time to connected clients. A Socket.IO application that triggers page refreshes when CSS files are edited. After building the example apps, we'll show you a few more ways you can use Socket.IO by briefly revisiting the "upload progress" example from Chapter 4. Let's start with the basics. 12.1 Socket.IO ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 348 Licensed to Roger Chen So let's say you wanted to build a quick little web application that constantly updated the browser in realtime with the server's UTC time. An app like this would be useful to identify a possible difference between the client's and server's clocks. Now try to think of how you could build this application using the modulehttp or the frameworks you have learned about so far. While it is possible to get something working using a trick like long-polling, using Socket.IO provides a cleaner interface for accomplishing this. Implementing this app with Socket.IO is about as minimal as you can get, so let's built it. You can install Socket.IO using :npm Listing 12.1 shows the server-side code, so save this file for now and we can try it out when you have the client-side code as well. Listing 12.1 clock-server.js: Socket.IO Server that updates its clients with the time Upgrade the regular HTTP server into a Socket.IO server 12.1.1 Creating a minimal Socket.IO application npm install var app = require('http').createServer(handler); var io = require('').listen(app); var fs = require('fs'); var html = fs.readFileSync('index.html', 'utf8'); function handler (req, res) { res.setHeader('Content-Type', 'text/html'); res.setHeader('Content-Length', Buffer.byteLength(html, 'utf8')); res.end(html); } function tick () { var now = new Date().toUTCString(); io.sockets.send(now); } setInterval(tick, 1000); app.listen(8080); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 349 Licensed to Roger Chen The HTTP server code always serves the 'index.html' file Get the UTC representation of the current time Send the time to all connected sockets Run the tick function once per second As you can see, Socket.IO minimizes the amount of extra code you need to add to the base HTTP server. It only took 2 lines of code involving the variableio (which is the variable for your Socket.IO server instance) to enable realtime messages between your server and clients. In this clock server example, you continually invoke the function once per second to notify all thetick() connected clients of the server's time. The server code first reads the file into memory, so we need toindex.html implement that now. Listing 12.2 shows the client-side of this application utilizing Socket.IO. Listing 12.2 index.html: Socket.IO client that displays the server's broadcasted time First connect to the Socket.IO server When a 'message' event is received, then the server has sent the time Update the "time" span element with the server time Current server time is: ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 350 Licensed to Roger Chen 1. 2. 3. 4. You are now ready to run the server. Fire it up with node clock-server.js and you'll see the text "info - started". This means that Socket.IO is set up and ready to receive connections, so open up your browser to the URL . With any luck you will be greeted by somethinghttp://localhost:8080/ similar to figure 12.1. The time will be updated every second from the message received by the server. Go ahead and open another browser at the same time to the same URL and you will see the values change together in sync. Figure 12.1 The clock server running in a Terminal with the browser connected to it So just like that, realtime cross-browser communication between the client and server become possible in just a few lines of code, thanks to Socket.IO. Now that you have an idea of the simple things possible with Socket.IO, let's take a look at another example of how server-sent events are beneficial to developers. NOTE Other kinds of messaging with Socket.IO Sending a message to all the connected sockets is only one way of interacting with the connected users that Socket.IO provides. There's also the ability to send messages to individual sockets, broadcast to all sockets except one, send volatile (optional) messages, and a lot more. Be sure to check out Socket.IO's documentation for more information.4 Footnote 4m Let's quickly take a look at the typical workflow for web page designers: Open the web page in multiple browsers. Look for styling on the page that needs adjusting. Make changes to one or more stylesheets. TRY IT OUT! 12.1.2 Using Socket.IO to trigger page and CSS reloads ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 351 Licensed to Roger Chen 4. 5. Manually reload the web browsers.all Go back to step #2. The biggest time waster in this list is definitely #4, where the designer needs to manually go into each web browser and click the "Refresh" button. This is especially time consuming when the designer needs to test different browsers on different computers, like a separate machine running Windows for testing Internet Explorer, and various mobile devices. But what if you could eliminate this "manual refresh" step completely? Imagine that when you saved the stylesheet in your text editor, that the web browsersall that had that page opened automatically reloaded the changes in the CSS sheet. This would be a huge time saver for devs and designers alike, and Node's and functions make it possible in just a few lines offs.watchFile code. Socket.IO matched with Node's function makes thisfs.watchFile() possible with just a few lines of code. We will use in thisfs.watchFile() example instead of the newer because we are assured this code work the same on all platforms, but we'll cover that in depth later. NOTE fs.watchFile() vs. Node.js provides two APIs for watching files: fs.watchFile()5 and fs.watchFile is rather expensive resource-wise, though more reliable and cross-platform. is highly optimized for each platform, but has behavioral differences on certain platforms. We will go over these functions in greater detail in section 12.3.2. Footnote 5m Footnote 6m In this example, we combine the Express framework with Socket.IO. Watch how they work together seamlessly in listing 12.3, just like the regular in the previous example. First let's take a look at the server code,http.Server it its entirety. Copy the file over and save it as , if you arewatch-server.js interested in running this example at the end: ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 352 Licensed to Roger Chen Listing 12.3 watch-server.js: Express and Socket.IO server that triggers "reload" and "stylesheet" events when files change var fs = require('fs'); var url = require('url'); var http = require('http'); var path = require('path'); var express = require('express'); var app = express(); var server = http.createServer(app); var io = require('').listen(server); var root = __dirname; app.use(function (req, res, next) { req.on('static', function () { var file = url.parse(req.url).pathname; var mode = 'stylesheet'; if (file[file.length - 1] == '/') { file += 'index.html'; mode = 'reload'; } createWatcher(file, mode); }); next(); }); app.use(express.static(root)); var watchers = {}; function createWatcher (file, event) { var absolute = path.join(root, file); if (watchers[absolute]) { return; } fs.watchFile(absolute, function (curr, prev) { if (curr.mtime !== prev.mtime) { io.sockets.emit(event, file); } }); watchers[absolute] = true; } server.listen(8080); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 353 Licensed to Roger Chen First create the Express app server Then you wrap the HTTP server to create the instance The app uses a little middleware to begin watching files that are returned by the static middleware The "static" event gets emitted by the static() middleware, so register the event before it runs This event handler determines the filename that was served and calls the createWatcher() function The server is set up as a basic static file server "watchers" is an Object that keeps a list of active files being watched Begin watching the file, this callback gets invoked for any change to the file Check to see if the 'mtime' (last modified time) changed before firing the Socket.IO event Mark that this file is now being watched So at this point you have a fully functional static file server, that is prepared to fire "reload" and "stylesheet" events using across the wire to the client using Socket.IO. Next take a look at the basic client-side code in listing 12.4. Save it as so that it gets served at the root path, when you fire up the serverindex.html next: Listing 12.4 watch-index.html: The client-side code reloads the CSS stylesheets when events are received from the server Socket.IO dynamically reloading CSS stylesheets

This is our Awesome Webpage!

If this file (index.html) is edited, then the server will send a message to the browser using Socket.IO telling it to refresh the page.

If either of the stylesheets (header.css or styles.css) are edited, then the server will send a message to the browser using Socket.IO telling it to dynamically reload the CSS, without refreshing the page.

TRYING IT OUT $ node watch-server.js ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 355 Licensed to Roger Chen demonstrates how the object acts like a bi-directional EventEmitter,socket which you can use to emit arbitrary events that Socket.IO will transfer across the wire for you. So now that you know this, it should be no surprise to hear that simply emits a "message" event.socket.send() As you know, HTTP was never originally intended for any kinds of realtime communication. But with advances in browser technologies like WebSockets, and modules like Socket.IO, this limitation has been changed, opening a big door for all kinds of new applications that were never before possible in the web browser. Back in Chapter 4 we said that using Socket.IO would be great for relaying upload progress events back to the browser for the user to see. You can apply the techniques used in the examples in this section to do this. Using a custom "progress" event would work well: Getting access to the instance that matches the browser uploading thesocket file is another task that will need to be figured out to make this progress relaying work, however it is outside the scope of this book. Do not fret though, there are resources on the internet that can aid in figuring that out.7 Footnote 7m Socket.IO is game changing. As mentioned, developers interested in "realtime" web applications hear about Socket.IO before even knowing about Node.js, a testament to how influential and important it is. It is constantly gaining traction in web gaming communities and being used for more and more creative games and applications then one could ever have thought possible. It is also a very popular pick for use in applications written in Node.js competitions like Node Knockout .8 What awesome will you write?thing Footnote 8m 12.1.3 Other ideas for uses of Socket.IO // ... updated from the example in 4.4.1 form.on('progress', function(bytesReceived, bytesExpected){ var percent = Math.floor(bytesReceived / bytesExpected * 100); // here is where you can relay the uploaded percentage using Socket.IO socket.emit('progress', { percent: percent }); }); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 356 Licensed to Roger Chen Transitioning from to the low-level networking APIs that Node provides out of the box may feel like a bit of a gear switch, and it is. But just like HTTP servers, Node is well suited for kind of I/O bound task, and it is the same storyany for TCP-based networking. Node is a very good platform for writing, for example, an email server, file server, proxy server, just to name a few. But it can also be used as a client for these kinds of services as well. Node provides a few tools to aid in writing high quality and performant I/O applications, which you will learn about in this section. Some networking protocols require reading values at the byte level like chars, ints, floats, etc., involving binary data. JavaScript falls short in this category by not including any native binary data type to work with. The closest you could get is crazy hacks with Strings. So Node picks up the slack by implementing its own data type, acting as a piece of fixed-length binary data, which makes itBuffer possible to access the low-level bytes needed to implement other protocols. In this section you will learn about: Working with Buffers and binary data Creating a TCP server Creating a TCP client "Buffers" are a special data type that Node provides for developers that act as a slab of raw binary data with a fixed length. They can be thought of as the equivalent of the C function , or the keyword in C++. Buffers aremalloc() 9 new used all over the place throughout Node's core APIs, and are in fact returned in 'data' events by all "Stream" classes by default. Footnote 9m Buffers are very fast and light objects. Node exposes the constructorBuffer globally, encouraging you to use it just like an extension of the regular JavaScript data types. From a programming point of view, you can think of them as similar to Arrays, except they are not resizable, and can only contain the numbers 0 through 255 as values. This makes them ideal for storing binary data of, well, anything really. Since buffers work with raw bytes, you can use them to implement any low-level protocol that you desire. 12.2 TCP/IP networking in-depth 12.2.1 Working with Buffers and binary data ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 357 Licensed to Roger Chen Say you wanted to store the number "121234869" into memory using a .Buffer By default Node assumes that you want to work with text-based data in Buffers, so when you pass the string to the constructor function, a"121234869" Buffer new buffer object will be allocated with the string value written to it. In this case, it would return a 9-byte buffer. This is because the string was written to the buffer using the default human-readable text-based encoding (UTF-8), where in this case, the string is represented with 1 byte per character. Node also includes helper functions for reading and writing binary (machine-readable) integer data. These are needed for implementing machine-protocols that send raw data types (like ints, floats, doubles, etc.) over the wire. Since you want to store a number value in this case, it's possible to be more efficient by utilizing the helper function to write the numberwriteInt32LE() "121234869" as a machine-readable binary integer (assuming a little-endian processor) into a 4-byte Buffer. There are other variations of the Buffer helper functions as well, to name a few: writeInt16LE() for smaller integer values. writeUInt32LE() for unsigned values. writeInt32BE() for big-endian values. There are lots more so be sure to check the Buffer API doc page if you're10 interested in them all. Footnote 10m In this code snippet, the number will be written using the writeInt32LE binary helper function. TEXT DATA VS. BINARY DATA var b = new Buffer("121234869"); console.log(b.length); 9 console.log(b); var b = new Buffer(4); b.writeInt32LE(121234869, 0); console.log(b.length); 4 ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 358 Licensed to Roger Chen By storing the value as a binary integer instead of a text string in memory, it was possible to save over 50% of the memory used, from 9 bytes down to 4. Take a look at the breakdown of these two buffers in figure 12.2. Figure 12.2 The difference between representing "121234869" as a text string vs. a little-endian binary integer at the byte-level In essence, this is the difference between human-readable (text) protocols, and machine-readable (binary) protocols. But regardless of what kind of protocol you are working with, Node's class will be able to take care of handling theBuffer proper representation. console.log(b); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 359 Licensed to Roger Chen NOTE Byte Endianness The term "endianness" refers to the order of the bytes within a11 multibyte sequence. When bytes are orderd "little-endian" then that means the most significant byte is the last byte (right-to-left). Conversely, "big-endian" order is when the first byte is the most significant byte (left-to-right). Node.js offers equivalent helper functions for both little-endian and big-endian data types. Footnote 11m Now it's time to actually put these Buffer objects to use by creating a TCP server and interacting with it. Node's core API sticks to being low-level, with only the bare essentials exposed for modules to build on top of. Node's module is a good example of this;http building on top of the module to implement the HTTP protocol. Othernet protocols, like SMTP for email or FTP for file transer, will need to be implemented on top of the module as well, since Node's core API does not implement anynet other higher level protocols. The module offers a raw TCP/IP socket interface for your applications to The API for creating a server is very similar to creating an HTTP server: you call and give it a callback function that will be invokednet.createServer() upon each connection. The main difference is that the callback function only takes one argument (usually named ) which is the object, as opposedsocket Socket to the and arguments when creating an HTTP Server.req res NOTE The classSocket The class is used by both the client and server aspects ofSocket the module in Node. It is a subclass that is both net Stream and (bi-directional). That is, it emits "data"readable writable events when input data has been read from the socket, and has and functions for sending output data.write() end() So first let's do something quick and silly, to show a "barebones" that waits for connections and then invokes our given callbacknet.Server 12.2.2 Creating a TCP server WRITING DATA ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 360 Licensed to Roger Chen function. In this case, the application login inside the callback function simply writes "Hello World!" to the socket and closes the connection cleanly. So fire up the server for some testing: Now if you were to try to connect to the server in a web browser, it would not work since this server does not speak HTTP, only raw TCP. So in order to connect to this server and see our message you need to connect with a proper TCP client, like :netcat(1) Great! Ok, now let's try using :telnet(1) telnet is usually meant to be run in an interactive mode, so it prints out its own stuff as well, but the "Hello World" message does get printed right before the connection gets closed. Just as expected. So writing data to the socket is easy, it's just calls and a final write() end() call. In case you didn't notice, this is exactly the same API as the HTTP objectres when writing a response to the HTTP client. var net = require('net'); net.createServer(function (socket) { socket.write('Hello World!\r\n'); socket.end(); }).listen(1337); console.log('listening on port 1337'); $ node server.js listening on port 1337 $ netcat localhost 1337 Hello World! $ telnet localhost 1337 Trying Connected to localhost. Escape character is '^]'. Hello World! Connection closed by foreign host. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 361 Licensed to Roger Chen It's common for servers to follow the "request -> response" paradigm, where the client connects and immediately sends a "request" of some sort. The server reads the request and processes a "response" of some sort to write back to the socket. This is exactly how the HTTP protocol works, as well as the majority of other networking protocols that are common in the wild, so it's important to know how to read data as well. Fortunately, if you remember how to read a request body from an HTTP req object then reading from a TCP socket should be a piece of cake. Complying with the readable interface, all you have to do is listen for "data" events whichStream contain the input data that was read from the socket: By default there is no encoding set on the , so the argument willsocket data be a instance. Usually, this is exactly how you want it (that's why it's theBuffer default), but when it's more convenient you can call the setEncoding() function to have the argument be the decoded Strings instead of You also listen for the "end" event to know when the client has closed their end of the socket, and won't be sending and more data. We can easily write a quick TCP client that looks up the versions string of the given SSH server, by simply waiting for the first "data" event. READING DATA socket.on('data', function (data) { console.log('got "data"', data); }); socket.on('end', function () { console.log('socket has ended'); }); var net = require('net'); var socket = net.connect({ host: process.argv[2], port: 22 }); socket.setEncoding('utf8'); socket.once('data', function (chunk) { console.log('SSH server version: %j', chunk.trim()); socket.end(); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 362 Licensed to Roger Chen Now you can try it out. Note that this over-simplified example assumes that the entire version string will come in one chunk. Most of the time this works just fine, but a proper program would buffer the input until a char was found. Let's check what SSH server "" uses: Using in conjunction with either the readable or writableStream#pipe()12 portions of a object is also a good idea. In fact, if you wanted to write aSocket simple TCP server that simply echoed everything that was sent to it back to the client, then you could do that with a single line of code in your callback function: Footnote 12m This example shows that it only takes one line of code to implement the IETF Echo Protocol , but more importantly it demonstrates that you can both 13 pipe() and the object. Though you would usually do so with moreto from socket meaningful Stream instances, like a file-system or gzip stream. Stream#pipe() is one of the few times in Node's asynchronous APIs where doing the right thing ends up taking less code, so it is recommended to use it when possible! Footnote 13m The last thing that should be said about TCP servers is anticipating clients that disconnect, but don't cleanly close the socket. In the case of , thisnetcat(1) would happen when you press Ctrl+C to kill the process, rather than pressing Ctrl+D to cleanly close the connection. To detect this situation you listen for the "close" event. }); $ node client.js SSH server version: "SSH-2.0-OpenSSH_5.5p1 Debian-6+squeeze1+github8" CONNECTING TWO STREAMS WITH SOCKET.PIPE() socket.pipe(socket); HANDLING UNCLEAN DISCONNECTIONS ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 363 Licensed to Roger Chen If you have cleanup to do after a socket disconnects then you should do it from the "close" event, not the "end" event since it will not fire when the connection was not closed cleanly. Let's take all these events and create a simple echo server that logs stuff to the terminal when the various events occur, show in listing 12.5: Listing 12.5 tcp-echo-server.js: A simple TCP server that echoes any data it receives back to the client Fire up the server and connect to it with / and play aroundnetcat telnet with it a bit. You should see the calls for the events beingconsole.log() printed to the server's stdout as you mash around on the keyboard in the client app. Now that you can build low-level TCP servers in Node, you're probably wondering how to write a client program in Node to interact with these servers. Let's do that now. socket.on('close', function () { console.log('client disconnected'); }); PUTTING IT ALL TOGETHER var net = require('net'); net.createServer(function (socket) { console.log('socket connected!'); socket.on('data', function (data) { console.log('"data" event', data); }); socket.on('end', function () { console.log('"end" event'); }); socket.on('close', function () { console.log('"close" event'); }); socket.on('error', function (e) { console.log('"error" event', e); }); socket.pipe(socket); }).listen(1337); "data" can happen multiple times "end" can only happen once per socket "close" can also only happen once per socket always remember to set the "error" handler to prevent uncaught exceptions ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 364 Licensed to Roger Chen Node isn't only about server software; creating client networking programs is equally as useful, and just as easy in Node. The heart of creating raw connections to TCP servers is the function. This function accepts an optionsnet.connect() argument with and values and returns a instance. The host port Socket returned from starts off disconnected from the server.socket net.connect() So usually you wait for the "connect" event before doing any work with the socket: Once the instance is connected to the server then it behaves identicallysocket to the instances that you get inside of a callback function.socket net.Server Let's demonstrate by writing a basic replica of the command,netcat(1) implemented in listing 12.6. Basically the program connects to the specified remote server and pipes stdin from the program to the socket, and pipes the socket's response to the program's stdout. Listing 12.6 netcat.js: A basic replica of the command implementednetcat(1) using Node.js 12.2.3 Creating a TCP client var net = require('net'); var socket = net.connect({ port: 1337, host: 'localhost' }); socket.on('connect', function () { // begin writing your "request" socket.write('HELO\r\n'); ... }); var net = require('net'); var host = process.argv[2]; var port = Number(process.argv[3]); var socket = net.connect(port, host); socket.on('connect', function () { process.stdin.pipe(socket); socket.pipe(process.stdout); process.stdin.resume(); }); socket.on('end', function () { ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 365 Licensed to Roger Chen parse the "host" and "port" from the command-line arguments create the "socket" instance, begin connecting to the server "connect" is emitted once the connection to the server is established pipe the process' stdin to the socket pipe the socket's data to the process' stdout we must call "resume()" on stdin to begin reading data when the "end" event happens, then pause stdin You will be able to connect to the TCP server examples you wrote before, or for you Star Wars fans, try invoking our netcat replica script with the following arguments for a special easter egg: Sit back and enjoy the special easter egg show (figure 12.3). You deserve a break: process.stdin.pause(); }); $ node netcat.js 23 ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 366 Licensed to Roger Chen Figure 12.3 Connecting to the ASCII Star Wars server with the netcat.js script So really that's all it takes to write low-level TCP servers and clients using Node.js. The module provides a simple, yet comprehensive API, and the net class follows both the readable and writable Stream interfaces as youSocket would expect. Essentially, the module is a showcase of the core fundamentalsnet of Node. Let's switch gears once again, and look at Node's core APIs that allow you to interact with the process' environment and query information about the runtime and operating system. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 367 Licensed to Roger Chen Often times you will find yourself in situations where you want to interact with the environment that Node is running in. Some examples of this could be checking environment variables to enable debug-mode logging, implementing a Linux joystick driver using the low-level functions to interact with (thefs /dev/js0 device file for a game joystick), or launching an external child process like tophp compile a legacy PHP script to serve to your HTTP clients. Doing any of these kinds of actions requires using some of the Node core APIs, and we will cover some of these modules throughout this section: The global object, which contains information about the current process like theprocess arguments given to it and the environment variables that are currently set. The module, which contains the high level ReadStream and WriteStream classesfs which you are familiar with by now, but also houses the low-level functions, which is what we will go over. The module, which contains both a low-level and high-level interface forchild_process spawning child processes, and well as a special way to spawn instances with anode 2-way message-passing channel. The object is one of those APIs that a large majority of programsprocess will interact with, so let's start with that. Every Node process has a single global object that every module sharesprocess access to. Various useful information about the process and the context it's running in can be found on this object. For example, the arguments that were invoked with Node to run the current script can be accessed with , or theprocess.argv enviroment variables can be get or set using the object. But theprocess.env most interesting feature of the object is that it's an process EventEmitter instance, which has very special events emitted on it like and "exit" ."uncaughtException" The object has a lot of bells and whistles under the hood, and weprocess will cover some of the APIs not mentioned in this section later in the chapter. Here though, you will learn about: Using to get and set environment variables.process.env Listening for special events emitted by , namely and process "exit" ."uncaughtException" Listening for signal events emitted by , like and .process "SIGUSR2" "SIGKILL" 12.3 Tools for interacting with the operating system environment 12.3.1 The Process global singleton ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 368 Licensed to Roger Chen Environment variables are a great way to alter the way your program or module will work. For example, you can use these variables as configuration for your server, specifying which port to listen on. Another example is that the operating system can set the variable to specify where your programs should outputTMPDIR temporary files that may be cleaned up later. NOTE Environment Variables? In case you're not already familiar with what exactly "environment variables" are, they are simply a set of key-value pairs that any14 given process can use to affect the way it will behave. For example, all operating systems use the environment variable as a list ofPATH file paths to search when looking up a program's location by name (i.e. being resolved to ).ls /bin/ls Footnote 14m Suppose you wanted to enable special debug-mode logging while developing or debugging your module, but not during regular usage, since that would be annoying for consumers of your module. A great way to do this is with environment variables. You could check what the variable is set to byDEBUG checking and seeing if it's set to a truthy value, likeprocess.env.DEBUG shown in listing 12.7. Listing 12.7 debug-mode.js: Conditionally defining a "debug" function depending on what the DEBUG environment variable is set to USING `PROCESS.ENV` TO GET AND SET ENVIRONMENT VARIABLES var debug; if (process.env.DEBUG) { debug = function (data) { console.error(data); }; } else { debug = function () {}; } debug('this is a debug call'); console.log('Hello World!'); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 369 Licensed to Roger Chen The first step is to set the "debug" function based on process.env.DEBUG When it is set, then the "debug" function will log the given argument to stderr When it's not set, then the "debug" function is empty and does nothing Your script calls the "debug" function in various places throughout your code This is a nice way to get diagnostic reporting to stderr when debugging a problem in your code Now if you try running this script regularly (without the environment variable set) then you will see that calls toprocess.env.DEBUG "debug" do nothing, since the empty function is being called. To test out "debug mode" you need to set the process.env.DEBUG environment variable. The simplest way to do this when launching a Node instance is to prepend the command with . When in debug mode, then calls to theDEBUG=1 "debug" function will be printed to the console as well as the regular output. The community module by TJ Holowaychuk encapsulates preciselydebug15 this functionality with some additional features. If you like the debugging technique presented here then you should definitely check it out! Footnote 15m Normally there are two special events that get emitted by the object:process "exit" which gets emitted right before the process exits. "uncaughtException" which gets emitted any time an unhandled Error is thrown. The "exit" event is essential for any application that needs to do something right before the program exits, like clean up an object or print a final message to the debug('this another debug call'); $ node debug-mode.js Hello World! $ DEBUG=1 node debug-mode.js this is a debug call Hello World! this is another debug call SPECIAL EVENTS EMITTED BY `PROCESS` ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 370 Licensed to Roger Chen console. One important thing to note is that the "exit" event gets fired after the event loop has already stopped, therefore you do not have the opportunity to start any asynchronous tasks during the "exit" event. The exit code is passed as the first argument, which is on a successful exit. Let's write a script that listens on the0 "exit" event to print an "Exiting..." message: The other special event emitted by is the "uncaughtException" event.process In the "perfect program" there will never be any uncaught exceptions. But in the real world we all know that this is impossible, and it's better to be safe than sorry. The only argument given to the "uncaughtException" event is the uncaught Error object. When there are no listeners for "uncaughtException" then any uncaught Errors will crash the process (this is the default behavior for most applications), but when there's at least one listener then it's up to that listener to decide what to do with the Error and Node will not exit automatically, though it is considered mandatory to do so in your own callback. The Node.js documentation explicitly warns that any usage of this event should contain a call within the callback,process.exit() otherwise you leave the application in an undefined state which is bad. Let's listen for "uncaughtException", then throw an uncaught Error to see it in action: Unix has the concept of "signals", which are a basic form of inter-process communication (IPC ). These signals are very primitive, allowing for only a fixed16 set of names to be used, and no arguments to be passed. Node has default behavior for a few signals, which we will go over now: Footnote 16m process.on('exit', function (code) { console.log('Exiting...'); }); process.on('uncaughtException', function (err) { console.error('got uncaught exception:', err.message); process.exit(1); }); throw new Error('an uncaught exception'); CATCHING SIGNALS SENT TO THE PROCESS ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 371 Licensed to Roger Chen SIGINT - Gets sent by your shell when you press Ctrl+C. Node's default behavior is to kill the process, but this can be overridden with a single listener for "SIGINT" on .process SIGUSR1 - When this signal is received, Node will enter its built-in debugger. SIGWINCH - Gets sent by your shell when the terminal is resized. Node resets and and emits a "resize" event whenprocess.stdout.rows process.stdout.columns this is received. So those are the three signals that Node handles for you by default, but you can also listen for any of these events and invoke a callback function by listening for the event name of the object.process Say you have your server that you are writing. However, when you press Ctrl+C to kill the server, it is an unclean shutdown, and any pending connections will be dropped. The solution to this would be to catch the "SIGINT" signal and stop the server from accepting connections, letting any existing connections complete before the process completes. This is done by simply listening for . The name of the event emitted is the sameprocess.on('SIGINT', ...) as the signal name. Now, when you press Ctrl+C on your keyboard then the "SIGINT" signal will be sent to the Node process from your shell, which will invoke the registered callback instead of killing the process. Since the default behavior of most applications is to exit the process, it is usually a good idea to do the same in your own "SIGINT" handler, after any necessary "shut down" actions happen. In this case, stopping a server from accepting connections will do the trick. This also works on Windows, despite its lack of proper signals, due to libuv handling the equivalent Windows actions and simulating artificial signals in node. You can apply this same technique to catch any of the Unix signals that get17 sent to your Node process. Unfortunately signals in general don't work on Windows, except for the few simulated supported signals like "SIGINT" mentioned before. Footnote 17m process.on('SIGINT', function () { console.log('Got Ctrl+C!'); server.close(); }); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 372 Licensed to Roger Chen The "fs" module provides all the different functions for interacting with the filesystem of the computer that node is running on. Most of the functions are 1-1 mappings to their C function counterparts, but there are also higher level abstractions like and , and the fs.readFile() fs.writeFile() and classes, which build on top of fs.ReadStream fs.WriteStream , / , and .open() read() write() close() Nearly all of the low-level functions are identical in function to their C versions of the functions. In fact, most of the Node documentation simply says to refer to the equivalent page explaning the matching C function. You can easilyman identify these low-level functions because they will always have a synchronous counterpart. For example, and are the low-levelfs.stat() fs.statSync() bindings to the C function.stat(2) NOTE Synchronous functions in Node.js As you already know, Node.js is all about asynchronous functions and never blocking the event loop. So why bother including synchronous versions of these filesystem functions? The answer to that is because of Node's own module system, where the function is synchronous. Since that is implementedrequire() using the module functions, then synchronous counterpartsfs were necessary. However, the golden rule is that they should only be used during startup, or when your module is initially loaded, and after that.never Let's take a look at some examples of interacting with the filesystem. A seemingly simple, yet very common task, when interacting with the filesystem is the act of moving a file from one directory to another. On Unix platforms you use the command for this, on Windows it's the command. So doing the samemv move thing in Node should be similarly simple, you are probably thinking. Well if you browse through the module in the REPL or in thefs documentation , you'll notice that there's no function. However18 fs.move() there is an function, which is the same thing if you think about it.fs.rename() Perfect! Footnote 18m 12.3.2 Using the Filesystem module MOVING A FILE ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 373 Licensed to Roger Chen Not so fast there cowboy. maps directly to the Cfs.rename() rename(2) function. One gotcha with this function is that it doesn't work across physical devices (like two hard drives). So this would not work properly, and would throw an error:EXDEV So what do you do now? Well you can still create new files on , and readD:\ files from , so copying the file over will work. With this knowledge, you canC:\ create an optimized function. By optimized we mean that it 1) calls themove() very fast when possible and 2) works across devices by copyingfs.rename() the file over in that case using and . Onefs.ReadStream fs.WriteStream such implemententaion is shown in listing 12.8. Listing 12.8 copy.js: An optimized "move()" function that renames when possible (fast), and falls back to copying (slow) fs.rename('C:\\hello.txt', 'D:\\hello.txt', function (err) { // err.code === 'EXDEV' }); var fs = require('fs'); module.exports = function move (oldPath, newPath, callback) { fs.rename(oldPath, newPath, function (err) { if (err) { if (err.code === 'EXDEV') { copy(); } else { callback(err); } return; } callback(); }); function copy () { var readStream = fs.createReadStream(oldPath); var writeStream = fs.createWriteStream(newPath); readStream.on('error', callback); writeStream.on('error', callback); readStream.on('close', function () { fs.unlink(oldPath, callback); }); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 374 Licensed to Roger Chen First step is to try calling fs.rename() and hope it works If we got an EXDEV error, then fall back to the "copy" technique Any other kind of error we fail and report back to the caller If fs.rename() worked successfully, then we are done. This is the optimal case. The "copy" technique reads the original file and pipes it out to the destination path Once the copy is done, then "unlink" (delete) the original file You can test out this module directly in the node REPL if you like: Note that this only works with files, and not directories. To make directories work you would have to first check if the given path is a directory, and if it is then call and as necessary, but you can implementfs.readdir() fs.mkdir() that at some point in the future on your own. NOTE fs module error codes The module returns standard Unix names for the file systemfs 19 error codes, thus some familiarity with those names is required. These names get normalized, even on Windows, by libuv so that your application only needs to check for one error code at a time. According to the gnu docs page, an EXDEV error happens when "an attempt to make an improper link across file systems was detected." Footnote 19m readStream.pipe(writeStream); } } $ node > var move = require('./copy') > move('copy.js', 'copy.js.bak', function (err) { if (err) throw err }) ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 375 Licensed to Roger Chen fs.watchFile() has been around since the early days. It is expensive on some platforms because it uses polling to see if the file has changed. That is, it sstat() the file, waits a short period of time, then s again in a continuous loop,stat() invoking the watcher function any time the file has changed. So say you are writing a module that logs changes from the log file.systemd To do this you would want to have a callback function be invoked any time the global "system.log" file was modified. The and variables are the previous and current objects,curr prev fs.Stat which should have a different timestamp for one of the file times attached. In this example, the "mtime" values are being compared, as you only want to be notified when the file is modified, and not when it is simply accessed. was introduced in the Node v0.6 release. As we mentioned earlier, it is actually more optimized because, underneath the hood, it uses the platform's native file change notification API for watching files. Because of this, the function is also capable of watching directories, and have its callback function be invoked for a change to any file in the directory. In practice less reliable though, because of various differences between the platforms' underlying file watching mechanisms. For example, the filename parameter does not get reported on OS X when watching a directory, and it is up to Apple to change that in a future release of OS X. So the module, like all of Node's core API as you have learned, sticks to beingfs low-level. This means that there is plenty of room to innovate and create awesome abstractions on top of it. Node's active collection of modules is growing on npm every day, and as you could guess there are some quality ones out there that extend the module to go even further.fs "fstream" by Isaac Schlueter, is one of the core pieces of npm itself. This20 WATCHING A DIRECTORY OR FILE FOR CHANGES var fs = require('fs'); fs.watchFile('/var/log/system.log', function (curr, prev) { if (curr.mtime.getTime() !== prev.mtime.getTime()) { console.log('"system.log" has been modified'); } }); USING COMMUNITY MODULES: "FSTREAM" AND "FILED" ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 376 Licensed to Roger Chen module is interesting because it began life as a part of npm, and then got extracted, because its general-purpose functionality is useful to many kinds of command-line applications and sysadmin scripts. One of the awesome features that sets apart is its seamless handling of permissions and symbolic links whenfstream dealing with directories, which are all automatically transferred over by default. Footnote 20m Using , the equivalent of fstream cp -rp sourceDir destDir (copying a directory and its contents recursively, and transfering over ownership and permissions) involves simply piping a Reader instance to a Writer instance. In the example here, we're also utilizing the "filter" feature of tofstream conditionally exclude files based on a callback function. "filed" by Mikeal Rogers, is another influential module, mostly because it is21 written by the same author as the highly popular module, . Theserequest modules made popular a new kind of flow control over Stream instances: listening for the "pipe" event, and acting differently based on what is being piped to it (or what it is being piped to). Footnote 21m To demonstrate the power of this, take a look a how turns a regularfiled HTTP server into a full-featured static file server with just one line of code! This actually takes care of sending "Content-Length", sending the proper caching headers. In the case where the browser already has the file cached, then fstream .Reader("path/to/dir") .pipe(fstream.Writer({ path: "path/to/other/dir", filter: isValid ) // checks the file that is about to be written and // returns whether or not it should be copied over function isValid () { // ignore temp files from text editors like TextMate return this.path[this.path.length - 1] !== '~'; } http.createServer(function (req, res) { req.pipe(filed('path/to/static/files')).pipe(res); }); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 377 Licensed to Roger Chen will actually respond to the HTTP request with a 304 response, skippingfiled the whole opening and reading the file from the disk process. These are the kinds of optimizations that acting on the "pipe" event make possible, because the filed instance has access to both the and objects of the HTTP request.req res We have only demonstrated two examples of good community modules that extend the base module to do awesome things or expose beautiful APIs. The fs command is a good way to find published modules for a given task.npm search Say you wanted to find another module that simplifies copying files from one destination to another. Executing could bring up somenpm search copy useful results in this case. When you find a published module that looks interesting, then you can execute to get informationnpm info module-name about the module like its description, homepage and published versions. Just remember that for any given task, it's very likely that someone has attempted to solve the problem in the form of an npm module, so always check there before writing something from scratch. Node provides the module to create child subprocesses fromchild_process within a Node server or script. There are two APIs for this: a high-level one, called , and a low-level one, called . Either one may be appropriate forexec() spawn() you to use depending on your needs. There is also a special way to create child processes of Node itself, with a special Inter-Process Communication (IPC) channel built-in called . All of these functions are meant for differentfork() use-cases which we will go over now. cp.exec() - the high level API for spawning commands and buffering the result into a callback cp.spawn() - the low level API for spawning single commands into a ChildProcess object cp.fork() - the special way to spawn additional Node processes with a built-in IPC channel 12.3.3 Spawning external processes ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 378 Licensed to Roger Chen NOTE Pros and cons to child processes There are great benefits to using child processes but there are downsides as well. The obvious one is that the program being executed needs to actually be installed on the user's machine, making it a dependency of your application. The alternative would be to do whatever the child process does in JavaScript itself. A good example of this is , which once upon a time used thenpm system command when extracting node packages. But eventar though is available on every Unix computer out there, theretar were many conflicts relating to incompatible versions of tar itself, and on top of that it's very rare for a Windows computer to have installed. These factors, as well as the importance for npm to work everywhere, lead to node-tar being written entirely in JS, not22 using any child processes. Conversely, you would never want to re-implement if you were serving PHP scripts, so that's a casephp where child processes are good. Footnote 22m The high level API, , is useful for when you want to invoke acp.exec() command, and only care about the final result, and don't care about accessing the data from child's stdio streams as they come. Another benefit is that you can enter full sequences of commands, including multiple processes being piped to one another. One good use-case for the API is when you are accepting userexec() commands to execute. Say you are writing an IRC bot, and would like to execute commands when any user says something beginning with ".". So if a user typed ".ls" as their IRC message, then the bot would execute and print the output backls to the IRC room. Be sure to set the option, so that any neverendingtimeout processes get automatically killed after a certain period of time, like shown in listing 12.9. Listing 12.9 room.js: Using cp.exec() to spawn user-given commands from an IRC room bot BUFFERING COMMAND RESULTS USING CP.EXEC() var cp = require('child_process'); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 379 Licensed to Roger Chen "room" is an object representing a connection to an IRC room (from some theoretical irc module) The "message" event gets emitted for each IRC message sent to the room This script checks the message contents and checks if it begins with "." Spawn a child process and have node buffer the result into a callback for us, timing out after 15 seconds There are some good modules already in the npm registry implementing the IRC protocol, so if you would like to write an IRC bot like in this example for real, then you should definitely use one of the existing modules (both or irc irc-js in the npm registry are popular). For times when you need to buffer a command's ouput, but would like node to automatically escape the arguments for you, there's the function.execFile() This function takes four arguments, rather than three, and you pass the executable to run, along with an array of arguments to invoke the executable with. This is useful when you have to incrementally build up the arguments that the child process is going to use: room.on('message', function (user, message) { if (message[0] === '.') { var command = message.substring(1); cp.exec(command, { timeout: 15000 }, function (err, stdout, stderr) { if (err) { room.say( 'Error executing command "' + command + '": ' + err.message ); room.say(stderr); } else { room.say('Command completed: ' + command); room.say(stdout); } } ); } }); cp.execFile('ls', [ '-l', process.cwd() ], function (err, stdout, stderr) { if (err) throw err; console.error(stdout); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 380 Licensed to Roger Chen The low level API for spawning child processes in Node is . Thiscp.spawn() function differs from because it returns a objectcp.exec() ChildProcess which you interact with. Rather than giving it a single callback function when the process completes, lets you interact with each stdio stream of thecp.spawn() child process as individual instances, which include all the usual benefitsStream of Streams in Node. The most basic use of looks like:cp.spawn() The first argument is the program to execute. This can be a single program name, which will be looked up in the current PATH, or an absolute path to a program to invoke. The second argument is an Array of string arguments to invoke the process with. In the default case, a object contains 3 built-in ChildProcess instances which your script is meant to interact with:Stream child.stdin is the Stream that represents the child's .writable stdin child.stdout is the Stream that represents the child's .readable stdout child.stderr is the Stream that represents the child's .readable stderr You can do whatever you want with these streams, like pipe them to a file or socket or some other kind of writable stream. You can even just completely ignore them if you'd like. The other interesting event that happens on objects is theChildProcess "exit" event, which is fired when the process has exited and the associated stream objects have all ended. One good example module that abstracts the use of into helpfulcp.spawn() }); SPAWNING COMMANDS WITH A STREAM INTERFACE USING CP.SPAWN() var child = cp.spawn('ls', [ '-l' ]); // stdout is a regular Stream instance, which emits 'data', // 'end', etc. child.stdout.pipe(fs.createWriteStream('ls-result.txt')); child.on('exit', function (code, signal) { // emitted when the child process exits }); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 381 Licensed to Roger Chen functionality is "node-cgi" , which allows you to reuse legacy CGI scripts in your23 Node HTTP servers. Older style websites were written using the "Common Gateway Interface", which was really just a standard for responding to HTTP requests by invoking arbitrary "CGI scripts" as child processes of an HTTP server with special environment variables describing the request. For example, you can write a CGI script that uses as the CGI interface:sh Footnote 23m If you were to name that file "hello.cgi" (don't forget to chmod +x to make it executable!), then you could easily invoke it as thehello.cgi response logic for HTTP requests in your HTTP server with a single line of code. With this server setup, when an HTTP request hits the server, then node-cgi will handle the request by: Spawning the "hello.cgi" script as a new child process using .cp.spawn() Passing the new process contextual information about the current HTTP request using a custom set of environment variables. The "hello.cgi" script uses one of the CGI-specific environment variables, , which contains the query-string portion of the request URL,QUERY_STRING which the script uses in the response which gets written to the script's . Ifstdout you were to fire up this example server and send an HTTP request to it using curl , you will see something like: #!/bin/sh echo "Status: 200" echo "Content-Type: text/plain" echo echo "Hello $QUERY_STRING" var http = require('http'); var cgi = require('cgi'); var server = http.createServer( cgi('hello.cgi') ); server.listen(3000); $ curl http://localhost:3000/?nathan Hello nathan ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 382 Licensed to Roger Chen There are a lot of very good use cases for child processes in Node. node-cgi is just one example of them, but you will find that while getting your server or application to do what it needs to do, you will inevitably have to utilize them at some point. The last API offered by the module is a specialized way ofchild_process spawning additional Node processes, but with a special IPC channel built in. Since you're always spawning Node itself, the first argument passed to is acp.fork() path to a Node.js module to execute. Like , this function returns a object. Howevercp.spawn() ChildProcess the major difference is the API added by the IPC channel. So, the child process now has a function, and the script being invoked by child.send(message) can listen for events.fork() process.on('message') So say you wanted to write a Node HTTP server that calculated the fibonacci sequence. You may try naively writing the server all in one shot, like shown in listing 12.10: Listing 12.10 fibonacci-naive.js: A non-optimal implementaion of a fibonacci HTTP server in Node.js Fibonacci number calculation function This basic HTTP server simply calculates the requested fibonacci number DISTRIBUTING THE WORKLOAD USING CP.FORK() var http = require('http'); function fib (n) { if (n < 2) { return 1; } else { return fib(n - 2) + fib(n - 1); } } var server = http.createServer(function (req, res) { var num = parseInt(req.url.substring(1), 10); res.writeHead(200); res.end(fib(num) + "\n"); }); server.listen(8000); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 383 Licensed to Roger Chen If you fire up the server with and send annode fibonacci-naive.js HTTP request to then it will work as expected, buthttp://localhost:8000 calculating the fibonacci sequence for a given number is an expensive, CPU-bound computation, and while your Node server's single thread is grinding away at calculating the result there, no additional HTTP requests can be served during this time. Additionally, you're only utilizing one CPU core here, where you likely have more than that which are just sitting there doing nothing. This is bad. cp.fork() offers a clean interface for us to do this in Node. The trick is to fork Node processes during each HTTP request and have the child process do the expensive calculation, and report back to the server using fork's IPC channel. Doing this will take 2 files: "fibonacci-server.js" will be the server. "fibonacci-calc.js" does the calculation. First the server: So now the server is using to have the fibonacci calculation logiccp.fork() be done in a separate Node process, which will report back to the parent process using as shown below in the "fibonacci-calc.js" script:process.send() var http = require('http'); var cp = require('child_process'); var server = http.createServer(function(req, res) { var child = cp.fork(__filename, [ req.url.substring(1) ]); child.on('message', function(m) { res.end(m.result + '\n'); }); }); server.listen(8000); function fib(n) { if (n < 2) { return 1; } else { return fib(n - 2) + fib(n - 1); } } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 384 Licensed to Roger Chen This time you can start the server with and,node fibonacci-server.js again send an HTTP request to . This is just onehttp://localhost:8000 great example of how dividing up the various components that make your application into multiple processes can be a great benefit to you, and cp.fork() provides the perfect communication interface with and child.send() for the parent, and and child.on('message') process.send() for the child. Use them!process.on('message') Another very common task fulfilled by Node scripts is to build command-line tools. By now you should be familiar with the largest command-line tool written in 100% Node: "node package manager", a.k.a . As a package manager, it does anpm lot of filesystem operations and spawning of child processes, and all of this is done using Node and its asynchronous APIs. npm takes advantage of this by installing packages in parallel, rather than serially, making the overall process faster. So if a command-line tool complicated can be written in Node, then anything can.that Most command-line programs have common process-related needs, like parsing command-line arguments used, and reading from stdin and writing to stdout and stderr. In this section, you will learn about the common requirements for writing a full command-line program, including: Parsing the command-line arguments Working with the stdin and stdout Streams Adding pretty colors to the output using ansi.js To get started on building awesome command-line programs, you need to be able to read the arguments the user invoked your program with. We will take a look at that first. var input = parseInt(process.argv[2], 10); process.send({ result: fib(input) }); 12.4 Developing command-line tools ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 385 Licensed to Roger Chen Parsing arguments is a very easy and straightforward process. Node provides you with the property, which is an Array of Strings which are theprocess.argv arguments that were used when Node was invoked. The first entry of the Array is the "node" executable, while the second entry is the name of your script. Now parsing and acting on these arguments simply requires iterating through the Array entries and inspecting each argument. To demonstrate, write a quick script called that simply prints out theargs.js result of . Since 90% of the time you don't care about the first 2process.argv entries, you can them off before processing.slice() Now when you invoke this script standalone, you will get an empty Array, since no additional arguments were passed in: But when you pass along "hello" and "world" as arguments, then the Array contains string values as you would expect: And as with any terminal application, you can use quotes for arguments that have spaces in them to combine them into a single argument. This is not a feature of Node, but rather the shell that your are using (likely on a Unix platform orbash on Windows).cmd.exe By Unix convention, every command-line program should respond to the -h and flags by printing out usage instructions and then exiting. Listing--help 12.4.1 Parsing command-line arguments var args = process.argv.slice(2); console.log(args); $ node args.js [] $ node args.js hello world [ 'hello', 'world' ] $ node args.js "tobi is a ferret" [ 'tobi is a ferret' ] ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 386 Licensed to Roger Chen 12.11 shows an example of using to iterate on theArray#forEach() arguments and parse them in the callback, printing out the usage instructions when the expecting switch is encountered. Listing 12.11 args-parse.js: Parsing using Array#forEach() and aprocess.argv switch block. Start off by slicing off the first two entries, which you're not interested in Iterate through the arguments, looking for -h or --help You can add additional flags/switches here that your application responds to When "help" is requested, just print out a helpful message then quit You can easily extend that block to parse additional switches.switch Community modules like , , , and commander.js nopt optimist nomnom (just to name a few) all attempt to solve this problem in their own way, so don't feel like using a block is the only way to parse the arguments. In reality,switch like so many things in programming, there is no single correct way to do it. Another task that every command line program will need to deal with is reading input from , and writing structured data to . Let's take a look atstdin stdout how this is done in Node next. var args = process.argv.slice(2); args.forEach(function (arg) { switch (arg) { case '-h': case '--help': printHelp(); break; } }); function printHelp () { console.log(' usage:'); console.log(' $ AwesomeProgram '); console.log(' example:'); console.log(' $ AwesomeProgram --make-awesome not-yet.awesome'); process.exit(0); } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 387 Licensed to Roger Chen A common Unix idiom is to have programs be small, self-contained, and focused on a single task that they are really good at. This is usually facilitated by using pipes, feeding the results of one process to the next, repeated until the end of the command chain. For example, using standard Unix commands to, say, retrieve the list of unique authors from any given git repository, you could combine the git , and commands, like this:log sort uniq These commands run in parallel, feeding the output of the first process to the next, continuing on until the end. To adhere to this "piping" idiom, Node provides two objects for your command-line program to work with:Stream process.stdin - a that is used to read input dataReadStream process.stdout - a to write output data to.WriteStream These objects act like the familiar Stream interfaces that you've already learned about. You've already been using the writable stream implicitlyprocess.stdout every time you have called . Internally, the console.log() console.log() function calls after formatting the inputprocess.stdout.write() arguments. But the functions are more for debugging and inspectingconsole objects, for the times when you need to write structured data to stdout, you can call directly. Say your program connects to an HTTPprocess.stdout.write() url and writes the response to stdout. Utilizing works well, asStream#pipe() shown here: 12.4.2 Working with stdin and stdout $ git log --format='%aN' | sort | uniq Mike Cantelon Nathan Rajlich TJ Holowaychuk WRITING OUTPUT DATA WITH PROCESS.STDOUT var http = require('http'); var url = require('url'); var target = url.parse(process.argv[2]); var req = http.get(target, function (res) { res.pipe(process.stdout); }); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 388 Licensed to Roger Chen And voilà! An absolutely minimal replica in only 7 lines of code. Not toocurl bad, huh? Next up let's cover .process.stdin You must initially call to signify that your scriptprocess.stdin.resume() is interested in data from stdin. After that it acts like any other readable stream, by emitting "data" events as data is received from the output of another process, or as the user enters keystrokes into the terminal window. Listing 12.12 shows a quick command-line program that prompts the user for their age before deciding to continue executing. Listing 12.12 stdin.js: An age-restricted program that prompts the user for their age, reading from stdin This example program is age-restricted READING INPUT DATA WITH PROCESS.STDIN var requiredAge = 18; process.stdout.write('Please enter your age: '); process.stdin.setEncoding('utf8'); process.stdin.on('data', function (data) { var age = parseInt(data, 10); if (isNaN(age)) { console.log('%s is not a valid number!', data); } else if (age < requiredAge) { console.log('You must be at least %d to enter, ' + 'come back in %d years', requiredAge, requiredAge - age); } else { enterTheSecretDungeon(); } process.stdin.pause(); }); process.stdin.resume(); function enterTheSecretDungeon () { console.log('Welcome to The Program :)'); } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 389 Licensed to Roger Chen First write a basic question for the user to answer Setup stdin to emit utf8 strings instead of Buffers Parse the data into a Number If the user didn't give a valid number, then print a message saying so If the user's given age is less than 18, then print a message saying to come back in a few years If those past 2 conditions are ok, then the user is old enough and you can continue executing your program This program only waits for a single 'data' event before closing stdin process.stdin begins in a paused state, so call resume() to start reading There is also a writable stream in every Node process, whichprocess.stderr acts identical to the stream, except it writes to process.stdout stderr instead. Since is usually reserved for debugging, and not so much forstderr sending structured data and piping, you usually end up simply using instead of accessing directly.console.error() process.stderr So now that you are familiar with the built-in stdio Streams in Node, which is crucial knowledge for any command-line program, let's move on to something a bit more colorful (pun intended). Lots of command-line tools use colored text to make things easier to distingush on the screen. Node itself does this in its REPL, as does npm, for its various logging levels. It's a nice bonus feature that any command-line program can easily benefit from, and luckily adding colored output to your programs is rather easy, especially with the support of community modules. Colors on the terminal happen through something called ANSI escape codes (the name comes from the American National Standards Institute). These "escape codes" are simple text sequences written to the stdout that have special meaning to the terminal, signifying to change the text color, change the position of the cursor, make a beep sound, and more interactions. Let's start simple. To print the word "hello" in the color green in your script, a single call is all itconsole.log() takes: If you look closely, you can see the word "hello" in the middle of the string DIAGNOSTIC LOGGING WITH PROCESS.STDERR 12.4.3 Adding colored output CREATING AND WRITING ANSI ESCAPE CODES console.log('\033[32mhello\033[39m'); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 390 Licensed to Roger Chen with some weird looking characters on either side. This may look confusing at first but it's rather simple really. Figure 12.4 breaks up the green "hello" string into its three distinct pieces: Figure 12.4 "hello" in green text using ANSI escape codes Now, there are a lot of escape codes that terminals recognize and most developers have better things to do with their time than memorize them all. Thankfully, the Node community comes in to the rescue once again with multiple modules to pick from, like , , and to name acolors.js cli-color ansi.js few, which all try to make using colors in your programs easy and fun. NOTE ANSI escape codes on Windows Technically, Windows and its command prompt ( ) don'tcmd.exe support ANSI escape codes. Fortunately for us, Node interprets the escape codes on Windows when your scripts write them to stdout and then calls the appropriate Windows functions that would do the equivalent action or color. This is just an interesting tidbit, and not something you'll have to think about while writing your Node application. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 391 Licensed to Roger Chen Let's take a look at ( ). This module is niceansi.js24 npm install ansi because it's a very thin layer on top of the raw ANSI codes, which gives you greater flexibility compared to the other color modules, which only work with a single string at a time. In , you set the "modes" (like "bold") of theansi.js Stream, which are persistent until cleared by one of the calls. And as anreset() added bonus, is the first module to support 256 color terminals, and hasansi.js capabilities to convert CSS color codes (such as ) into ANSI color#FF0000 codes. Footnote 24m ansi.js works with the concept of a "cursor", which is really just a wrapper around a writable Stream instance with lots of convenience functions to write ANSI codes to the Stream, all of which support chaining. To print the word "hello" in green text again using syntax, you would write:ansi.js You can see here that to use you first have to create a ansi.js cursor instance from a writable Stream. Since you're interested in coloring your program's output, you pass as the writable stream that the willprocess.stdout cursor use. Once you have the you can invoke any of the methods it provides tocursor alter the way that the text output will be rendered to the terminal. In this case the result is equivalent to the call from before, we're calling:console.log() to set the foreground color to green cursor.write('Hello') writes the text "Hello" to the terminal in green cursor.fg.reset() resets the foreground color back to the default cursor.write('\n') to finish up with a newline FORMATTING FOREGROUND COLORS USING ANSI.JS var ansi = require('ansi'); var cursor = ansi(process.stdout); cursor .write('Hello') .fg.reset() .write('\n'); ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 392 Licensed to Roger Chen Background colors are also supported. To set the background color instead of the foreground color with you simply replace the portion of the call withansi.js fg . So for example, to set a red background color you would call bg . Let's wrap up with a quick program that prints this book' title all colorful to the terminal like shown in figure 12.5. Figure 12.5 The result of the "ansi-title.js" script printing out the name of this book and the authors in different colors The code to output fancy colors like this is verbose, but very straightforward, since each function call maps 1 to 1 to the corresponding escape code being written to the stream. The example program, shown in listing 12.13, is really just the 2 lines of initialization at the top followed by one really long chain of function calls that end up writing color codes and strings to .process.stdout Listing 12.13 ansi-title.js: A simple program that prints this book's title and authors in pretty colors FORMATTING BACKGROUND COLORS USING ANSI.JS var ansi = require('ansi'); var cursor = ansi(process.stdout); cursor .reset() .write(' ') .bold() ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 393 Licensed to Roger Chen Color codes are only one of the key features of . We haven't evenansi.js touched on any of the cursor positioning codes, or how to make a beep sound, or hiding and showing the cursor, but you can consult the documentationansi.js and examples to see how that works. Node is primarily designed for general I/O related tasks, and HTTP server are only the most common type of task that Node is used to fulfill. However, as you've learned throughout this chapter, Node is well suited for a large variety of different tasks, such as a command-line interface to your application server, a client program that connects to the ASCII Star Wars server, a program that fetches and displays statistics from the stock market servers... The possibilites are really endless, and only limited by your imagination. Take a look at or for a couplenpm node-gyp complicated examples of command line programs being written using Node. They're a couple great examples to learn from. In this past chapter we talked about a couple community modules that could aid in development of your next application. In this next chapter we're gonna focus on how you can find these awesome modules throughout the Node community, and how you can contribute the modules you've developed back to the community for feedback and improvements. The social interaction is the exciting stuff! .underline() .bg.white() .write('Node.js in Action') .fg.reset() .bg.reset() .resetUnderline() .resetBold() .write(' \n') .write(' by:\n') .fg.cyan() .write(' Mike Cantelon\n') .fg.magenta() .write(' TJ Holowaychuk\n') .fg.yellow() .write(' Nathan Rajlich\n') .reset() 12.5 Summary ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 394 Licensed to Roger Chen 13 In this chapter Finding online help with Node Collaborating on Node development using GitHub Publishing your work using the Node Package Manager To get the most out of Node development, it pays to know where to go for help and how to share your contributions with the rest of the community. If you’re already familiar with how to work in Node, you may want to skim this chapter; otherwise, read on. As with most open source communities, the development of Node and related projects happens via online collaboration. Many developers work together to submit and review code, document projects, and report bugs. When developers are ready to release a new version of Node it’s published on the official Node website. When a release-worthy third-party module has been created, it can be published to the npm repository to make it easy for others to install. Online resources provide the support you need to work with Node and related projects. Figure 13.1 shows how you can use online resources for Node-related development, distribution, and support: The Node ecosystem ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 395 Licensed to Roger Chen Figure 13.1 Node-related projects are created online collaboratively, often via the GitHub website. They’re then published to npm with documentation and support provided via online resources. As you’ll likely need support before you need to collaborate, let’s first look at where to go online to get help when you need it. As the Node world is an ever-changing one, you’ll find the most up-to-date references online. At your disposal are numerous websites, online discussion groups, and chat rooms where you can find the information you need. Table 13.1 lists a number of Node-related online references and resources. The most useful websites for referencing the Node APIs and learning about available third-party modules are and, respectively. 13.1 Online resources for Node developers 13.1.1 Node and module references ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 396 Licensed to Roger Chen When you attempt to implement something using Node, or any of its built-in modules, the Node homepage is an invaluable resource. The site (shown in figure 13.2) documents the entirety of the Node framework, including each of its APIs. You’ll always find documentation for the most recent version of Node on the site. The official blog also documents the latest Node advances and shares important community news. You’ll even find a job board. Table 13.1 Useful node.js references.m Resource URL Node.js homepage Node.js up-to-date core documentation Node.js blog Node.js job board Node Package Manager homepage ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 397 Licensed to Roger Chen Figure 13.2 In addition to providing links to useful Node-related resources, offers authoritative API documentation for every released version of Node. When you’re shopping for third-party functionality, the npm repository search page is the place to go. It lets you use keywords to search through the thousands of modules available in npm. If you find a module that you’d like to check out, click on the module’s name to bring up its detail page, where you’ll find links to the module’s project homepage, if any, and useful information such as what other npm packages depend on the module, the module’s dependencies, which versions of Node the module is compatible with, and its license information. If you have a question about how to use Node or other third-party modules, these websites may not always help. Luckily, we can point you to some great places to ask for help online. ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 398 Licensed to Roger Chen Google Groups have been set up for Node and some other popular third-party modules, including npm, Express, node-mongodb-native, and Mongoose. Google Groups are useful for tough or in-depth questions. For example, if you were having trouble figuring out how to delete MongoDB documents using the node-mongodb-native module you could go to the node-mongodb-native Google Group and search it to see if anyone else had the same problem. If no one has1 dealt with your problem, the next step would be to join the Google Group and post your question. You can also write lengthy Google Groups posts, helpful for complicated questions as you have the opportunity to explain your issue thoroughly. Footnote 1 m!forum/node-mongodb-native But no central list exists that includes all Node-related Google Groups. In order to find them they’ll either be mentioned in project documentation or you’ll have to search the web. You could, for example, search “nameofsomemodule node.js google group” on Google to check if a Google Group existed for a particular third-party module. The drawback to using Google Groups is that often you have to wait anywhere from hours to days to get a response, depending on the parameters of the Google Group. For simple questions where you need a quick reply, you should consider entering an internet chat room, where you can often get a quick answer. Internet Relay Chat (IRC) was created way back in 1988, and while some think it archaic, it’s still alive and active—and is the best online means, in fact, to get answers to quick questions about open source software. IRC rooms are called “channels” and exist for Node and various third-party modules. You won’t find a list of Node-related IRC channels anywhere but third-party modules that have a corresponding IRC will sometimes mention them in their documentation. To get your question answered on IRC, connect to an IRC network , change to2 the appropriate channel, and send your question to the channel. Out of respect to the folks in the channel, it’s good to do some research beforehand to make sure your question can’t be solved with a quick web search. Footnote 2m If you’re new to IRC, the easiest way to get connected is using a web-based 13.1.2 Google Groups 13.1.3 IRC ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 399 Licensed to Roger Chen client. Freenode, the IRC network on which most Node-related IRC channel exists, has a web client available at To join a chatroom, enter the appropriate channel into the connection form. You don’t need to register and you can enter any nickname (if someone already uses the one you choose, the character ‘_’ will be appended to the end of your nickname to differentiate you). Once you click “Connect” you’ll end up in a chat room with a list in the right sidebar of any other users in the room. If a project’s development occurs on GitHub, another place to look for solutions to problems with Node modules is the project’s GitHub issue queue. To get to the issue queue navigate to the project’s main GitHub page and click the “Issues” tab. You can use the search form to look for issues related to your problem. An example issue queue is shown in figure 13.3: Figure 13.3 For projects hosted on GitHub, the issue queue can be helpful if you believe you've identified a problem in the project's code. If you’re unable to find an issue that addresses your problem, and you think it may be due to a bug in the project's code, then you can click the “New Issue” 13.1.4 GitHub issues ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 400 Licensed to Roger Chen button on the issues page to describe the bug. Once you’ve created an issue the project maintainers will be able to reply on that issue page to address the issue, or possibly ask questions to get a better idea of your problem if they need more information. NOTE Issue Tracker is not a Support Forum Depending on the project, it may not be considered appropriate for you to open general support questions on the project's GitHub Issue Tracker. This is usually the case if the project has set up another means for users to get general support like a Google Group for example. It's a good idea to read the project's README file to see if it has a preference regarding general support or questions. Now that you know where to go online to file issues for projects, we’re going to talk about GitHub’s non-support role as the website through which most Node development collaboration takes place. GitHub is the center of gravity for much of the open source world and critical to Node developers. The GitHub service provides hosting for Git, a powerful version control system (VCS), and a web interface that allows you to easily browse Git repositories. GitHub is free to use for open source projects. NOTE Git The Git VCS has become a favorite among open source projects. It’s a distributed version control system (DVCS), which, unlike Subversion and many other VCSs, you can use without a network connection to a server. Git was released in 2005, inspired by a another, proprietary, VCS called “BitKeeper.” The publisher of BitKeeper had granted the Linux kernel development team free use of the software, but revoked it when suspicion arose that members of the team were attempting to figure out BitKeeper’s inner workings. Linus Torvalds, the creator of Linux, decided to create an alternative VCS with similar functionality and, within months, Git was being used by the Linux kernel development team. In addition to Git hosting, GitHub provides projects with issue tracking, wiki functionality, and web page hosting. As most Node projects in the npm repository are hosted on GitHub, knowing how to use GitHub is helpful for getting the most 13.2 GitHub ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 401 Licensed to Roger Chen out of Node development. GitHub gives you a convenient way to browse code, check for unresolved bugs, and, if need be, contribute fixes and documentation. Another use of GitHub is to “watch” a project. Watching a project provides you with notification of any changes to the project. The number of people watching a project is often used to gauge a project’s overall popularity. GitHub may be powerful, but how do you use it? Let’s delve into that next. When you’ve come up with an idea for a Node-based project or a third-party module you’ll want to set up an account on GitHub, if you haven’t already, for easy access to Git hosting. After you’re set up you can add your projects, which you’ll learn to do in the next section. Because GitHub requires use of Git, you’ll want to configure it before continuing to GitHub. Thankfully, GitHub offers help pages for Mac , Window ,3 4 and Linux to help you properly get set up. Once you’ve configured Git, you’ll5 need to get set up on GitHub by registering on its website and providing a Secure Shell (SSH) public key. You need the SSH key to keep your interactions with GitHub secure. You’ll learn the details of each of these steps in the next section. Note that you only have to do these steps once, not every time you add a project to GitHub. Footnote 3m Footnote 4m Footnote 5m To use GitHub you need to configure your Git tool, letting it know your name and email address. Enter the following two commands: Next, register on the GitHub website. Click to find the sign-up form , fill it in,6 and click “Create an account.” Footnote 6 m 13.2.1 Getting started on GitHub GIT CONFIGURATION AND GITHUB REGISTRATION git config --global "Bob Dobbs" git config --global ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 402 Licensed to Roger Chen Once you’re registered, the next thing you’ll need to do is provide GitHub with an SSH public key . You’ll use this to authenticate your Git transactions by taking the7 following steps: Footnote 7 m 1. Click “Account Settings” 2. Click “SSH Keys” 3. Click “Add SSH Key” At this point what you need to do varies depending on your operating system. GitHub will detect your operating system and show the relevant instructions. Once you’re set up on GitHub, you can add a project to your account and begin pushing commits to it. First, you’ll create a GitHub repository for your project which we will go over shortly. After that you create a Git repository on your local workstation, which is where you do your work before pushing to the GitHub repository. Figure 13.4 outlines this process, the steps of which we’ll cover individually throughout this section. You can also view your project files using GitHub’s web interface. Figure 13.4 The steps to take to add a Node project to GitHub. PROVIDING GITHUB WITH AN SSH PUBLIC KEY 13.2.2 Adding a project to GitHub ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 403 Licensed to Roger Chen 1. 2. 3. 4. 5. 6. Creating a repository on GitHub consists of the following steps: Log in to in your web browser Click on “Dashboard” Click on the “New Repository” icon at the top right Fill out the resulting form describing your repository and click “Create Repository” GitHub then creates an empty Git repository and issue queue for your project You’ll be presented with the steps you need to take to use Git to push your code to GitHub Because it’s helpful to understand what each of these steps does, in this section we’ll run through an example and you’ll learn the bare essentials of using Git. To add an example project to GitHub you’re going to make a Node module with URL-shortening logic called “node-elf.” First, create a temporary directory in which to put your project using the following commands: To use this directory as a Git repository, enter the following command (which will create a directory called “.git” that contains repository metadata): Now that you’ve set up an empty repository, you’ll want to add some files. For this example, we’ll add a file containing the URL-shortening logic. Save the following listing’s 13.1 content into a file called “index.js” in this directory: Listing 13.1 A Node module for URL shortening CREATING A GITHUB REPOSITORY SETTING UP AN EMPTY GIT REPOSITORY mkdir -p ~/tmp/node-elf cd ~/tmp/node-elf git init ADDING FILES TO A GIT REPOSITORY exports.initPathData = function(pathData) { pathData = (pathData) ? pathData : {}; pathData.count = (pathData.count) ? pathData.count : 0; = ( ? : {}; } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 404 Licensed to Roger Chen The initialization function is called implicitly by shorten() and expand() Accepts a "path" string and returns a shortened URL mapping to it Accepts a previously shortened url and returns the expanded URL Next, let Git know that you want this file in your repository. The git “add” command works differently than other version control systems. Instead of adding files to your repository, the git “add” command adds files to Git’s “staging area.” The staging area can be thought of as a checklist where you indicate newly added files, or files that you’ve changed that you’d like to be included in the next revision of your repository. Git now knows that it should track this file. You could add other files to the “staging area” if you wanted to, but for now we only need to add this one file. To let Git know you’d like to make a new revision in the repository, including the changed files you’ve selected in the staging area, use the “commit” command. The commit command can, as with other version control tools, take a “-m” command-line flag to indicate a message describing the changes in the new revision. The version of the repository on your workstation now contains a new revision. For a list of repository changes, enter the following command. exports.shorten = function(pathData, path) { exports.initPathData(pathData); pathData.count++;[pathData.count] = path; return pathData.count.toString(36); } exports.expand = function(pathData, shortened) { exports.initPathData(pathData); var pathIndex = parseInt(shortened, 36); return[pathIndex]; } git add index.js git commit -m "Added URL shortening functionality." ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 405 Licensed to Roger Chen If your workstation was suddenly struck by lightning you’d lose all of your work. To safeguard against unexpected events, and get the full benefits of GitHub’s web interface, you’ll want to send changes you’ve made in your local Git repository to your GitHub account. But before doing this, you’ve got to let Git know where it should send changes to. To do this, you add a Git remote repository. These are referred to as “remotes.” The following line shows how you add a GitHub remote to your repository. Substitute “username” with your username, and note that “node-elf” indicates the name of the project. Now that you’ve added a remote, you can send your changes to GitHub. In Git terminology sending changes is called a repository “push.” In the following command, tell Git to push your work to the “origin” remote you defined earlier. Every Git repository can have one or more branches, which are, conceptually, separate working areas in the repository. You want to push your work into the “master” branch: In the push command the “-u” option tells Git that this remote is the “upstream” remote and branch. The upstream remote is the default remote used. After doing your first push with the “-u” option, you’ll be able to do future pushes by using the following command, which is easier to remember: If you go to GitHub and refresh your repository page you should now see your file. Creating a module and hosting it on GitHub is a quick-and-dirty way to be able to reuse it. If you want, for example, to use your sample module in a project, you could enter the commands in the following example. The require('elf') git log PUSHING FROM GIT TO GITHUB git remote add origin git push -u origin master git push ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 406 Licensed to Roger Chen would then provide access to the module. Note that when cloning the repository, you use the last command-line argument to name the directory into which you’re cloning. You now know how to add projects to GitHub, including how to create a repository on GitHub, how to create and add files to a Git repository on your workstation, and how to push your workstation repository to GitHub. Given the open source world’s embrace of Git, you’ll find many excellent resources to help you go further. If you’re looking for comprehensive instruction on how to use Git, Scott Chacon, one of the founders of GitHub, has written a thorough book that you can purchase or read free online . If a hands-on approach is more your style, the8 official Git site’s documentation page lists a number of tutorials that will get you9 up and running. Footnote 8 m Footnote 9 m Now that you know how to create a GitHub repository from scratch, we’ll look at how you can use GitHub to collaborate with others. Let’s say you’re using a third-party module and run into a bug. You may be able to examine the module’s source code and figure out a way to fix it. If you do, you could email the author of the code, describing your fix and attaching files containing your fixes. But this would require the author to do some tedious work. The author would have to compare your files to her own and combine the fixes from your files with the latest code. But if the author was using GitHub, you could make a “clone” of the author’s project repository, make some changes, then inform the author via GitHub of the bug fix. GitHub would then show the author, on a web page, the differences between your code and the version you duplicated and, if the bug fix is acceptable, combine the fixes with the latest code via a single mouse click. In GitHub parlance, duplicating a repository is known as “forking.” Forking a project allows you to do anything you want to your copy with no danger to the mkdir ~/tmp/my_project/node_modules cd ~/tmp/my_project/node_modules git clone elf cd .. 13.2.3 Collaborating using GitHub ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 407 Licensed to Roger Chen original repository. You don’t need the permission of the original author to fork: anyone can fork any project and submit their contributions back to the original project. The original author may not approve your contribution, but if not you still have your own fixed version, which you can continue to maintain and enhance independently. If your fork were to grow in popularity, others might well fork your fork, and offer contributions of their own. Once you’ve made changes to a fork, submitting these changes to the original author is called a “pull request.” A pull request is a message asking a repository author to “pull” changes. Pulling, in Git parlance, means importing work from a fork and combining the work with your own. Figure 13.5 shows, visually, an example GitHub collaboration scenario. Figure 13.5 An example of a typical GitHub development scenario. Now, let’s walk through an example, represented visually in figure 13.6, of forking a GitHub repository for the purpose of collaboration. Figure 13.6 The process of collaborating on GitHub via forking. Forking starts the collaboration process by duplicating the repository on GitHub to your own account (known as "forking") [A]. You then clone the forked ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 408 Licensed to Roger Chen repository to your workstation [B], make changes to it, commit the changes [C], push your work back to GitHub [D], and send a pull request to the owner of the original repository asking them to consider your changes [E]. If they want to include your changes in their repository they will approve your pull request. Let’s say you want to fork the repository you created earlier in thisnode-elf chapter and add it to the URL-shortening module, which would specify the module’s version number by the module logic itself. This would allow anyone using the module to ensure they’re using the right version and not some other version by accident. First, log into GitHub then navigate to the repository’s main page: Once at the repository page you’d click “Fork” to duplicate the repository. The resulting page is similar to the original repository page with “forked from mcantelon/node-elf” displayed under the repository name. After forking, the next step is to clone the repository to your workstation, make your changes, and push the changes to GitHub. Using this example, the following commands will do the same using the “node-elf” repository: Once you’ve pushed your changes, click “Pull Request” on your fork’s repository page and enter the subject and body of a message describing your changes. Click “Send pull request.” Figure 13.7 shows a screenshot containing typical content: Figure 13.7 The details of a GitHub pull request. The pull request then gets added to the issue queue of the original repository. mkdir -p ~/tmp/forktest cd ~/tmp/forktest git clone cd node-elf echo "exports.version = '0.0.2';" >> index.js git add index.js git commit -m "Added specification of module version." git push origin master ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 409 Licensed to Roger Chen The owner of the original repository can then, after reviewing your changes, incorporate them by clicking “Merge pull request,” entering a commit message, and clicking “Confirm Merge.” This automatically closes the issue. Once you’ve collaborated with someone and created a great module, the next step it getting it out to the world. The best way is to add it to the npm repository. Next, you’ll learn how to publish to npm. Suppose you’ve worked on the URL-shortening module for some time and think it would be useful to other Node users. To publicize it, you could post on Node-related Google Groups talking about its functionality. But you’d be limited in the number of Node users you’d reach, and as people start using your module you wouldn’t have a way to let them know about updates to your module. To solve the problems of discoverability and providing updates you can publish to npm. With npm you can easily define a project’s dependencies allowing them to be automatically installed at the same time as your module. If you’ve created a module designed to store comments about content (blog posts, for example), you could have a module handling MongoDB storage of comment data included as a dependency. Another example would be a module that provides a command-line tool that might have as a dependency a helper module for parsing command-line arguments. Up to this point in the book, you’ve used npm to install everything from testing frameworks to database drivers, but you haven’t yet published anything. In the next section we’ll show you each step you need to take to publish your own work on npm: Preparing a package Writing a package specification Testing a package Publishing a package Any Node module you want to share with the world should be accompanied by related resources such as documentation, examples, tests, and related command-line utilities. The module should come with a “readme” file that provides enough information to get you started quickly. 13.3 Contributing to the Node Package Manager repository 13.3.1 Preparing a package ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 410 Licensed to Roger Chen The package directory should be organized using subdirectories. Table 13.2 lists conventional subdirectories “bin,” “docs,” “example,” “lib,” and “test,” and for what you’d use each of them. Once you’ve organized your package, you’ll want to prepare it for publishing to npm by writing a package specification. When you publish a package to npm, you need to include a machine-readable package specification file. This JSON file is called “package.json” and includes information about your module, such as its name, description, version, dependencies, and other characteristics. Nodejitsu has a handy website that10 shows a sample package.json file and explains what each part of the sample file is for when you hover your mouse over it. Footnote 10 m In a package.json file, only the name and version are mandatory. Other characteristics are optional, but some, if defined, can make your module more useful. By defining a “bin” characteristic, for example, you can let npm know which files in your package are meant to be command-line tools and npm will make them globally available. Table 13.2 Conventional subdirectories in a Node projectm Directory Use bin Command-line scripts docs Documentation example Example application uses lib Core application functionality test Test scripts and related resources 13.3.2 Writing a package specification ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 411 Licensed to Roger Chen The following shows what a sample specification might look like: For comprehensive documentation on available package.json options, enter the following: Because generating JSON by hand is only slightly more fun than hand-coding XML, let’s look at some tools that make it easier. One such tool, ngen, is an npm package that when installed, adds a command-line tool called “ngen.” After asking a number of questions, ngen will generate a package.json file. It’ll also generate a number of other files that are normally included in npm packages, such as a “” file. You can install ngen using the following command line: After you’ve run ngen in the root directory of a module and you’re ready to publish to npm, you’ll end up with output like you see in the following code. Some files may be generated that you don’t need. Delete them. A file will.gitignore be added, for example, that specifies a number of files and directories that shouldn’t normally be added to the Git repository of a project that will be published to npm. A file will also be added, which serves a similar.npmignore function, letting npm know what files can be ignored when publishing the package to npm. { "name": "elf" , "version": "0.0.1" , "description": "Toy URL shortener" , "author": "Mike Cantelon " , "main": "index" , "engines": { "node": "0.4.x" } } npm help json npm install -g ngen Project name: elf Enter your name: Mike Cantelon Enter your email: Project description: URL shortening library create : /Users/mike/programming/js/shorten/node_modules/.gitignore ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 412 Licensed to Roger Chen Generating a package.json file is the hardest part of publishing to npm. Now that you’ve taken this step, you’re ready to publish your module. Publishing a module to npm involves three steps which we will go over in this section: Testing the installation of your package locally Adding an npm user, if you haven’t already Publishing the package to npm To test a package locally, use npm’s “link” command from the root directory of your module. This command makes your package globally available on your workstation, where Node can use it, similar to a package conventionally installed by npm. Next, create a test directory in which you can link-install the package to test it: When you’ve link installed the package, do a quick test of requiring the module by executing the function in the Node REPL, as the following coderequire shows. In the results you should see the variables or functions that your module provides. create : /Users/mike/programming/js/shorten/node_modules/.npmignore create : /Users/mike/programming/js/shorten/node_modules/ create : /Users/mike/programming/js/shorten/node_modules/index.js ... 13.3.3 Testing and publishing a package TESTING PACKAGE INSTALLATION sudo npm link npm link elf node > require('elf'); { version: '0.0.1', initPathData: [Function], shorten: [Function], expand: [Function] } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 413 Licensed to Roger Chen If your package passed the test and you’ve finished developing it, use npm’s “unlink” command from the root directory of your module. Your module will no longer be globally available on your workstation, but later, once you’ve completed publishing your module to npm, you’ll be able to install it normally using the “install” command. Having tested your npm package, the next step is to create an npm publishing account, if you haven’t previously set one up. Enter the following to create your own npm publishing account: You’ll be prompted for a username, an email address, and a password. If your account is successfully added, you won’t see an error message. The next step is to publish. Enter the following to publish your package to npm: You may see the warning “Sending authorization over an insecure channel” but if you don’t see additional errors, your module was published successfully. You can verify your publish was successful by using npm’s “view” command: If you’d like to include one or more private repositories as npm package dependencies, you can. Perhaps you have a module of useful helper functions you’d like to use, but not release publicly on npm. To add a private dependency, where you would normally put the dependency module’s name, you can put any name that’s different than the other dependency names. Where you would normally put the version, you put a Git repository URL. In the following example, an excerpt from a package.json file, the last dependency is a private repository: sudo npm unlink ADDING AN NPM USER npm adduser PUBLISHING TO NPM npm publish npm view elf description ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 414 Licensed to Roger Chen Note that any private modules should also include package.json files. To make sure you don’t accidentally publish one of these modules, set the “private” property in its package.json file to “true”: Now that you know how to get help, collaborate, and contribute you’re ready to dive right into Node community waters. As with most successful open source projects, Node has an active online community, which means you’ll find plenty of available online resources as well as quick answers to your questions using online references, Google Groups, IRC, or GitHub issue queues. In addition to a place where projects keep track of bugs, GitHub also provides Git hosting and the ability to browse Git repository code using a web browser. Using GitHub, other developers can easily “fork” your open source code if they want to contribute bug fixes, add features, or take a project in a new direction. You can also easily submit changes made to a fork back to the original repository. Once a Node project has reached the stage where it’s worth sharing with the world, you can submit it to the Node Package Manager repository. Inclusion in npm makes your project easier for others to find, and if your project is a module, inclusion in npm means your module will be easy to install. You know how to get the help you need, collaborate online, and share your work. Now it’s time to get the most out of Node development. "dependencies" : { "optimist" : ">=0.1.3", "iniparser" : ">=1.0.1", "mingy": ">=0.1.2", "elf": "git://" }, "private": true, 13.4 Summary ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 415 Licensed to Roger Chen A Node is easy to install on most operating systems. Node can either be installed using conventional application installers or by using the command-line. Command-line installation is easy on OS X and Linux, but not recommended for Windows. To help you get started, we've detailed the Node installation on OS X, Windows, and Linux operating systems. We've also explained how you can use the Node Package manager (npm) to find and install useful add-ons. Installing Node on OS X is quite straightforward. The official installer , shown in1 figure A.1, provides an easy way to install a precompiled version of Node and npm. Footnote 1 m Installing Node and community add-ons A.1 OS X setup ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 416 Licensed to Roger Chen Figure A.1 The official Node installer for OS X If you'd rather install from source, however, you can either use a tool called Homebrew , which automates installation from source, or you can manually install2 from source. Installing Node from source on OS X, however, requires you to have XCode developer tools installed. Footnote 2m ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 417 Licensed to Roger Chen NOTE XCode If you don't have XCode installed, you can download XCode from Apple's website . You'll have to register with Apple as a3 developer, which is free, to access the download page. The full XCode installation is a large download (approximately 4GB), so as an alernative Apple also offers Command Line Tools for Xcode which is available for download on the same web page and gives you the minimal functionality needed to compile Node and other open source software projects. Footnote 3 m To quickly check if you have XCode, you can start the Terminal application and run the command . If you have XCodexcodebuild installed you should get an error indicating that your current directory "does not contain an Xcode project". Either method requires entering OS X's command-line interface by running the Terminal application that is usually found in the "Utilities" folder in the main "Applications" folder. If compiling from source, see A.4 "Compiling Node", later in this section, for the necessary steps. An easy way to install Node on OS X is by using Homebrew, an application for managing the installation of open source software. Install Homebrew by entering the following into the command-line: Once Homebrew is installed you can install Node by entering the following: As Homebrew compiles code you'll see a lot of text scroll by. The text is information related to the compiling process and can be ignored. A.1.1 Installation with Homebrew ruby -e "$(curl -fsSkL" brew install node ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 418 Licensed to Roger Chen Node can be most easily installed on Windows by using the official stand-alone installer . After installing you'll be able to run Node and npm from the Windows4 command-line. Footnote 4 m An alternative way to install Node on Windows involves compiling it from source code. This is more complicated and requires the use of a project called Cygwin that provides a Unix compatible environment. You'll likely want to avoid using Node through Cygwin unless you're trying to use modules that won't otherwise work on Windows or need to be compiled, such as some database driver modules. To install Cygwin, navigate to the Cygwin installer download link in your web5 browser and download setup.exe. Double-click setup.exe to start installation then click "Next" repeatedly to select the default options until you reach the "Choose a Download Site" step. Select any of the download sites from the list and click "Next". If you see a warning pop-up about Cygwin being a major release, click "OK" to continue. Footnote 5 m You should now see Cygwin's package selector, as shown in figure A.2. A.2 Windows setup ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 419 Licensed to Roger Chen Figure A.2 The Cygwin's package selector allows you to select open source software that will be installed on your system. You'll use this selector to pick what software functionality you'd like installed in your Unix-like environment (see table A.1 for a list of Node development-related packages to install). ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 420 Licensed to Roger Chen Once you've selected the packages click "Next". You'll then see a list of packages that the ones you've selected depend on. You need to install those as well, so click "Next" again to accept them. Cygwin will now download the packages you need. Once the download has completed, click "Finish". Start Cygwin by clicking the desktop icon or start menu item. You'll be presented with a command-line prompt. You then compile Node (see A.4 for the necessary steps). Table A.1 Cygwin packages needed to run Nodem Category Package devel gcc4-g++ devel git devel make devel openssl-devel devel pkg-config devel zlib-devel net inetutils python python web wget ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 421 Licensed to Roger Chen Installing Node on Linux is usually painless. We'll run through installations, from source code, on two popular Linux distributions: Ubuntu and CentOS. Node is also available through package managers on a number of distributions .6 Footnote 6 Before installing Node on Ubuntu, you'll need to install prerequisite packages. This is done on Ubuntu 11.04 or later using a single command: NOTE Sudo The "sudo" command is used to perform another command as "superuser" (also referred to as "root"). Sudo is often used during software installation because files needs to be placed in protected areas of the filesystem and the superuser can access any file on the system regardless of file permissions. Before installing Node on CentOS, you'll need to install prerequisite packages. This is done on CentOS 5 using the following commands: Now that you've installed the prerequisites you can move on to compiling Node. Compiling Node involves the same steps on all operating systems. In the command-line, you first enter the following command to create a temporary folder in which to download the Node source code: A.3 Linux setup A.3.1 Ubuntu installaton prerequisites sudo apt-get install build-essential libssl-dev A.3.2 CentOS installaton prerequisites sudo yum groupinstall 'Development Tools' sudo yum install openssl-devel A.4 Compiling Node ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 422 Licensed to Roger Chen Next you navigate into the directory created in the previous step: You now enter the following command: Next, you'll see text indicating the download progress. Once progress reaches 100% you're returned to the command prompt. Enter the following command to decompress the file you received: You should then see a lot of output scroll past be returned to the command prompt. Once returned to the prompt, enter the following to list the files in the current folder, which should include the name of the directory you just decompressed: Next, enter the following command to move into this directory: You are now in the directory containing Node's source code. Enter the mkdir tmp cd tmp curl -O tar zxvf node-latest.tar.gz ls cd node-v* ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 423 Licensed to Roger Chen following to run a configuration script that will prepare the right installation for your specific system: Next, enter the following to compile Node: NOTE Cygwin Quirk If you're running Cygwin on Windows 7 or Vista you may run into errors during this step. These are due to an issue with Cygwin rather than an issue with Node. To address them, exit the Cygwin shell then run the ash.exe command-line application (located in the Cygwin directory: usually c:\cygwin\bin\ash.exe). In the ash command-line enter "/bin/rebaseall -v". When this completes, restart your computer. This should fix your Cygwin issues. Node normally takes a little while to compile, so be patient and expect to see a lot of text scroll by. The text is information related to the compiling process and can be ignored. At this point, you're almost done! Once text stops scrolling and you again see the command prompt you can enter the final command in the installation process: When finished, enter the following to run Node and have it display its version number, verifying that it has been successfully installed: ./configure make sudo make install node -v ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 424 Licensed to Roger Chen You should now have Node on your machine! With Node installed you'll be able to use built-in modules that provide you with APIs to perform networking tasks, interact with the file system, and do other things commonly needed in applications. Node's built-in modules are referred to collectively as the Node "core". While Node's core encompasses a lot of useful functionality, you'll likely want to use community-created functionality as well. Figure A.3 shows, conceptually, the relationship between the Node core and add-on modules. Figure A.3 The Node stack is composed of globally available functionality, core modules, and community-created modules. Depending on what language you've been working in, you may or may not be familiar with the idea of community repositories of add-on functionality. These repositories are akin to a library of useful application building blocks that can help you do things that the language itself doesn't easily allow out-of-the-box. These repositories are usually modular: rather than fetching the entire library all at once, you can usually fetch just the libraries you need. The Node community has its own tool for managing community add-ons: the Node Package Manager (npm). In this section you'll learn how to use npm to find A.5 Using the Node Package Manager ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 425 Licensed to Roger Chen community add-ons, view add-on documentation, and explore the source code of add-ons. SIDEBAR npm is missing on my system If you've installed Node, then npm is likely already installed. You can test by running on the command-line and see if the command isnpm found. If not, you can install npm by doing the following: Once you've installed npm, enter the following on a command-line to verify npm is working (by asking it to output its version number): If npm has installed correctly, you should see a number similar to the following: If you do, however, run into problems installing npm, the best thing to do is to visit the npm project on Github where the latest installation7 instructions can be found. Footnote 7 m The npm command-line tool provides convenient access to community add-ons. These add-on modules are referred to as "packages" and are stored in an online repository. For users of PHP, Ruby, and Perl npm is analogous to Pear, Gem, and CPAN respectively. The npm tool is extremely convenient. With npm you can download and install cd /tmp git clone git:// cd npm sudo make install npm -v 1.0.3 A.5.1 Searching for packages ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 426 Licensed to Roger Chen a package using a single command. You can also easily search for packages, view package documentation, explore a package's source code, and publish your own packages so they can be shared with the Node community. You can use npm's command to find packages available in itssearch repository. For example, if you wanted to search for an XML generator, you could simply enter the command: The first time npm does a search, there's a long pause as it downloads repository information. Subsequent searches, however, are quick. As an alternative to command-line searching, the npm project also maintains a web search interface to the repository. This website, shown in figure A.4, also8 provides statistics on how many packages exist, which packages are the most depended on by others, and which packages have recently been updated. Footnote 8 m npm search xml generator ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 427 Licensed to Roger Chen Figure A.4 The npm search website provides useful statistics on module popularity. The npm web search interface also lets you browse individual packages, which shows useful data such as the package dependencies and the online location of a project's version control repository. Once you've found packages you'd like to install, there are two main ways of doing so using npm: locally and globally. Locally installing a package puts the downloaded module into folder called "node_modules" in the current working directory. If this folder doesn't exist, npm will create it. Here's an example of installing the "express" package locally: Globally installing a package puts the downloaded module into the "/usr/local" directory on non-Windows operating systems, a directory traditionally used by A.5.2 Installing packages npm install express ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 428 Licensed to Roger Chen Unix to store user installed applications. In Windows, the "Appdata\Roaming\npm" subdirectory of your user directory is where globally installed npm modules are put. Here's an example of installing the "express" package globally: If you don't have sufficient file permissions when installing globally you may have to prefix your command with sudo. For example: After you've installed a package, the next step is figuring out how it works. Luckily, npm makes this easy. The npm tool offers a convenient way to view a package author's online documentation, when available. The "docs" npm command will open a web browser with a specified package's documentation. Here's an example of viewing documentation for the "express" package: You can view package documentation even if the package isn't installed. If a package's documentation is incomplete or unclear, it's often handy to be able to check out the package's source files. The npm tool provides an easy way to spawn a subshell with the working directory set to the top-level directory of a package's source files. Here's an example of exploring the source files of a locally installed "express" package: npm install -g express sudo npm install -g express A.5.3 Exploring Documentation and Package Code npm docs express npm explore express ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 429 Licensed to Roger Chen To explore the source of a globally installed package, simply add the "-g" command-line option after "npm". For example: Exploring a package is also a great way to learn. Reading Node source code often introduces you to unfamiliar programming techniques and ways of organizing code. npm -g explore express ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 430 Licensed to Roger Chen B During development, and especially while learning a new language or framework, debugging tools and techniques can be helpful. In this appendix you'll learn a number of ways to figure out exactly what's going on with your Node application code. Syntax and scope-related errors are a common pitfall of development. The first line of defense, when attempting to determine the root of an application problem, is to look at the code. If you look at the source code, however, and don't immediately see a problem another thing worth doing is running a utility to check your source code for problems. JSHint is one such utility. This utility can alert you to show-stopping errors, such as functions called in code that aren't defined anywhere, as well as stylistic concerns, such as not heeding the JavaScript convention of capitalizing class constructors. Even if you never run JSHint, reading over the types of errors it looks for will alert you to possible coding pitfalls. JSHint is a project based on JSLint, a JavaScript source code analysis tool that has been around for a decade. JSLint, however, is not very configurable and that's where JSHint comes in. JSLint is, in the opinion of many, overly strict in terms of enforcing stylistic recommendations. JSHint, conversely, allows you to tell it what you want it to check for and what you want it to ignore. Semi-colons, for example, are technically required by JavaScript interpreters, but most interpreters use automated semi-colon insertion (ASI) to insert them where they're missing. Because of this, some developers omit them in their source code to lessen visual noise and their code runs Debugging Node B.1 Analyzing code with JSHint ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 431 Licensed to Roger Chen without issue. While JSLint would complain that a lack of semi-colons is an error, JSHint can be configured to ignore this "error" and check for show-stopping issues. Installing JSHint makes available a command line tool called thatjshint checks source code. JSHint should be installed globally using npm by executing the following command. Once you've installed JSHint, you can check JavaScript files by simply entering something like the following example. You'll most likely want to create a configuration file for JSHint that indicates what you want it to check. One way to do so is to copy the default configuration file, available on GitHub , to your workstation and modify it.1 Footnote 1m If you name your version of the config file and include it in your.jshintrc application directory, or in any parent directory of your application directory, JSHint will automatically find and use it. Alternatively, you can run JSHint using the flag to specify aconfig configuration file location. The following example shows JSHint being told to use a configuration file with a non-standard filename. For details about each specific configuration option, check out the JSHint website .2 Footnote 2m npm install -g jshint jshint my_app.js jshint my_app.js --config /home/mike/jshint.json ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 432 Licensed to Roger Chen If your code appears to be legitimate, but your application is still behaving unexpectedly, you may want to add debugging output to get a better sense of what's going on underneath the hood. The module is a built-in Node module that provides functionality usefulconsole for console output and debugging. The function is used to output application status information toconsole.log standard output. The is another name for the same Arguments can be provided, style , as the following example shows.printf() 3 Footnote 3m For outputting warnings and errors, the and console.warn functions operate similarly. The only difference is that insteadconsole.error of printing to standard output they print to standard error. This enables you to, if desired, redirect warnings and errors to a log file as the example below shows. When outputting the contents of an object, the function willconsole.dir output an object's contents. The following example output shows what this looks like. B.2 Outputting debugging information B.2.1 Debugging with the console module OUTPUTTING APPLICATION STATUS INFORMATION console.log('Counter: %d', counter); node server.js 2> error.log { name: 'Paul Robeson', interests: [ 'football', 'politics', 'music', 'acting' ] } ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 433 Licensed to Roger Chen The module includes two functions that, when used together, allow youconsole to time the execution of parts of your code. More than one thing can be timed simultaneously. To start timing, add the following line to your code at the point you'd like timing to start. To end timing, returning the time elapsed since timing started, add the following line to your code at the point where timing should stop. The above line will display the elapsed time. A stack trace provides you with information about what functions executed before a certain point in application logic. When Node encounters an error during application execution, for example, it outputs a stack trace to provide information about what led, in the application's logic, to the error. At any point during your application you can output a stack trace, without causing your application to stop, by executing .console.trace() This will produce output similar to the following example stack trace. Note that stack traces display execution in reverse chronological order. OUTPUTTING TIMING INFORMATION console.time('myComponent'); console.timeEnd('myComponent'); OUTPUTTING STACK TRACES Trace: at lastFunction (/Users/mike/tmp/app.js:12:11) at secondFunction (/Users/mike/tmp/app.js:8:3) at firstFunction (/Users/mike/tmp/app.js:4:3) at Object. (/Users/mike/tmp/app.js:15:3) ... ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 434 Licensed to Roger Chen While debugging output is useful, if you're not activately troubleshooting an issue debugging output can end up just being visual noise. Ideally you could switch debugging output on or off. One technique to toggle debugging output is through the use of an environmental variable. TJ Holowaychuk's module provides a handy tooldebug for this, allowing you to manage debugging output using the DEBUG environmental variable. Chapter 12 details the use of the module.debug For debugging needs beyond adding simple debugging output, Node comes with a built-in command-line debugger. The debugger is invoked by starting your application using the "debug" keyword, as the following example shows. When running a Node application this way, you'll be shown the first few lines of your application and presented with a debugger prompt, as shown in figure B.1. Figure B.1 Starting the build-in Node debugger The "break in server.js:1" means the debugger has stopped before executing the first line. B.2.2 Using the debug module to manage debugging output B.3 Node's built-in debugger node debug server.js ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 435 Licensed to Roger Chen On the debugger prompt you can control the execution of your application. You could enter (or just ) to execute the next line or, alternatively, you couldnext n enter (or just ) to have it execute until interrupted.cont c The debugger can be interrupted by the termination of the application or by what is called a "breakpoint". Breakpoints are points where you wish the debugger to stop execution so you can examine application state. One way to add a breakpoint is by adding a line to your application where you wish to put the breakpoint. This line should contain the statement as listing B.1debugger; shows. The line won't do anything while running Node normally sodebugger; you can leave it and there will be no ill effects. Listing B.1 breakpoint.js: Adding a breakpoint programmatically When running listing B.1 in debug mode it will first break at line one. If you now enter into the debugger it will proceed to create the HTTP server,cont awaiting a connection. If you create a connection by visiting with a browser then you'll see that it breaks at the line.debugger; Enter to continue to the next line of code. The current line will now be anext function call to . If you again enter to continue to thehandleRequest next next line the debugger won't descend into each line of . If you,handleRequest however, enter then the debugger will descend into the step handleRequest B.3.1 Debugger navigation var http = require('http'); function handleRequest(req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello World\n'); } http.createServer(function (req, res) { debugger; handleRequest(req, res); }).listen(1337, ''); console.log('Server running at'); Added a breakpoint to code ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 436 Licensed to Roger Chen function, allowing you to troubleshoot any issues with this particluar function. If you changed your mind about wanting to debug you can thenhandleRequest enter (or ) to "step out" of the function.out o Breakpoints can be set from within the debugger in addition to being specified in source code. To put a breakpoint in the current line in the debugger enter (or ) into the debugger. It is possible to set asetBreakpoint() sb() breakpoint at a specifiec line ( ) or when a specific function is beingsb(line) executed ( ).sb('fn()') For the times that you want to unset a breakpoint, there's the function ( for short). This function takes the sameclearBreakpoint() cb() arguments as the function, only it does the inverse.setBreakpoint() If you want to keep an eye on particular values in the application, you can add what are called "watchers". Watchers inform you of the value of a variable as you navigate through code. For example, when debugging the code in listing B.1 you could enter and, for each step, you'd seewatch("req.headers['user-agent']") what type of browser made the request. To see a list of watchers you'd enter the command. To remove a watch you'd use the command, watchers unwatch for example.unwatch("req.headers['user-agent']") If at any point during debugging, you want to be able to fully examine, or manipulate, the state you can use the command. This allows you to enter anyrepl JavaScript expression and have it evaluate. To exit the REPL and return to the debugger press .CTRL+C Once you're done debugging, you can exit the debugger by pressing CTRL-C twice, pressing , or by entering the command.CTRL-D .exit These are the basics of debugger use. For more information on what can be done with the debugger, visit the offial Node page for it.4 Footnote 4m Node Inspector is an alternative to Node's built-in debugger that uses a WebKit-based browser such as Chrome or Safari, rather than the command-line, as an interface. B.3.2 Examining and manipulating state in the debugger B.4 Node Inspector ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 437 Licensed to Roger Chen Before you begin debugging, Node Inspector should be installed globally with the following npm command. After installation, the commandnode-inspector with be available on your system. To debug a Node application, start it using the command-line--debug-brk option, as shown in the following example. Using the option causes the debugging to insert a breakpoint--debug-brk before the first line of your application. If this isn't desired, you can use the option instead.--debug Once your application is running, start Node Inspector by entering the following command. Node Inspector is interesting because it actually uses the same code as WebKit's Web Inspector, just plugged into Node's JS engine instead, so web developers should feel right at home using it. Once Node Inspector is running, navigate to in your WebKithttp:// browser and you should see the Web Inspector. If you've ran Node Inspector using the option it will immediately show the first script in your--debug-brk application, similar to Figure B.2. If you've used the option you'll have--debug to use the script selector, indicated by the script name "step.js" in figure B.2, to select the script you'd like to debug. B.4.1 Starting Node Inspector npm install -g node-inspector node --debug-brk server.js node-inspector ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 438 Licensed to Roger Chen Figure B.2 The Web Inspector A red arrow is shown to the left of the line of code that will execute next. To step to the next function call in your application, click the button that looks like a small circle with an arc over it. Node Inspector, like the command-line Node debugger, allows you to step into functions as well. When the red arrow is to the left of a function call you can descend into the function by clicking the button that looks like a small circle with an arrow pointing down at it. To step out of the function click the button that looks like a small circle with an arrow pointing up. If you're using Node core or community modules the debugger will switch to script files for these modules as you step through your application. Don't be alarmed: it will at some point return to your application's code. To add breakpoints while running Node Inspector, click the line number to the left of any line of a script. If you'd like to clear all breakpoints, click the button to the right of the step out button. Node Inspector also has the interesting feature of allowing you to change code as your application runs. If you'd like to change a line of code, simply double click on it, edit it, then click out of the line. B.4.2 Node Inspector navigation ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 439 Licensed to Roger Chen As you debug your application, you can inspect state using the collapsible panes under the buttons that allow you to navigate in the debugger, as shown in figure B.3. These allow you to inspect the call stack and variables in the scope of code currently being executed. Variables can be manipulated by double-clicking on them and changing their values. You can also, as with Node's built-in command-line debugger, add watch expressions that will display as you step through your application. Figure B.3 Browsing application state using Node Inspector For more details about how to get the most out of Node Inspector, visit its GitHub project page .5 Footnote 5m TIP When in doubt, refresh While using Node Inspector and running into any odd behavior, refreshing the browser may help. If that doesn't work, try restarting both your Node application and Node Inspector. B.4.3 Browsing state in Node Inspector ©Manning Publications Co. Please post comments or corrections to the Author Online forum: 440




需要 10 金币 [ 分享pdf获得金币 ] 0 人已下载





下载需要 10 金币 [金币充值 ]
亲,您也可以通过 分享原创pdf 来获得金币奖励!