Node: Up and Running

Let’s Build Twitter

The previous example shows how easy it is to write something extremely real-time with Node, but often you just want to write a web application. Let’s try to create something similar to Twitter with Node so we can see what it’s like to make a web application. The first thing we should do is install the Express module (Example 2-13). This web framework for Node makes it much easier to create web applications by adding support for common tasks, such as MVC, to the existing http server.

Example 2-13. Installing the Express module

Enki:~ $ npm install express
express@2.3.12 ./node_modules/express 
├── mime@1.2.2
├── connect@1.5.1
└── qs@0.1.0
Enki:~ $

Installing Express is easy using the Node Package Manager (npm). Once we have the framework installed, we can make a basic web application (Example 2-14). This looks a lot like the application we built in Chapter 1.

Note

You can read more about npm in Chapters 6 and 7.

Example 2-14. A basic web server with Express

var express = require('express')

var app = express.createServer()

app.get('/', function(req, res) {
  res.send('Welcome to Node Twitter')
})

app.listen(8000)

This code looks pretty similar to the basic web server code from Chapter 1. Instead of including the http module, however, we include express. Express is actually getting http behind the scenes, but we don’t have to get that ourselves, because Node will automatically resolve the dependencies. Like with http and net, we call createServer() to make a server and call listen() to make it listen to a specific port. Instead of attaching an event listener to the request event with Express, we can call methods matching the HTTP verbs. In this case, when we call get(), we can create a callback function that will match GET requests only to a URL that matches the first argument of the call. This has immediately added two things that the http server didn’t have: the ability to filter based on HTTP verbs, and the ability to filter based on specific URLs.

When we get the callback, it looks a lot like the one from the http server—because it is. However, Express has added some extra methods. With the http server, we needed to create the HTTP headers and send them to the client before sending the body of the request. Express provides a convenience method on the res (http.response) object call named send(), and this method issues both the HTTP headers as well as a response.end() call. So far, we haven’t done much more than the original Hello World server from Chapter 1. However, this server will respond only to a GET request to / without throwing an error. This is in contrast to the previous example, which would respond to any request with any path.

Let’s start adding some features to this server in order to provide some of the Twitter functionality (Example 2-15). At least to start with, we aren’t going to worry about making it super-robust or scalable. We are going to make a few assumptions so you can see how to create applications.

Example 2-15. Adding a basic API

var express = require('express')

var app = express.createServer()
app.listen(8000)

var tweets = []

app.get('/', function(req, res) {
  res.send('Welcome to Node Twitter')
})

app.post('/send', express.bodyParser(), function(req, res) {
  if (req.body && req.body.tweet) {
    tweets.push(req.body.tweet)
    res.send({status:"ok", message:"Tweet received"})
  } else {
    //no tweet?
    res.send({status:"nok", message:"No tweet received"})
  }
})

app.get('/tweets', function(req,res) {
  res.send(tweets)
})

Building on the basic Express app, we’ve added a couple of functions to provide an extremely basic API. But first let’s talk about another change we made. We moved the app.listen() call to the top of the file. It’s important to understand why this doesn’t cause a race condition for the functions that respond to requests. You might imagine that when we call app.listen(), any requests that happen between the app.listen() call and the time it takes to run those functions will be ignored. This is incorrect for two reasons. The first is that in JavaScript everything happens in an event loop. That means new events don’t get called until we’ve finished evaluating the code of the existing loop pass. In this case, no request events will be called (and thus our request-based functions) until we’ve evaluated all the initialization code in the file. The other reason is that the app.listen() call is actually asynchronous because binding to a TCP port takes time. The addition of event listeners (via app.get() and app.post()), on the other hand, is synchronous.

To get some very basic tweeting action going, we’ve added a POST “route” for /send using the app.post() call. This call is a little bit different from the previous example. Obviously, it’s an app.post() rather than an app.get() request. This simply means it accepts HTTP POST requests instead of HTTP GET requests. The significant difference is that we’ve passed an extra argument to the function. You don’t need to do this on all app.post() calls, or any, in fact. The extra argument after the url is a middleware.

A middleware is a small piece of code that sits in between the original request event and the route we defined with app.post(). We use middleware to reuse code for common tasks such as authentication or logging. In this case the middleware’s job is to stream the POST data from the client and then turn it into a JavaScript object that we can use. This middleware is one that is included in Express itself, called bodyParser. We simply include it by specifying it in the arguments we give to the app.post() route. Notice that we call express.bodyParser(). This function call actually returns another function. We use this standard style for middleware to allow you to pass configuration to the middleware if you want to.

If we didn’t include the middleware, we would have to manually write code to accept the data event provided by the request (req) object. Only after we had streamed in all the POST data could we call the code in the app.post() route. Using the middleware not only helps with code reuse but also with clarity.

The express.bodyParser adds a property to req called req.body. This property (if it exists) contains an object representing the POST data. The express.bodyParser middleware will work only for POST requests with the content-type HTTP header of application/x-www-form-urlencoded or application/json. Both of these are easy to parse into key/value pairs as properties of the req.body object.

This means that in the app.post() route we made, the first thing we do is check whether express.bodyParser found any data. We can simply check to see whether req.body was created. If it was, we look for a property called req.body.tweet to represent the tweet. If we find a tweet, we stash it in a global array called tweets and send a JSON string back to the client noting success. If we couldn’t find req.body or req.body.tweet, we send JSON back to the client, noting the failure. Notice how we didn’t serialize the data in the res.send() calls. If we give res.send() an object, it automatically serializes it as JSON and sends the correct HTTP headers.

Finally, to make our basic API complete, we create an app.get() route that listens to /tweets. This route simply sends back JSON for the tweets array.

We can write a few tests for our simple API to make sure it’s working (Example 2-16). This is a good habit to get into, even if you don’t do full test-driven development (TDD).

Example 2-16. A test for the POST API

var http = require('http'),
    assert = require('assert')

var opts = {
  host: 'localhost',
  port: 8000,
  path: '/send',
  method: 'POST',
  headers: {'content-type':'application/x-www-form-urlencoded'}
}

var req = http.request(opts, function(res) {
  res.setEncoding('utf8')

  var data = ""
  res.on('data', function(d) {
    data += d
  })

  res.on('end', function() {
    assert.strictEqual(data, '{"status":"ok","message":"Tweet received"}')
  })
})

req.write('tweet=test')
req.end()

We need the http and assert^[3] modules in order to send HTTP requests and then test the values returned. assert is a core module in Node that lets us test return values in various ways. When a value doesn’t match the expected criteria, an exception is thrown. By making test scripts that check an expected behavior of our program, we can ensure that it is doing what it should be.

The http library doesn’t just contain objects to serve HTTP; it also provides a client. In this test program, we use the http.request() factory method to create a new http.Request object. To create an http.Request, we need an options object. This is a configuration object we pass that has a list of properties defining the functionality we want the http.Request to exhibit. You’ll see config objects used for constructing other Node objects. In this case, we include the hostname (which will be resolved by dns), the port, URL path, HTTP method, and some HTTP headers. Here the settings of the config object reflect what we used when creating our Express server.

The http.request() constructor takes two arguments: the first is the config object, and the second is a callback. The callback is attached to the response event for the http.Request. It’s similar to an http.Server, except we have only one object in the response.

The first thing we do with the response is call setEncoding(). This allows us to define the encoding of all the received data. By setting this to utf8, we ensure that any data we receive will be treated as the right kind of string. Next, we define a variable, data, which we are going to use to stream all the responses from the server. In Express, we can use express.bodyDecoder to catch all the data in a request and stream it, but we don’t have the same luxury in the client, so we’ll do it by hand. It’s really easy. We simply attach a function to the data event on response. Whenever data happens, we append it to our data variable. We can listen for the end event of the response and then take further action on all of the data. The API is set up this way because there are many applications in which it is possible to stream data. In these cases, we can do all of the work in the data event listener rather than aggregating first.

When we get the end event on response, it’s because we have all the data from the server. Now we can run our test on whatever the server sent. Our test in this case is to check whether the data variable has received what we expected from the server. If the server is acting correctly, it should send us back a piece of JSON. By using assert.strictEqual, we are checking that data matches the expected data using ===. If it doesn’t, an assert exception is thrown. We are using the x-www-form-urlencoded format because that’s what a web page form would send.

Now that we’ve set up the request and the event handlers for it, we need to write some data to the server. Calling write() on request lets us send data (since this is a POST request). We send some test data to ensure that the server will respond correctly. Finally, we call end() to indicate that we are finished sending data with the request object.

When we call this script, it will access the server we set up (if it is running) and send a POST request. If it gets back the correct data, it will finish without output. If it can’t connect to the server or if the server responds with the wrong output, it will throw an exception. The goal is to have a set of scripts we can run to check that the server is behaving correctly as we build it.

Now that we have an API, we can start adding a web interface so that people can use our app. Right now, it’s basic, but the API allows people to send messages that everyone can receive. Let’s make an interface to that.

Express supports an MVC (model, view, controller) approach oriented around the routing of requests. The routes act like controllers, providing a way to join the data model with a view. We’ve already used a route (app.get('/', function)). In the folder structure shown in Example 2-17, we can see where we host the different parts of the views. By convention, the views folder holds the view templates, and within it a partials folder contains the “partial views” (we’ll discuss these more later). For applications that don’t use a content delivery network (CDN), the public folder is used to store static files, such as CSS and JavaScript.

Example 2-17. The basic folder structure of an Express app

.
├── app.js
├── public
└── views
    └── partials

To start connecting our very simple model (var tweets = []) with a view, we need to create some views first. We are going to create some basic view files and put them in the views folder. Express offers a few different templating languages and is extensible to allow the addition of more. We are going to start with EJS.^[4] EJS simply embeds JavaScript into the templates with a few simple tags to define how the JavaScript is interpreted. Let’s take a look at an example of EJS, starting with the layout file in Example 2-18.

Example 2-18. EJS layout file

<!DOCTYPE html>
<html lang="en">
    <head>
      <meta charset="utf-8">
      <%- partial('partials/stylesheet', stylesheets) %>
      <title><%= title %></title>
    </head>
    <body>
        <h1><%= header %></h1>
        <%- body %>
    </body>
</html>

The layout file in Express defines a skeleton to use for your site. It’s some basic view boilerplate you will use almost everywhere. In this case, we’ve used a very simple HTML5 page. It has a head with some stylesheet definitions and a body. The body consists of an h1 header element and some content. Notice the <% tags. These are the places in which we are going to insert JavaScript variables. The JavaScript to be evaluated is between the <% and %> tags. The tags can also start with = or -, which we will discuss in more detail shortly. Mostly you’ll just reference a piece of data. You can simply list the variable or reference you wish to include in the page. For example, <h1><%= header %></h1> includes the variable header into the h1 element.

There are two special things used in this template. The first is the call to partial(). Partials are mini-templates for code that is expected to repeat again and again with different data. For example, you can imagine the comments on a blog post are the same snippet of HTML repeating many times, but with different pieces of information for each commenter and the comment she made. The actual HTML template doesn’t change. Partials are a way to represent and store those small pieces of code that repeat often, independently of the pages that include them, to make it easy to update the code on all the pages at once. The other special thing in this layout template is the body variable. Because we use the layout template on all the pages on the site (unless we turn it off), we need some way to say where the specific template being rendered goes. Express provides the body variable for this task. This variable will contain the rendered contents of the specific template we load.

Let’s make a render call from a route to see what that looks like before we explore the other templates we’ll need (Example 2-19).

Example 2-19. Rendering the index template from the '/' route

app.get('/', function(req, res) {
  var title = 'Chirpie',
      header = 'Welcome to Chirpie'

  res.render('index', {
    locals: {
      'title': title,
      'header': header,
      'tweets': tweets,
      stylesheets: ['/public/style.css']
    }
  })
})

The route code looks like the other route code we’ve used. However, instead of calling res.send(), we use res.render() as the call to render a template. The first argument is the name of the specific template we want to render. Remember that whatever is in the index template will be rendered into the layout template where the body variable was. The second argument we pass to res.render() is a configuration object. In this case, we haven’t done any configuration, except providing some local variables. The locals property of the config object contains the data used to render this template. We’ve passed in a title, a header, the array of tweets, and an array of stylesheets. All of these variables will be available to both the layout template and the index template.

We want to define an index template that is going to take the list of tweets and render them so that everyone can see the messages being posted (Example 2-20). We aren’t going to do individual tweet streams just yet, but we can make a page in which everyone can see all the messages being posted and post their own messages using the API.

Example 2-20. An index template to show tweets and let people post new tweets

<form action="/send" method="POST">
  <input type="text" length="140" name="tweet">
  <input type="submit" value="Tweet">
</form>
<%- partial('partials/chirp', tweets) %>

This index template is really simple. We have a small form to provide an input method for new tweets. That’s just regular HTML, but we can make it more AJAX-y later. We also have a partial for the tweets. Because they are all the same, we don’t want to put in an ugly loop with some markup embedded in the index template. By using a partial, we can make one smaller template to represent tweets in those templates in which we want to include them. This keeps the code nice and DRY.^[5] We can add more stuff later, but this gives us the basic functionality we need. We’ll still need to define the partial templates we use in the layout template and the index template (Examples 2-21 and 2-22).

Example 2-21. A partial template for rendering chirps

<p><%= chirp %></p>

Example 2-22. A partial template for rendering stylesheets

<link rel="stylesheet" type="text/css" href="<%- stylesheet %>">

Both of these templates are really simple as well. They take some data and insert it into the markup. Because they get passed an array, they will repeat for each item in the array; however, neither of them is doing anything complex with the items of data. The variable each partial is using to access the array is the same as the name of the template. The template called chirp accesses its data in a variable of the same name. In this case, the data is simple strings, but if we passed in an array of objects, we could do chirp.property or chirp['property'] to access the properties of the objects. Of course, you can also call methods, such as chirp.method().

Now we have an application that allows us to post tweets. It’s very basic, and there are some things that are pretty suboptimal. Let’s correct a few of those things. The first obvious problem is that when we post a new tweet, it takes us to the “send JSON” endpoint. It’s not bad that we are accessing /send, but rather that it treats all clients the same. The tweets are also coming out in chronological order and we haven’t been saving a timestamp, so we don’t know how fresh they are. We’ll fix that too.

Fixing the /send endpoint is pretty simple. When HTTP clients send a request, they can specify the kind of response they want in order of preference. Typical browsers request text/html first and then various other formats. When performing API requests, however, the client can specify application/json in order to get the correct output. By checking for the accept HTTP header, we can make sure we send browsers back to the home page but simply return JSON to API clients.

The accept HTTP header might look like text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. That header is from the Chrome browser, and it contains a number of MIME types separated by commas. First, we need a small function to figure out whether text/html is in the accept header (Example 2-23), and then we can use that to test the header and do some logic in the route.

Example 2-23. A small function to check for text/html in an accept header

function acceptsHtml(header) {
  var accepts = header.split(',')
  for(i=0;i<accepts.length;i+=0) {
    if (accepts[i] === 'text/html') { return true }
  }

  return false
}

This function splits the header across the commas. Then we iterate over that array and simply return true if any of them match text/html; otherwise, we’ll return false if none of them matched. We can use this in our route function to check whether it is a request from a web browser or an API request (Example 2-24).

Example 2-24. Redirect web browsers from the /send endpoint

app.post('/send', express.bodyParser(), function(req, res) {
  if (req.body && req.body.tweet) {

    tweets.push(req.body.tweet)
    
    if(acceptsHtml(req.headers['accept'])) {
      res.redirect('/', 302)
    } else {
      res.send({status:"ok", message:"Tweet received"})
    }

  } else {
    //no tweet?
    res.send({status:"nok", message:"No tweet received"})
  }
})

Much of this code is the same as Example 2-10, but now we have a check for whether the accept header asks for text/html. If it does, we redirect back to / using the res.redirect command. We use a 302 status code because this isn’t a permanent move. Instead, we want the browser to still go to /send each time before redirecting.

---

^[3] You can read more about assert in Chapter 5.

^[4] More of Express’s view languages are covered in Chapter 7.

^[5] Don’t repeat yourself.