Chapter 1. Infrastructure in the Cloud

The World Wide Web has grown quickly over the last couple of decades to become an invaluable resource for communication, research, and entertainment. The Web has also become an open platform on which powerful services and applications can be built by established companies and newcomers alike. It is a very accessible platform that allows even small companies to create web applications and build a business without requiring the backing of a large enterprise. A person or group with some expertise, some time, and a good enough idea can create a web application that competes with the offerings of larger corporations—or even carves out an entirely new market. On the Web, the size and marketing clout of a large corporation does not guarantee it a monopoly on the attention and patronage of a global audience.

The Web is full of opportunities for companies both large and small, but the smaller companies face a difficult problem: infrastructure.

Web applications that are popular and have thousands of users require significant infrastructure to provide the high performance and smooth experience that users demand. Industrial-strength infrastructure is very expensive to buy and maintain, so smaller companies with fewer users are often forced to do without. Yet in today’s world of web publicity flash storms caused by sites such as Slashdot and Digg, the difference between a web application serving a few dozen users and serving thousands may be no more than a glowing article and a few hours’ time.

Although this kind of attention may be exactly what you hope for, unless you have invested heavily in infrastructure, your application may not survive the onslaught. On the other hand, if you spend too much money on servers, bandwidth, hosting, and the management of all this infrastructure, there will be little left to develop the application itself. A dilemma facing many small development teams is how to strike the right balance between investing in application development and funding robust and scalable infrastructure.

Amazon offers a new and compelling solution to this dilemma in the form of infrastructure web services. These services allow application developers to avoid altogether the burden of buying and maintaining physical infrastructure by making it possible to rent virtual infrastructure instead. In this book we will show you how you can build your applications on top of Amazon’s services and effectively outsource your infrastructure.

Amazon Web Services for Infrastructure

In this book, we will examine four offerings from Amazon Web Services (AWS) that provide flexible and affordable infrastructure components on which you can build industrial-strength web applications.

Amazon Simple Storage Service: S3

Amazon Simple Storage Service (S3) offers secure online storage space for any kind of data, providing an alternative to building, maintaining, and backing-up your own storage systems. It makes your data accessible to any other applications or individuals you allow from anywhere on the Web. There are no limits on how much data you can store in the service, how long you can store it, or on how much bandwidth you can use to transfer or publish it.

S3 is a scalable, distributed system that stores your information reliably across multiple Amazon data centers, and it is able to serve it quickly to massive audiences. Its storage application programming interface (API) is deliberately simple and makes no assumptions about the nature of the data you are storing. This simplicity means you can maintain complete control over how your data is represented in the service.

Amazon Elastic Compute Cloud: EC2 (beta)

Amazon Elastic Compute Cloud (EC2) makes it possible to run multiple virtual Linux servers on demand, providing as many computers as you need to process your data or run your web application without having to purchase or rent physical machines. In EC2 you have full control over each server with root access to the operating system (the root user is the ultimate system administrator on Linux machines), a configurable firewall to manage network access, and the freedom to install any software you please. Once you have set up an EC2 server the way you like it, you can save it permanently as a server image. You can then launch new servers from this image to create virtual machines that are preconfigured and ready to do your bidding.

The EC2 service offers computing resources that are very flexible. You can run as many servers as you need for as long as you need them, and you can shut them all down when they have served their purpose. The service offers an API to start and stop server instances, apply access and networking permissions, and manage your server images. You manage each individual server using standard Linux tools over a secure shell session.

Note

At the time of this writing, the EC2 service is still in beta. This means that the service’s features are still evolving quickly, and there is an increased risk of problems that may make the service unsuitable for use in production systems.

Amazon Simple Queue Service: SQS

Amazon Simple Queue Service (SQS) delivers short messages between any computers or systems with access to the Internet, allowing the components of your distributed web applications to communicate reliably without you having to build or maintain your own messaging system. With SQS you can send an unlimited number of messages via an unlimited number of message queues, and you can configure the performance characteristics and access permissions for each queue. The service uses a message locking and timeout mechanism that helps prevent messages from being delivered more than once, while still ensuring they will be delivered despite any component failures or network dropouts.

SQS is implemented as a distributed application within Amazon. Your messages are stored redundantly across multiple servers and data centers. The service’s API allows you to send and receive messages, and to control their full life cycle.

Flexible Payments Service: FPS (Beta)

Amazon Flexible Payments Service (FPS) transfers money between individuals or companies that have Amazon Payments accounts, allowing you to build applications that provide an online store or that implement a marketplace between customers and third-party vendors. With FPS you can make payments from traditional sources, such as credit cards and bank accounts, or from sources internal to Amazon Payments accounts that have lower fees and are designed to make micro-payment transactions feasible.

All transactions need to be authorized by everyone involved in the transaction. The parties involved can impose detailed constraints on transactions, such as how and when transactions can be performed, how much money can be transferred, and who can send and receive the funds.

Customers interact with your FPS application through an Amazon Payments gateway using their Amazon.com account. Because the transactions are mediated by Amazon, your customers are not required to provide you with their personal banking information, and you do not have the burden of securely storing this highly sensitive information.

Note

At the time of this writing, the FPS service is still in beta. This means that the service’s features are still evolving quickly, and there is an increased risk of problems that may make the service unsuitable for use in production systems. The full functionality of the service is only available to developers or users with U.S.-based credit cards and bank accounts. International users can access only limited functionality.

Amazon SimpleDB (Beta)

Amazon SimpleDB (SimpleDB) stores small pieces of textual information in a simple database structure that is easy to manage, modify and search. If your application relies on a relatively simple database, this service can replace your traditional relational database (RDBMS) server leaving you with one less piece of infrastructure to purchase and maintain.

SimpleDB is designed to minimize the complexity and administrative overhead involved in managing your data. It does not require a pre-defined schema so you can alter the structure and content of your database whenever you need to. It indexes every piece of information you store so all your queries run quickly. And it stores your data securely, redundantly and safely within Amazon’s network of data centers.

Note

At the time of this writing, the SimpleDB service is still in beta. This means that the service’s features are still evolving quickly, and there is an increased risk of problems that may make the service unsuitable for use in production systems.

These five web services—S3, EC2, SQS, FPS, and SimpleDB—share the same fundamental characteristics. They are pay-as-you-go, meaning you pay predictable fees based on how much or how little you use the service. There are no initial costs to join, no long-term subscription payments, and the usage fees are attractively low. The services are highly scalable, performing equally well in modest or massively demanding usage scenarios. This means that the applications built on them can be similarly scalable and are able to grow rapidly at short notice without hitting limits imposed by insufficient infrastructure. One significant feature is that all the services are designed to be highly reliable and fault-tolerant: the services and data resources are distributed across multiple servers and data centers within Amazon’s infrastructure, and they are managed by a company with significant experience and investments in the operation of a global web business.

Signing Up for Amazon Web Services

To use AWS you first need to register for an account and provide a credit card to be billed for your service usage. If you already have an Amazon.com account for Amazon’s online store, you can associate your AWS membership with this existing account.

Create a new AWS account at the AWS home page—http://www.aws.amazon.com. This is where you can manage your AWS account, sign up for services, view your service activity, and track billing information.

Screenshot of important elements in the AWS home page

Figure 1-1. Screenshot of important elements in the AWS home page

Once you have registered for an AWS account, you need to sign up separately for each AWS service you wish to use. If you have not explicitly signed up for a service, you will not be able to access its API. To sign up for a service, visit the home page for that service and click on the button “Sign Up For This Web Service.”

Here are the home pages for the infrastructure services we discuss in this book:

To view a listing of the services you have signed up for and the billing history for these services, click on the “Your Web Services Account” button on the AWS home page. Figure 1-1 shows this button that leads to your AWS account information, and which also gives you access to the AWS Access Identifiers page (see Figure 1-2), where you can lookup the AWS access key and X.509 certificate credentials associated with your AWS account.

Screenshot of the AWS Access Identifiers page

Figure 1-2. Screenshot of the AWS Access Identifiers page

You will be billed monthly for your usage of AWS, at which time Amazon will automatically debit your usage fees from the credit card associated with your AWS account. All service charges and payments are in U.S. dollars.

Get Programming Amazon Web Services now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial