Web Operations

Using Metrics to Guide Load-Feedback Mechanisms

Another advantage of collecting time-series metrics is being able to programmatically adapt your application to its own metrics. Safe and sophisticated feedback loops can be built, and there are a number of examples where this is useful.

In the world of cloud computing, where provisioning new instances requires only a simple API call to your provider, knowing when to turn up more instances or destroy existing ones can get tricky. If you're making those provision/destroy judgments based on resource usage being collected, it gets a lot easier. This is a common scenario in which to use metrics as a feedback mechanism.

I have an example of a large project at Flickr where this sort of feedback loop proved to be invaluable.

In 2007, Yahoo! decided it would shut down Yahoo!Photos. The plan was pretty simple: notifications to Yahoo!Photos users would go out that the service was going away, and they would have an option to take all of their photos along with metadata to another service, including non-Yahoo! services such as Shutterfly and Kodak Gallery. Flickr was one of the options given.

Coming up with the capacity estimates for this project was going to be a tough exercise. Although we had some metrics describing typical Yahoo!Photos users, in terms of frequency of upload, size of photos, and other factors, we really had no idea how many people would choose the Flickr option, and after they did, how their usage patterns would change. Because ...