Programming Google App Engine with Python

Chapter 5. Using Modules

You can build large web applications using just App Engine’s automatically scaling instances, and many have. Automatic scaling is well suited to large scale user traffic and can accommodate real-world traffic spikes with ease. But it isn’t long before you want to do more computational tasks with App Engine besides serving user traffic and residual tasks. Other parts of a mature app’s architecture, such as batch jobs, long-running computing tasks, and special purpose always-on backend services, don’t quite fit the same mold. Performance tuning that’s suited to one kind of computation doesn’t suit another. Your web traffic may work well with small instance classes and aggressive pending queues, while your nightly data crawl needs more memory on a single instance. And you probably don’t want your user traffic instance pool saturated by a batch job.

App Engine lets you define sets of instances for different purposes, called modules. Each module can be configured and tuned separately using all of the options we’ve discussed so far, such as instance classes and automatic scaling parameters. Each module gets its own URL, so it can be addressed individually. You can make a module’s URL publicly accessible or just call it internally from other modules. Each module can scale its own instances according to its purpose-specific traffic patterns.

For even greater flexibility, App Engine offers two additional scaling patterns beyond automatic scaling: ...