Distributed computing with Celery

Distributed computing is where we use multiple different machines to perform a task. For example, if we wanted to perform some operation on every sentence in a large file, instead of using a single process on a single machine, we could break the file down to many smaller files and assign those files to different machines to process in parallel.

As exciting as this sounds, distributed computing has its own set of problems. Shared memory is not possible as tasks are performed on completely different machines, synchronization between machines becomes a challenge, failure handling becomes more complicated, and so on. To ease our problems, we will use a software that makes the task of distributed computing much ...

Get Python Social Media Analytics now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.