Expert Python Programming - Third Edition

Delegating work

If your application depends on many external resources, threads may really help in speeding it up.

Let's consider the case of a function that indexes files in a folder and pushes the built indexes into a database. Depending on the type of file, the function calls a different external program. For example, one is specialized in PDFs and another one in OpenOffice files.

Instead of processing all files in a sequence, by executing the right program and then storing the results into the database, your function can set up a single thread for each converter and push jobs to be done to each one of them through a queue. The overall time taken by the function will be closer to the processing time of the slowest converter than to the ...