O'Reilly logo

Python Social Media Analytics by Michal Krystyanczuk, Siddhartha Chatterjee

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Teamspeed forum spider

Before writing the code for our spider, we have to understand what are we going to use Scrapy for.

First of all, we want to visit all the forums on the Teamspeed website, which we will call categories.

Then, get all the threads for each forum (called subject).

Finally, we extract post content in each thread:

This means that we will need three methods to scan each level of the website's tree structure.

We can now create ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required