January 2020
Intermediate to advanced
640 pages
16h 56m
English
Content preview from Hands-On Software Engineering with Golang
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
Start your free trial
The crawler service
The business logic for the crawler service is quite straightforward. The service uses a timer to sleep until the next update interval is due and then executes the following steps:
- First, it queries the most recent information about partition assignments. This includes the partition number for the pod and the total number of partitions (pod count).
- Using the partition count information from the previous step, a new full Range is created and the extents (UUID range) for the currently assigned partition number are calculated.
- Finally, the service obtains a link iterator for the calculated UUID range and uses it as a data source to drive the crawler component that we built in Chapter 7, Data-Processing Pipelines.
The service ...