O'Reilly logo

Distributed Computing with Go by V.N. Nikhil Anurag

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Document feeder – the REST API endpoint

The main aim of /api/feeder is to receive documents to be indexed, process them, and forward the processed data to Librarian to be added to the index. This means we need to accurately process the document. But what do we mean by "processing a document?"

It can be defined as the following set of consecutive tasks:

  1. We rely on the payload to provide us with a title and link to the document. We download the linked document and use it in our index.
  1. The document can be thought of as one big blob of text, and it is possible that we might have multiple documents with the same title. We need to be able to identify each document uniquely and also be able to easily retrieve them.
  1. The result of a search query ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required