Neo4j: The Definitive Guide

Chapter 2. Importing (Much) More Data

A note for Early Release readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. The GitHub repo is https://github.com/neo4j-the-definitive-guide/book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

You and your team at ElectricHarmony are now faced with the challenge of importing a massive amount of data–which will better represent your total user base–to provide recommendations for users.

You try the method you used for ingesting data in Chapter 1, but it proves to be slow as you increase the size of the tracks dataset from a hundred thousand to a million rows. This often leads to the dreaded “spinning wheel of death,” making you wonder if your database choice is effective at scale.

Your concerns are valid. You need to answer key questions from your team and stakeholders, such as:

• Can you ingest all the necessary data into the graph in less than 24 hours if the business produces a certain amount of data in a day?

• Can the system serve recommendations in near real-time and keep up with the data produced by other systems every second?

• In case of a disaster, how long would recovery ...