Neo4j: The Definitive Guide

Chapter 2. Importing (Much) More Data

You and your team at ElectricHarmony are now faced with the challenge of importing a massive amount of data—which will better represent your total user base—to provide recommendations for users.

You try the method you used for ingesting data in Chapter 1, but it proves to be slow as you increase the size of the tracks’ dataset from a hundred thousand to a million rows. This often leads to the dreaded “spinning wheel of death,” making you wonder if your database choice is effective at scale.

Your concerns are valid. You need to answer key questions from your team and stakeholders, such as:

• Can the system ingest data as fast as the business produces it?

• Can the system serve recommendations in near real time and keep up with the data produced by other systems?

• In case of a disaster, how long would recovery take?

This chapter shows you how to ingest the large datasets necessary to answer these questions.

Your journey begins with an easy-to-understand introduction to database management system internals, including transactions and memory management. Next, you’ll learn how to optimize the LOAD CSV commands you used in Chapter 1. You’ll then move on to more production-like scenarios, importing data using automated programs in your preferred programming language. You will also experiment with different locking strategies to understand when and how parallel data import is feasible without negative impact and conclude with offline data-import ...