Behavioral Data Analysis with R and Python

Chapter 10. Cluster Randomization and Hierarchical Modeling

Our last experiment, while conceptually simple, will illustrate some of the logistical and statistical difficulties of experimenting in business. AirCnC has 10 customer call centers spread across the country, where representatives handle any issue that might come up in the course of a booking (e.g., the payment did not go through, the property doesn’t look like the pictures, etc.). Having read an article in the Harvard Business Review (HBR) about customer service,¹ the VP of customer service has decided to implement a change in standard operating procedures (SOP): instead of apologizing repeatedly when something went wrong, the call center reps should apologize at the beginning of the interaction, then get into “problem-solving mode,” then end up offering several options to the customer.

This experiment presents multiple challenges: due to logistical constraints, we’ll be able to randomize treatment only at the level of call centers and not reps, and we’ll have difficulties enforcing and measuring compliance. This certainly doesn’t mean that we can’t or shouldn’t run an experiment!

Regarding the randomization constraint, we’ll see that this makes the standard linear regression algorithm inappropriate and that we should use hierarchical linear modeling (HLM) instead.

As before, our approach will be:

• Planning the experiment

• Determining random assignment and sample size/power

• Analyzing the experiment

Planning the ...