Practical Predictive Analytics

Eliminating duplicate records

In many cases, you will only want to retain one record that results from a join. Let's say we only want to keep one record for each memberid, and we want it to be the highest TotalPurchase for each member. To eliminate duplicates, we can first sort the data frame by memberid and TotalPurchase amount. Then we will use the rev and duplicated functions to keep only the last purchase record for each memberid. This will leave us with one record for each of the original 1,000 members:

join3 <- join3[order(join3$memberid, join3$TotalAmount),] View(join3) dedup <- join3[!rev(duplicated(rev(join3$memberid))),] nrow(join3) nrow(dedup) > nrow(join3) [1] 1105 > nrow(dedup) [1] 1000