Mastering Machine Learning with Spark 2.x

The association rule mining problem

To advance our general introduction of concepts, let's next turn to association rules, as first introduced in Mining Association Rules between Sets of Items in Large Databases, available at http://arbor.ee.ntu.edu.tw/~chyun/dmpaper/agrama93.pdf. In contrast to solely counting the occurrences of items in our database, we now want to understand the rules or implications of patterns. What I mean is, given a pattern P₁ and another pattern P₂, we want to know whether P₂ is frequently present whenever P₁ can be found in D, and we denote this by writing P₁ ⇒ P₂. To make this more precise, we need a concept for rule frequency similar to that of support for patterns, namely confidence. For a rule P₁ ⇒ P₂, confidence ...