Hadoop Application Architectures

Appendix A. Joins in Impala

We provided an overview of Impala and how it works in Chapter 3. In this appendix, we look at how Impala plans and executes a distributed join query. At the time of this writing, Impala has two join strategies: broadcast joins and partitioned hash joins.

Broadcast Joins

The broadcast join is the first and the default join pattern of Impala. In a broadcast join Impala takes the smaller data set and distributes it to all the Impala daemons involved with the query plan. Once distributed, the participating Impala daemons will store the data set as an in-memory hash table. Then each Impala daemon will read the parts of the larger data set that are local to its node and use the in-memory hash table to find the rows that match between both tables, (i.e., perform a hash join). There is no need to read the entire large data set into memory, so Impala uses a 1 GB buffer to read the large table and perform the joining part by part.

Figures A-1 and A-2 show how this works. Figure A-1 shows how each daemon will cache the smaller data set. While this join strategy is simple, it requires that the join occur with at least one small table.

Figure A-1. Smaller table caching in broadcast joins

It’s important to note that:

• The smaller data set is now taking up memory on every node. So if you have three nodes with 50 GB of Impala memory, the smaller data set in a broadcast ...