Errata

The SQL sample code is not consistent with %scala & %python versions. It includes "distinct" which is absent in the scala/python example.

%sql
SELECT COUNT(DISTINCT ORIGIN_COUNTRY_NAME, DEST_COUNTRY_NAME)FROM dfTable

SQL sample code, second column should be 'collect_list as seen in python & scala versions.

%sql

SELECT
collect_set(Country),
collect_set(Country)
FROM
dfTable

the GLOBAL keyword is missing from the sql statement.

%sql
CREATE VIEW just_usa_global AS SELECT * FROM flights WHERE dest_country_name = 'United States'

in Chapter 6 "Working with Strings" -> Regular Expressions, code example has a function color_locator that incorrectly uses variable "c" when it should have been "color_string".

Under the sub section "sum", it is written:
Another simple task is to add all the values in a row using the sum function

I think it should be "values in a column", and not row.

The option("mode","overwrite") in this example doesn't have any effect when I tried it on Spark 2.2.

dataframe.write.format("csv")
.option("mode", "OVERWRITE")
.option("dateFormat", "yyyy-MM-dd")
.save("path/to/file(s)")

No matter what I give the value of option("mode","anyvalue"), it behaves like mode is "errorIfExists".

Instead, I need to use mode("overwrite")
dataframe.write.format("csv")
.mode("overwrite")
.option("dateFormat", "yyyy-MM-dd")
.save("path/to/file(s)")

Similarly, mode("append") works as expected but not option("mode",...)

Thanks,
Eddie

1. Global view
after creating a global temp view
CREATE GLOBAL TEMP VIEW just_usa_global_view_temp AS
SELECT *
FROM flights
WHERE dest_country_name = 'United States'

I suggest to add a sample query because it involves a built-in object global_temp not discussed in the chapter. A sample query is just

select * from global_temp.just_usa_global_view_temp

2. CASE WHEN
There is a mistake of wrong case for "UNITED STATES". The query should be:
SELECT
CASE WHEN DEST_COUNTRY_NAME = 'United States' THEN 1
WHEN DEST_COUNTRY_NAME = 'Egypt' THEN 0
ELSE -1 END
FROM
partitioned_flights;

3. "For example, if we want to see whether or not we have a flight that will take you back from your destination country we could do so by checking whether or not there was a flight that had the destination country as an origin and a flight that had the origin country as a destination."

It seems that the condition could be fulfilled by 2 flights instead of one flight. An example to meet this requirement "a flight that will take you back from your destination country" is
a flight from "United States" to "Japan"
the return flight should be from "Japan" to "United States"

The conditions and the sample query are saying these 2 flights can provide a return flight for "United States" to "Japan":
i. from "Japan" to "Germany"
ii. from "France" to "United States"

I think the query should look like:
SELECT *
FROM flights f1
WHERE EXISTS (
SELECT 1
FROM flights f2
WHERE f2.origin_country_name = f1.dest_country_name AND
f2.dest_country_name = f1.origin_country_name);

Then only a flight from "Japan" to "United States" can be a return flight of "United States" to "Japan".

4. I think the description of when to call "refresh table" is not clear unless one has background of Hive. Would you consider to elaborate under what circumstances "refresh table" is needed?

1) section glom
sc.parallelize(Seq("Hello", "World"), 2).glom().collect()

I suggest to use
words.glom().collect()

because this call in previous section mapPartitions gives very nice output

words.mapPartitionsWithIndex(indexedFunc).collect()
res6: Array[String] = Array(Partition: 0 => Spark, Partition: 0 => The, Partition: 0 => Definitive, Partition: 0 => Guide, Partition: 0 => :, Partition: 1 => Big, Partition: 1 => Data, Partition: 1 => Processing, Partition: 1 => Made, Partition: 1 => Simple)

calling words.glom().collect() gives
Array[Array[String]] = Array(Array(Spark, The, Definitive, Guide, :), Array(Big, Data, Processing, Made, Simple))

The output is an Array with 2 child Array's from each partition.

2) section reduceByKey

scala> KVcharaters.reduceByKey(addFunc).collect()
<console>:30: error: not found: value KVcharaters
KVcharaters.reduceByKey(addFunc).collect()

but
scala> KVcharacters.reduceByKey(addFunc(_,_)).collect()
res26: Array[(Char, Int)] = Array((d,4), (p,3), (t,3), (h,1), (l,1), (e,7), (a,4), (i,7), (u,1), (m,2), (b,1), (n,2), (f,1), (v,1), (:,1), (r,2), (s,4), (k,1), (o,1), (g,3), (c,1))

3) is it possible to elaborate on treeAgreegate. Even Spark documentation doesn't explain it well.

4. section Joins

scala> sc.parallelize(distinctChars.map(c => (c, new Random().nextDouble())))
<console>:36: error: type mismatch;
found : org.apache.spark.rdd.RDD[(Char, Double)]
required: Seq[?]
Error occurred in an application involving default arguments.
sc.parallelize(distinctChars.map(c => (c, new Random().nextDouble())))

It's due to re-using the variable distinctCharts. The nearest definition is
val distinctChars = words
.flatMap(word => word.toLowerCase.toSeq)
.distinct

which gives the error but the previous one in section sampleByKey
val distinctChars = words
.flatMap(word => word.toLowerCase.toSeq)
.distinct
.collect()

works for sc.parallelize(distinctChars.map(c => (c, new Random().nextDouble())))

Instead of having 2 definitions of distinctChars, I fixed my working examples by using one definition of distinctChars in sampleByKey
val distinctChars = words.
flatMap(word => word.toLowerCase.toSeq).
distinct

and then add collect() before toMap
val sampleMap = distinctChars.map(c => (c, new Random().nextDouble())).collect().toMap

Remove the definition of distinctChars in section Joins and define keyChars as
val keyedChars = distinctChars.map(c => (c, new Random().nextDouble()))

This is from page 42 of the abbreviated book provided directly by Databricks. Actual page number in final document is probably different.

GraphFrames syntax for pageRank is incorrect.

PDF has
ranks = stationGraph.pageRank(maxIter=10).resetProbability(0.15).run()

Correct syntax is
ranks = stationGraph.pageRank(maxIter=10, resetProbability=0.15)

first code snippet, for the python's example, it reads: df.select(map(col("Description")...

It should read
df.select(create_map(col("Description")...

the first grouping SQL command reads:

SELECT
COUNT(*)
FROM
DfTable
GROUP BY
InvoiceNo, CustomerID

The result shows 3 columns: InvoiceNo, CustomerID and Count.
To display a result, a column must be listed in a select statement. The correct query should be:

SELECT
InvoiceNo, CustomerID, COUNT(*)
FROM
DfTable
GROUP BY
InvoiceNo, CustomerID

In Python code, the call to print lr.explainParams() is missing parentheses.

This is from the Kindle version which doesn't include page numbers, so I included the "Location" instead.

In the section titled "Bisecting k-means Summary" in Chapter 29 I think there is a small typo. Instead of using the bisecting k-means model to find information you use the normal k-means model that was introduced in the previous section.

So I believe that:

kmModel.computeCost( sales)
println(" Cluster Centers: ")
kmModel.clusterCenters.foreach( println)

Should probably be:

bkmModel.computeCost( sales)
println(" Cluster Centers: ")
bkmModel.clusterCenters.foreach( println)