book

Learning Spark

by Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia

February 2015

Intermediate to advanced

276 pages

7h 18m

English

O'Reilly Media, Inc.

Read now

Unlock full access

AudienceHow This Book Is OrganizedSupporting BooksConventions Used in This BookCode ExamplesSafari® Books OnlineHow to Contact UsContent UpdatesMay 8, 2015Acknowledgments
What Is Apache Spark?A Unified StackSpark CoreSpark SQLSpark StreamingMLlibGraphXCluster ManagersWho Uses Spark, and for What?Data Science TasksData Processing ApplicationsA Brief History of SparkSpark Versions and ReleasesStorage Layers for Spark
Downloading SparkIntroduction to Spark’s Python and Scala ShellsIntroduction to Core Spark ConceptsStandalone ApplicationsInitializing a SparkContextBuilding Standalone ApplicationsConclusion
RDD BasicsCreating RDDsRDD OperationsTransformationsActionsLazy EvaluationPassing Functions to SparkPythonScalaJavaCommon Transformations and ActionsBasic RDDsConverting Between RDD TypesPersistence (Caching)Conclusion
MotivationCreating Pair RDDsTransformations on Pair RDDsAggregationsGrouping DataJoinsSorting DataActions Available on Pair RDDsData Partitioning (Advanced)Determining an RDD’s PartitionerOperations That Benefit from PartitioningOperations That Affect PartitioningExample: PageRankCustom PartitionersConclusion
MotivationFile FormatsText FilesJSONComma-Separated Values and Tab-Separated ValuesSequenceFilesObject FilesHadoop Input and Output FormatsFile CompressionFilesystemsLocal/“Regular” FSAmazon S3HDFSStructured Data with Spark SQLApache HiveJSONDatabasesJava Database ConnectivityCassandraHBaseElasticsearchConclusion
IntroductionAccumulatorsAccumulators and Fault ToleranceCustom AccumulatorsBroadcast VariablesOptimizing BroadcastsWorking on a Per-Partition BasisPiping to External ProgramsNumeric RDD OperationsConclusion
IntroductionSpark Runtime ArchitectureThe DriverExecutorsCluster ManagerLaunching a ProgramSummaryDeploying Applications with spark-submitPackaging Your Code and DependenciesA Java Spark Application Built with MavenA Scala Spark Application Built with sbtDependency ConflictsScheduling Within and Between Spark ApplicationsCluster ManagersStandalone Cluster ManagerHadoop YARNApache MesosAmazon EC2Which Cluster Manager to Use?Conclusion
Configuring Spark with SparkConfComponents of Execution: Jobs, Tasks, and StagesFinding InformationSpark Web UIDriver and Executor LogsKey Performance ConsiderationsLevel of ParallelismSerialization FormatMemory ManagementHardware ProvisioningConclusion

Linking with Spark SQLUsing Spark SQL in ApplicationsInitializing Spark SQLBasic Query ExampleDataFramesCachingLoading and Saving DataApache HiveData Sources/ParquetJSONFrom RDDsJDBC/ODBC ServerWorking with BeelineLong-Lived Tables and QueriesUser-Defined FunctionsSpark SQL UDFsHive UDFsSpark SQL PerformancePerformance Tuning OptionsConclusion
A Simple ExampleArchitecture and AbstractionTransformationsStateless TransformationsStateful TransformationsOutput OperationsInput SourcesCore SourcesAdditional SourcesMultiple Sources and Cluster Sizing24/7 OperationCheckpointingDriver Fault ToleranceWorker Fault ToleranceReceiver Fault ToleranceProcessing GuaranteesStreaming UIPerformance ConsiderationsBatch and Window SizesLevel of ParallelismGarbage Collection and Memory UsageConclusion
OverviewSystem RequirementsMachine Learning BasicsExample: Spam ClassificationData TypesWorking with VectorsAlgorithmsFeature ExtractionStatisticsClassification and RegressionClusteringCollaborative Filtering and RecommendationDimensionality ReductionModel EvaluationTips and Performance ConsiderationsPreparing FeaturesConfiguring AlgorithmsCaching RDDs to ReuseRecognizing SparsityLevel of ParallelismPipeline APIConclusion

Content preview from Learning Spark

Index

Symbols

=> lambda syntax, Introduction to Core Spark Concepts

A

AccumulatorParam, Custom Accumulators
accumulators, Accumulators
- and fault tolerance, Accumulators and Fault Tolerance
- custom, Custom Accumulators
- empty line count in Python and Scala (example), Accumulators
- error count in Python (example), Accumulators
- how they work, Accumulators
aggregate(), Aggregations
aggregations, Aggregations
- conditional aggregate operations in Spark SQL, Spark SQL Performance
- disabling map-side aggregation in combineByKey(), Aggregations
- distributed word count, Aggregations
- per-key average using combineByKey(), Aggregations
- per-key average using reduceByKey() and mapValues(), Aggregations
- per-key combiners, combineByKey() and, Aggregations
Akka actor stream, Akka actor stream
Alternating Least Squares (ALS) algorithm, Alternating Least Squares
Amazon EC2, Amazon EC2-Storage on the cluster
- launching clusters, Launching a cluster
- logging into a cluster, Logging in to a cluster
- pausing and restarting clusters, Pausing and restarting clusters
- storage on the cluster, Storage on the cluster
Amazon S3, Amazon S3
Amazon Web Services (AWS) account, Launching a cluster
Apache Flume, Apache Flume
- pull-based receiver for, Pull-based receiver
- push-based receiver for, Push-based receiver
Apache Hive, Apache Hive
- connecting Spark SQL to existing Hive installation, Apache Hive
- Hive JDBC/ODBC, JDBC/ODBC Server
  - (see also JDBC/ODBC server in Spark SQL))
- HiveServer2, JDBC/ODBC Server
- loading and saving data from, in Spark SQL, Apache Hive ...