book

Data Analytics with Spark Using Python, First edition

by Jeffrey Aven

June 2018

Beginner to intermediate

320 pages

10h 1m

English

Addison-Wesley Professional

Read now

Unlock full access

Introduction to Big Data, Distributed Computing, and HadoopA Brief History of Big Data and HadoopHadoop ExplainedIntroduction to Apache SparkApache Spark BackgroundUses for SparkProgramming Interfaces to SparkSubmission Types for Spark ProgramsInput/Output Types for Spark ApplicationsThe Spark RDDSpark and HadoopFunctional Programming Using PythonData Structures Used in Functional Python ProgrammingPython Object SerializationPython Functional Programming BasicsSummary

Spark Deployment ModesLocal ModeSpark StandaloneSpark on YARNSpark on MesosPreparing to Install SparkGetting SparkInstalling Spark on Linux or Mac OS XInstalling Spark on WindowsExploring the Spark InstallationDeploying a Multi-Node Spark Standalone ClusterDeploying Spark in the CloudAmazon Web Services (AWS)Google Cloud Platform (GCP)DatabricksSummary
Anatomy of a Spark ApplicationSpark DriverSpark Workers and ExecutorsThe Spark Master and Cluster ManagerSpark Applications Using the Standalone SchedulerSpark Applications Running on YARNDeployment Modes for Spark Applications Running on YARNClient ModeCluster ModeLocal Mode RevisitedSummary
Introduction to RDDsLoading Data into RDDsCreating an RDD from a File or FilesMethods for Creating RDDs from a Text File or FilesCreating an RDD from an Object FileCreating an RDD from a Data SourceCreating RDDs from JSON FilesCreating an RDD ProgrammaticallyOperations on RDDsKey RDD ConceptsBasic RDD TransformationsBasic RDD ActionsTransformations on PairRDDsMapReduce and Word Count ExerciseJoin TransformationsJoining Datasets in SparkTransformations on SetsTransformations on Numeric RDDsSummary
Shared Variables in SparkBroadcast VariablesAccumulatorsExercise: Using Broadcast Variables and AccumulatorsPartitioning Data in SparkPartitioning OverviewControlling PartitionsRepartitioning FunctionsPartition-Specific or Partition-Aware API MethodsRDD Storage OptionsRDD Lineage RevisitedRDD Storage OptionsRDD CachingPersisting RDDsChoosing When to Persist or Cache RDDsCheckpointing RDDsExercise: Checkpointing RDDsProcessing RDDs with External ProgramsData Sampling with SparkUnderstanding Spark Application and Cluster ConfigurationSpark Environment VariablesSpark Configuration PropertiesOptimizing SparkFilter Early, Filter OftenOptimizing Associative OperationsUnderstanding the Impact of Functions and ClosuresConsiderations for Collecting DataConfiguration Parameters for Tuning and Optimizing ApplicationsAvoiding Inefficient PartitioningDiagnosing Application Performance IssuesSummary
Introduction to Spark SQLIntroduction to HiveSpark SQL ArchitectureGetting Started with DataFramesUsing DataFramesCaching, Persisting, and Repartitioning DataFramesSaving DataFrame OutputAccessing Spark SQLExercise: Using Spark SQLUsing Spark with NoSQL SystemsIntroduction to NoSQLUsing Spark with HBaseExercise: Using Spark with HBaseUsing Spark with CassandraUsing Spark with DynamoDBOther NoSQL PlatformsSummary
Introducing Spark StreamingSpark Streaming ArchitectureIntroduction to DStreamsExercise: Getting Started with Spark StreamingState OperationsSliding Window OperationsStructured StreamingStructured Streaming Data SourcesStructured Streaming Data SinksOutput ModesStructured Streaming OperationsUsing Spark with Messaging PlatformsApache KafkaExercise: Using Spark with KafkaAmazon KinesisSummary
Spark and RIntroduction to RUsing Spark with RExercise: Using RStudio with SparkRMachine Learning with SparkMachine Learning PrimerMachine Learning Using Spark MLlibExercise: Implementing a Recommender Using Spark MLlibMachine Learning Using Spark MLUsing Notebooks with SparkUsing Jupyter (IPython) Notebooks with SparkUsing Apache Zeppelin Notebooks with SparkSummary

Overview

Spark for Data Professionals introduces and solidifies the concepts behind Spark 2.x, teaching working developers, architects, and data professionals exactly how to build practical Spark solutions. Jeffrey Aven covers all aspects of Spark development, including basic programming to SparkSQL, SparkR, Spark Streaming, Messaging, NoSQL and Hadoop integration. Each chapter presents practical exercises deploying Spark to your local or cloud environment, plus programming exercises for building real applications. Unlike other Spark guides, Spark for Data Professionals explains crucial concepts step-by-step, assuming no extensive background as an open source developer. It provides a complete foundation for quickly progressing to more advanced data science and machine learning topics. This guide will help you:

Understand Spark basics that will make you a better programmer and cluster “citizen”
Master Spark programming techniques that maximize your productivity
Choose the right approach for each problem
Make the most of built-in platform constructs, including broadcast variables, accumulators, effective partitioning, caching, and checkpointing
Leverage powerful tools for managing streaming, structured, semi-structured, and unstructured data