book

Big Data Analytics with Java

Name: Big Data Analytics with Java
Author: RAJAT MEHTA
ISBN: 9781787288980

by RAJAT MEHTA

July 2017

Beginner to intermediate

418 pages

9h 46m

English

Packt Publishing

Read now

Unlock full access

Big Data Analytics with Java
Table of Contents
Big Data Analytics with Java
Credits
About the Author
About the Reviewers
www.PacktPub.com
eBooks, discount offers, and moreWhy subscribe?
Customer Feedback
Preface
What this book covers
What you need for this book

Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example codeDownloading the color images of this bookErrataPiracyQuestions
1. Big Data Analytics with Java
Why data analytics on big data?Big data for analyticsBig data – a bigger pay package for Java developersBasics of Hadoop – a Java sub-projectDistributed computing on HadoopHDFS conceptsDesign and architecture of HDFSMain components of HDFSHDFS simple commandsApache SparkConceptsTransformationsActionsSpark Java APISpark samples using Java 8Loading dataData operations – cleansing and mungingAnalyzing data – count, projection, grouping, aggregation, and max/minActions on RDDsPaired RDDsTransformations on paired RDDsSaving dataCollecting and printing resultsExecuting Spark programs on HadoopApache Spark sub-projectsSpark machine learning modulesMLlib Java APIOther machine learning librariesMahout – a popular Java ML libraryDeeplearning4j – a deep learning libraryCompressing dataAvro and Parquet
Summary
2. First Steps in Data Analysis
Datasets
Data cleaning and munging
Basic analysis of data with Spark SQL
Building SparkConf and contextDataframe and datasetsLoad and parse dataAnalyzing data – the Spark-SQL waySpark SQL for data exploration and analyticsMarket basket analysis – Apriori algorithmFull Apriori algorithm
Implementation of the Apriori algorithm in Apache Spark
Efficient market basket analysis using FP-Growth algorithmRunning FP-Growth on Apache Spark
Summary
3. Data Visualization
Data visualization with Java JFreeChartUsing charts in big data analytics
Time Series chart
All India seasonal and annual average temperature series datasetSimple single Time Series chartMultiple Time Series on a single chart window
Bar charts
Histograms
When would you use a histogram?How to make histograms using JFreeChart?
Line charts
Scatter plots
Box plots
Advanced visualization technique
PrefuseIVTK Graph toolkitOther libraries
Summary
4. Basics of Machine Learning
What is machine learning?Real-life examples of machine learningType of machine learningA small sample case study of supervised and unsupervised learningSteps for machine learning problemsChoosing the machine learning modelWhat are the feature types that can be extracted from the datasets?How do you select the best features to train your models?How do you run machine learning analytics on big data?Getting and preparing data in HadoopPreparing the dataFormatting the dataStoring the dataTraining and storing models on big dataApache Spark machine learning APIThe new Spark ML API
Summary
5. Regression on Big Data
Linear regressionWhat is simple linear regression?Where is linear regression used?Predicting house prices using linear regressionDatasetData cleaning and mungingExploring the datasetRunning and testing the linear regression model
Logistic regression
Which mathematical functions does logistic regression use?Where is logistic regression used?Predicting heart disease using logistic regressionDatasetData cleaning and mungingData explorationRunning and testing the logistic regression model
Summary
6. Naive Bayes and Sentiment Analysis
Conditional probability
Bayes theorem
Naive Bayes algorithm
Advantages of Naive BayesDisadvantages of Naive Bayes
Sentimental analysis
Concepts for sentimental analysisTokenizationStop words removalStemmingN-gramsTerm presence and Term FrequencyTF-IDFBag of wordsDatasetData exploration of text dataSentimental analysis on this dataset
SVM or Support Vector Machine
Summary
7. Decision Trees
What is a decision tree?Building a decision treeChoosing the best features for splitting the datasetsAdvantages of using decision treesDisadvantages of using decision treesDatasetData explorationCleaning and munging the dataTraining and testing the model
Summary
8. Ensembling on Big Data
EnsemblingTypes of ensemblingBaggingBoostingAdvantages and disadvantages of ensemblingRandom forestsGradient boosted trees (GBTs)Classification problem and dataset usedData explorationTraining and testing our random forest modelTraining and testing our gradient boosted tree model
Summary
9. Recommendation Systems
Recommendation systems and their types
Content-based recommendation systems
DatasetContent-based recommender on MovieLens datasetCollaborative recommendation systemsAdvantagesDisadvantagesAlternating least square – collaborative filtering
Summary
10. Clustering and Customer Segmentation on Big Data
ClusteringTypes of clusteringHierarchical clusteringK-means clusteringBisecting k-means clustering
Customer segmentation
Dataset
Data exploration
Clustering for customer segmentation
Changing the clustering algorithm
Summary
11. Massive Graphs on Big Data
Refresher on graphsRepresenting graphsCommon terminology on graphsCommon algorithms on graphsPlotting graphs
Massive graphs on big data
Graph analyticsGraphFramesBuilding a graph using GraphFramesGraph analytics on airports and their flightsDatasetsGraph analytics on flights data
Summary
12. Real-Time Analytics on Big Data
Real-time analyticsBig data stack for real-time analyticsReal-time SQL queries on big dataReal-time data ingestion and storageReal-time data processingReal-time SQL queries using ImpalaFlight delay analysis using ImpalaApache KafkaSpark StreamingTypical uses of Spark StreamingBase project setupTrending videosSentiment analysis in real time
Summary
13. Deep Learning Using Big Data
Introduction to neural networks
Perceptron
Problems with perceptronsSigmoid neuronMulti-layer perceptronsAccuracy of multi-layer perceptrons
Deep learning
Advantages and use cases of deep learning
Flower species classification using multi-Layer perceptrons
Deeplearning4j
Hand written digit recognizition using CNN
Diving into the code:More information on deep learning
Summary
Index

Content preview from Big Data Analytics with Java

Hand written digit recognizition using CNN

This is one of the classic "Hello World" type problem in the field of deep learning. We already covered one very simple case study of flower classification earlier and in this one we are going to classify hand written digits. For this case study we are using the MNIST dataset. The MNIST database of handwritten digits is available at http://yann.lecun.com/exdb/mnist/. It has a training set of 60,000 examples, and a test set of 10,000 examples. Some of the sample images in this dataset are as shown:

A typical hello world neural network that we are building is to train our network with the training set and ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781787288980

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Big Data Analytics with Java

by RAJAT MEHTA

Hand written digit recognizition using CNN

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.