Skip to content
O'Reilly home
Parquet as a Best Practice with DataFrames
How to Store a DataFrame with Parquet
How to Read a DataFrame Back in From Parquet
Parquet as a Best Practice with DataFrames
How to Store a DataFrame with Parquet
How to Read a DataFrame Back in From Parquet

Architect and Build Big Data Applications

Instructor Ben Lorica
    1. Overview
      1. Introduction To The Course
        58s
      2. About The Author
        1m 20s
      3. Big Data Challenges
        2m 4s
      4. Big Data Characteristics
        5m 50s
      5. Problems In Capitalizing On Big Data
        3m 10s
      6. Solving Big Data Problems
        2m 10s
      7. The Challenges Of Relational Databases
        3m 38s
      1. MapReduce And Hadoop
        2m 27s
      2. MapReduce Algorithm
        6m 8s
      3. Introducing Hadoop
        3m 26s
      1. Hadoop Distributed File System
        5m 41s
      2. Interacting With HDFS
        2m 57s
      1. Hadoop Infrastructure
        3m 48s
      2. YARN
        2m 19s
      1. Programming Hadoop
        4m 54s
      1. Hive
        6m 12s
      2. Hive Architecture
        4m 45s
      3. Hive Data Model
        5m 19s
      4. Hive Queries
        3m 9s
      5. When To Use Hive
        3m 6s
      1. Pig
        3m 1s
      2. Pig Data Model
        3m 30s
      3. Pig Latin
        5m 56s
      4. Pig Example
        3m 40s
      5. When To Use Pig
        3m 12s
      1. Scalding
        3m 14s
      2. Programming With Scalding
        5m 43s
      3. When To Use Scalding
        2m 47s
      1. Hadoop Ecosystem
        8m 49s
      2. HBase
        8m 19s
      3. When To Use HBase
        2m 8s
      4. Beyond Classic Hadoop - Spark And Flink
        7m 15s
      1. NoSQL Stores
        7m 48s
      2. Key-Value Stores
        2m 51s
      3. Columnar Stores
        3m 25s
      4. Document Stores
        2m 36s
      5. Graph Stores
        2m 28s
      6. Data Modeling For NoSQL Stores
        3m 34s
      1. Streaming
        1m 58s
      2. Storm
        3m 45s
      3. Spark And Flink Streaming
        2m 9s
      4. Lambda Architecture
        2m 50s
      1. Introducing Big Data And NoSQL In The Enterprise
        6m 34s
      2. Polyglot Persistence
        5m 27s
      3. Seven Habits Of Successful Big Data And NoSQL Projects
        2m 41s
      1. Wrap-Up
        17s
    1. Overview
      1. Introducing The Course
        4m 41s
      2. Understanding What Cassandra Is
        4m 58s
      3. Learning What Cassandra Is Being Used For
        4m 56s
      4. Understanding The System Requirements
        6m 54s
      5. How To Access Your Working Files
        1m 15s
      6. Opening The Main Virtual Machine
        2m 53s
      7. Pop Quiz
        1m 24s
      1. Understanding That Cassandra Is A Distributed Database
        2m 23s
      2. Learning What Snitch Is For
        3m 53s
      3. Learning What Gossip Is For
        1m 52s
      4. Learning How Data Gets Distributed
        5m 35s
      5. Learning About Replication
        2m 12s
      6. Learning About Virtual Nodes
        3m 1s
      7. Pop Quiz
        1m 25s
      1. Downloading Cassandra
        2m 48s
      2. Ensuring Oracle Java 7 Is Installed
        2m 2s
      3. Installing Cassandra
        3m 44s
      4. Viewing The Main Configuration File
        2m 46s
      5. Providing Cassandra With Permission To Directories
        1m 46s
      6. Starting Cassandra
        3m 41s
      7. Checking Status
        4m
      8. Accessing The Cassandra system.log File
        2m 6s
      9. Pop Quiz
        1m 28s
      1. Understanding Ways To Communicate With Cassandra
        3m 47s
      2. Using CQLSH
        2m 29s
      3. Pop Quiz
        1m 8s
      1. Understanding A Cassandra Database
        1m 54s
      2. Defining A Keyspace
        4m 57s
      3. Deleting A Keyspace
        52s
      4. Pop Quiz
        1m 53s
      5. Lab: Create A Second Database
        2m 39s
      1. Creating A Table
        1m 49s
      2. Defining Columns And Data Types
        2m 48s
      3. Defining A Primary Key
        1m 49s
      4. Recognizing A Partition Key
        2m 44s
      5. Specifying A Descending Clustering Order
        3m 2s
      6. Pop Quiz
        1m 54s
      7. Lab: Create A Second Table
        2m 33s
      1. Understanding Ways To Write Data
        1m 28s
      2. Using The INSERT INTO Command
        4m 45s
      3. Using The COPY Command
        5m 53s
      4. How Data Is Stored In Cassandra
        4m 21s
      5. How Data Is Stored On Disk
        5m 29s
      6. Pop Quiz
        2m 15s
      7. Lab: Insert Data
        9m 10s
      1. Understanding Data Modeling In Cassandra
        1m 21s
      2. Using A WHERE Clause
        4m 17s
      3. Understanding Secondary Indexes
        2m 18s
      4. Creating A Secondary Index
        1m 38s
      5. Defining A Composite Partition Key
        9m 34s
      6. Pop Quiz
        3m 34s
      1. Understanding Cassandra Drivers
        2m 31s
      2. Exploring The DataStax Java Driver
        3m 14s
      3. Setting Up A Development Environment
        4m 4s
      4. Creating An Application Page
        4m 51s
      5. Acquiring The DataStax Java Driver Files
        3m 24s
      6. Getting The DataStax Java Driver Files Through Maven
        2m 23s
      7. Providing The DataStax Java Driver Files Manually
        2m 36s
      8. Connecting To A Cassandra Cluster
        3m 39s
      9. Executing A Query
        7m 47s
      10. Displaying Query Results - Part 1
        5m 59s
      11. Displaying Query Results - Part 2
        7m 20s
      12. Using An MVC Pattern
        4m 59s
      13. Pop Quiz
        2m 50s
      14. Lab: Create A Second Application - Part 1
        5m 20s
      15. Lab: Create A Second Application - Part 2
        9m 49s
      16. Lab: Create A Second Application - Part 3
        3m 8s
      1. Updating Data
        3m 39s
      2. Understanding How Updating Works
        3m 55s
      3. Deleting Data
        7m 10s
      4. Understanding Tombstones
        7m 18s
      5. Using TTLs
        5m 9s
      6. Updating A TTL
        2m 38s
      7. Pop Quiz
        2m 38s
      8. Lab: Update And Delete Data
        7m
      1. Understanding Hardware Choices
        30s
      2. Understanding RAM And CPU Recommendations
        2m 45s
      3. Selecting Storage
        4m 8s
      4. Deploying In The Cloud
        4m 7s
      5. Pop Quiz
        2m 6s
      1. Understanding Cassandra Nodes
        3m 39s
      2. Having A Network Connection - Part 1
        5m 35s
      3. Having A Network Connection - Part 2
        5m 2s
      4. Having A Network Connection - Part 3
        4m 46s
      5. Specifying The IP Address Of A Node In Cassandra
        4m 12s
      6. Specifying Seed Nodes
        6m 30s
      7. Bootstrapping A Node
        6m 18s
      8. Cleaning Up A Node
        2m 59s
      9. Using cassandra-stress
        10m 33s
      10. Pop Quiz
        1m 39s
      11. Lab: Add A Third Node
        10m 42s
      1. Understanding Cassandra Monitoring Tools
        46s
      2. Using Nodetool
        4m 54s
      3. Using JConsole
        3m 24s
      4. Learning About OpsCenter
        3m 24s
      5. Pop Quiz
        1m 49s
      1. Understanding Repair
        5m 17s
      2. Repairing Nodes
        4m 17s
      3. Understanding Consistency - Part 1
        6m 26s
      4. Understanding Consistency - Part 2
        4m 33s
      5. Understanding Hinted Handoff
        3m 30s
      6. Understanding Read Repair
        1m 58s
      7. Pop Quiz
        3m 30s
      8. Lab: Repair Nodes For A Keyspace
        5m 45s
      1. Understanding Removing A Node
        54s
      2. Decommissioning A Node
        4m 36s
      3. Putting A Node Back Into Service
        6m 38s
      4. Removing A Dead Node
        6m 42s
      5. Pop Quiz
        4m 10s
      6. Lab: Put A Node Back Into Service
        5m
      1. Redefining For Multiple Data Centers - Part 1
        4m 50s
      2. Redefining For Multiple Data Centers - Part 2
        5m 59s
      3. Changing Snitch Type
        5m 25s
      4. Modifying cassandra-rackdc.properties
        7m 45s
      5. Changing Replication Strategy - Part 1
        5m 55s
      6. Changing Replication Strategy - Part 2
        3m 58s
      7. Pop Quiz
        2m 30s
      1. Accessing Documentation
        2m 51s
      2. Reading Blogs And Books
        4m 53s
      3. Watching Video Recordings
        4m 5s
      4. Posting Questions
        4m 10s
      5. Attending Events
        3m
      6. Wrap Up
        1m 3s
    1. Overview
    2. The Case for Kafka
      11m 23s
    3. The Basics
      9m 10s
    4. Setting up a Kafka Cluster
      15m 30s
    5. Writing a Kafka Producer
      14m 33s
    6. Writing a Kafka Consumer
      16m 34s
    7. Using Kafka from Python
      8m 3s
    8. Troubleshooting Kafka
      29m 29s
    9. Integrating Kafka and Hadoop with Flafka
      26m 6s
    10. Kafka Availability and Consistency
      22m 38s
    11. Kafka Ecosystem
      13m 13s
    12. Future of Kafka
      8m 53s
    1. Overview
    2. Pre-Flight Check
      13m 8s
    3. Spark Deconstructed
      14m 31s
    4. A Brief History
      23m 28s
    5. Simple Spark Apps
      25m 7s
    6. Spark Essentials
      35m 18s
    7. Spark Examples
      21m 55s
    8. Unifying the Pieces - Spark SQL
      24m 7s
    9. Unifying the Pieces - Spark Streaming
      14m 48s
    10. Unifying the Pieces - MLlib and GraphX
      20m
    11. Unified Workflows Demo
      22m 35s
    12. The Full SDLC
      4m 1s
    13. Developer Certification
      6m 10s
    14. Resources
      4m 44s
    15. Introduction - Why DataFrames?
      2m 28s
    16. ETL to Prepare the Data from Capital Bikeshare
      2m 46s
    17. Create a DataFrame, Explore using SQL
      2m 47s
    18. Data Preparation for Machine Learning Models
      5m 33s
    19. Build a Classifier Using Naive Bayes
      4m 43s
    20. Build a Classifier Using Decision Trees
      2m 26s
    21. Build a Classifier Using Random Forests
      2m 20s
    22. Use a DataFrame to Compare Models
      4m 15s
    23. Parquet as a Best Practice with DataFrames
      58s
    24. Taken from
      Introduction to Apache Spark
      By
      Paco Nathan
      Published by
      O'Reilly Media, Inc., 2015
    25. How to Read a DataFrame Back in From Parquet
      2m 57s
    26. Use SQL to Estimate Route Durations
      1m 41s
    27. Data Preparation for GraphX - Model Route Costs
      4m 43s
    28. Use PageRank to Rank Popular Stations
      3m 14s
    29. Optimize Routes to Columbus Circle
      3m 43s
    30. Compare Results with Google Maps
      1m 58s
    31. Analyze a Popular Tourist Route
      2m 30s
    32. Examples of How to Use DataFrames in Python
      2m 57s
    33. Summary - The New DataFrames Features in Spark
      1m 3s
    1. Overview
    2. Introduction - Building big data platforms at Strata+Hadoop World - Ben Lorica
      53s
    3. Big Data at Netflix: Faster and Easier - Kurt Brown
      40m 27s
    4. Building Interactive Data Applications at Scale - Fangjin Yang and Vadim Ogievetsky
      42m 56s
    5. Open Source Real Time BI using Storm, Hadoop, Titan, Druid & D3 - Anil Madan
      50m 36s
    6. Building Real-time Data Products at LinkedIn with Apache Samza - Martin Kleppmann
      49m 42s
    7. An Open Source Approach to Gathering and Analyzing Device Sourced Health Data - Ian Eslick
      41m 41s
    8. Ticketmaster: Marketing and Selling the World's Tickets - John Carnahan
      39m 35s
    9. Unlocking Big Data at CERN - Matthias Braeger and Manish Devgan
      41m 13s
    10. Unboxing Data Startups - Michael Abbott
      38m 50s
    1. Overview
    2. Introduction to Clickstream Case Study
      11m 19s
    3. Requirements
      8m 4s
    4. Data Modeling
      14m 55s
    5. Data Ingest
      16m 16s
    6. Data Processing Engines - Part 1
      16m 23s
    7. Data Processing Engines - Part 2
      10m 59s
    8. Data Processing Patterns
      9m 32s
    9. Orchestration
      14m 34s
    10. Putting It All Together
      3m 8s
    11. Demo
      21m 47s
    12. Q&A
      24m 35s
    1. Overview
      1. Introduction to Time Series Problems
        9m 58s
      1. Kafka Architecture and Deployment
        11m 33s
      2. Kafka Usage
        3m 42s
      1. Introduction to Spark
        15m 43s
      2. Spark Architecture
        12m 2s
      1. Spark Streaming: Windows & Slides
        8m 35s
      2. Spark Streaming: Ingestion Sources & Using Kafka
        8m 32s
      3. Sparks Streaming: Operations on the Stream
        1m 30s
      1. Introduction to Cassandra
        8m 56s
      2. Cassandra Basic Architecture
        11m 59s
      3. Replication, High Availability and Multi Datacenter
        14m 6s
      4. Cassandra Weather Website Example
        11m 46s
      5. Cassandra Query Language (CQL)
        18m
      6. Cassandra Partitions & Clustering
        8m 22s
      7. Cassandra Read and Write Path
        12m 17s
      8. Working with Cassandra
        6m 32s
      9. Cassandra Drivers and Access Patterns
        10m 37s
      1. Spark and Cassandra Architecture
        12m
      2. Analyzing Cassandra Data & Spark SQL
        12m 12s
      3. Spark and Cassandra DataStax Enterprise
        4m 31s
      1. Real World Use Cases: Streaming Problems
        17m 11s
      2. Real World Use Cases: In-place Analytic Problems
        10m 58s