O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Apache Mesos Essentials

Book Description

Build and execute robust and scalable applications using Apache Mesos

  • Deploy Apache Mesos to concurrently run cutting edge data processing frameworks like Spark, Hadoop and Storm in parallel
  • Share resources between various cluster computing applications and web applications
  • Detailed guidance on Mesos best practices in a stable production environment

In Detail

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It allows developers to concurrently run the likes of Hadoop, Spark, Storm, and other applications on a dynamically shared pool of nodes. With Mesos, you have the power to manage a wide range of resources in a multi-tenant environment.

Starting with the basics, this book will give you an insight into all the features that Mesos has to offer. You will first learn how to set up Mesos in various environments from data centers to the cloud. You will then learn how to implement self-managed Platform as a Service environment with Mesos using various service schedulers, such as Chronos, Aurora, and Marathon. You will then delve into the depths of Mesos fundamentals and learn how to build distributed applications using Mesos primitives.

Finally, you will round things off by covering the operational aspects of Mesos including logging, monitoring, high availability, and recovery.

What You Will Learn

  • Get to grips with setting up a Mesos cluster in a data centre or in the Cloud
  • Perform data analysis on Mesos using frameworks such as Hadoop, Spark, and Storm
  • Familiarize yourself with managing services on Mesos using Marathon, Chronos, and Aurora
  • Gain insight into how to write a distributed application using the Mesos API
  • Discover how to automate and administer a Mesos Cluster and other operations such as logging and monitoring
  • Explore the fundamentals and internal working of Mesos

Downloading the example code for this book You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.

Table of Contents

  1. Apache Mesos Essentials
    1. Table of Contents
    2. Apache Mesos Essentials
    3. Credits
    4. About the Author
    5. About the Reviewers
    6. www.PacktPub.com
      1. Support files, eBooks, discount offers, and more
        1. Why subscribe?
        2. Free access for Packt account holders
    7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Downloading the color images of this book
        3. Errata
        4. Piracy
        5. Questions
    8. 1. Running Mesos
      1. Modern data centers
      2. Cluster computing frameworks
      3. Introducing Mesos
        1. The master
        2. Slaves
        3. Frameworks
      4. Why Mesos?
      5. Single-node Mesos clusters
        1. Mac OS
        2. Fedora
        3. Installing prerequisites
          1. CentOS
          2. Ubuntu
        4. Build Mesos
        5. Start Mesos
      6. Running test frameworks
      7. Mesos Web UI
      8. Multi-node Mesos clusters
      9. Mesos cluster on Amazon EC2
      10. Running Mesos using Vagrant
      11. The Mesos community
        1. Case studies
          1. Twitter
          2. HubSpot
          3. Airbnb
        2. Mailing lists
      12. Summary
    9. 2. Running Hadoop on Mesos
      1. An introduction to Hadoop
      2. Hadoop on Mesos
      3. Installing Hadoop on Mesos
      4. An example Hadoop job
      5. Advanced configuration for Hadoop on Mesos
        1. Task resource allocation
        2. Metrics reporting
          1. CSV
          2. Graphite
          3. Cassandra
        3. Authentication
        4. Container isolation
        5. Additional configuration parameters
      6. Summary
    10. 3. Running Spark on Mesos
      1. Introducing Spark
      2. Spark job scheduling
      3. Spark Standalone mode
      4. Spark on Mesos
      5. Tuning Spark on Mesos
      6. Summary
    11. 4. Complex Data Analysis on Mesos
      1. Complex data and the rise of the Lambda architecture
      2. Storm
        1. Storm on Mesos
        2. Storm-Mesos configuration
      3. Spark Streaming
        1. Running Spark Streaming on Mesos
        2. Tuning Spark Streaming
          1. Selecting the batch size
          2. Garbage collection
          3. Concurrency
          4. Handling failures
          5. Task overheads
      4. NoSQL on Mesos
        1. Cassandra on Mesos
        2. Summary
    12. 5. Running Services on Mesos
      1. Introduction to services
      2. Marathon
        1. The Marathon API
        2. Running Marathon
        3. Marathon example
        4. Constraints
        5. Event bus
        6. The artifact store
        7. Application groups
        8. Application health checks
      3. Chronos
        1. The Chronos REST API
        2. Running Chronos
        3. A Chronos example
      4. Aurora
        1. Job life cycle
        2. Running Aurora
        3. Aurora cluster configuration
        4. Aurora job configuration
        5. An Aurora client
        6. An Aurora example
        7. Aurora cron jobs
      5. Service discovery
        1. Mesos-DNS
        2. Installing Mesos-DNS
        3. Mesos-DNS configuration
        4. Running Mesos-DNS
      6. Packaging
      7. Summary
    13. 6. Understanding Mesos Internals
      1. The Mesos architecture
        1. Mesos slave
        2. Mesos master
        3. Frameworks
        4. Communication
        5. Auxiliary services
      2. Resource allocation
        1. The Mesos scheduler
        2. Weighted DRF
        3. Reservation
          1. Static reservation
          2. Dynamic reservation
      3. Resource isolation
        1. Mesos containerizer
        2. Docker containerizer
        3. External containerizer
      4. Fault tolerance
        1. ZooKeeper
        2. Failure detection and handling
        3. Registry
      5. Extending Mesos
        1. Mesos modules
          1. Module naming
          2. Module compatibility
        2. Allocation module
        3. Mesos hooks and decorators
        4. Task labels
      6. Summary
    14. 7. Developing Frameworks on Mesos
      1. The Mesos API
        1. Mesos messages
        2. The scheduler API
        3. The SchedulerDriver API
        4. The executor API
        5. The ExecutorDriver API
      2. Developing a Mesos framework
        1. Setting up the development environment
        2. Adding the framework scheduler
        3. Adding the framework launcher
        4. Deploying our framework
      3. Building our framework
        1. Adding an executor to our framework
        2. Updating our framework scheduler
        3. Running multiple executors
      4. Advanced topics
        1. Reconciliation
        2. Stateful applications
      5. Developer resources
        1. Framework design patterns
        2. Framework testing
        3. RENDLER
        4. Akka-mesos
      6. Summary
    15. 8. Administering Mesos
      1. Deployment
      2. Upgrade
      3. Monitoring
        1. Container network monitoring
      4. Multitenancy
        1. Authorization and authentication
        2. API rate limiting
      5. High availability
        1. Master high availability
        2. Slave removal rate limiting
        3. Slave recovery
      6. Maintenance
      7. Mesos interfaces
        1. The Mesos REST interface
        2. The Mesos CLI
      8. Configuration
      9. The Mesos master
        1. The Mesos slave
          1. Mesos build options
        2. Summary
    16. Index