Chapter 1. Introduction

What is MCollective?

MCollective provides a framework for server orchestration or parallel job execution. It is commonly used to synchronize changes across clusters of servers in near realtime. It is not entirely inaccurate to imagine the classic marionette controller with puppets dancing on strings. (Yes it’s a pun, but it is more apt than you may realize.)

MCollective is an adjunct tool in your toolbox that cooperates and enhances the capabilities of configuration management tools like Puppet, Chef, and Salt. Whereas these tools ensure configuration consistency, MCollective provides orchestration for parallel action.

Let’s talk about the difference between MCollective and using Chef, Puppet, Capistrano, Salt Overstack, or hand-built tools for orchestration. You may have used or built some home-built command-and-control systems before. You may have built or improved one yourself. And you know their limitations well - any or all of the following:

  • Loop through systems in order, processing a few at a time
  • Simplistic (or overly complex) authentication mechanisms
  • Requires customization for each alternative environment
  • Unable to respond to deviations in response
  • Too easy to overlook fatal error messages output to the screen
  • Doesn’t support or extend existing management tools

MCollective differs from these tools in a wide variety of ways:

  1. Uses a master-less environment that allows parallel execution on thousands of systems
  2. Allows for custom authentication and authorization mechanisms
  3. Transparently handles different platforms, architectures, and locals environments
  4. Full data sets can be returned as result codes, allowing intelligent response
  5. Results can be directed to a processor which takes action on responses
  6. Integrates cleanly with configuration management tools like Puppet and Chef

Let’s talk about how MCollective’s design makes this possible.

Why Parallel Execution?

All of us have spent time trying to make something happen at the same time on a number of systems. How many times have you done something like this?

$ for host in bunch of hosts
     scp config-file $host:/some/path
     ssh $host "service apache restart"

You tell yourself that this is happening all at once, but you know better. Even with more complex parallel execution processors like pssh, ClusterSSH, or Fabric the sequential ordering ensures a delay that creates drift between hosts.

If you have more than a few hosts, the output from the commands drifts off the screen. Did you notice that the 15th host command failed? The better parallel SSH processors will keep the output from each host, but you need to examine it for errors. None of them can identify an error in the middle of a multi-command sequence and bring it to your attention.

When it’s time to do something on a lot of hosts, you want it to happen fast, and you need to know that it succeeded.


Puppet users might want to stand up right now saying, “But I can do this! Puppet can ensure that these things happen on a whole bunch of systems all at once!”

You’re right, Puppet is a great tool for making changes on systems, and ensuring that those changes happen. But Puppet is not intended to make this happen on many systems at the same time. Puppet is intended for eventual compliance over time. Each agent checks in over a period of time, allowing the puppetmaster to process only a few at a time.

In very few environments could every puppet client get a catalogue from the puppetmaster and execute it at the same moment in time. Additionally, each puppet client would take differing times to process their catalogues. MCollective is the perfect complement for Puppet, designed specifically to do things that Puppet wasn’t intended for.

How MCollective Works

MCollective was designed from the ground up to achieve true parallel execution with consistent and repeatable results. MCollective avoids the use of a centralized master for command and control, thus avoiding centralized resource problems. It also doesn’t reach out to the clients in an ordered loop, thus avoiding drift between each of the systems.

MCollective uses Publish-Subscribe Middleware to transport requests to servers. Any node you want to control by MCollective is an MCollective server running mcollectived. Any system which has the mco command installed is an MCollective client which can issue commands.

On each server mcollectived registers with the middleware broker and remains in a listening or IDLE state. Whenever a client sends a request to the middleware, each server receives and evaluates the request immediately and independently. If mcollectived has an agent installed capable of process the request, it executes it immediately and sends back the results. In this model you can have a command execute on tens, hundreds, or thousands of hosts at exactly the same time. This Publish-Subscribe infrastructure delivers a scalable and fast parallel execution environment.

One message goes to many servers
Figure 1-1. One-to-Many Publishing Model used by MCollective

Now you might be thinking to yourself, “What if I only want the command executed on a subset of hosts?” MCollective provides a rich language for describing which systems should execute the commands. You can send filters based on hostname, operating system, packages installed, processes running, and many other criteria. Best of all, new criteria custom to your environment will be available when you create your own agent.

Is this like Multicast IP?

If you are familiar with IP networking and are thinking to yourself that this looks like multicast, then yes it shares a lot of the same benefits. The sending host submits a single message, thus consuming very few resources. Each host which receives the message determines if the message applies to itself and either acts on or discards the message. Like multicast, IP latency is the only factor which influences drift between systems around the world.

Why Use MCollective

As we have discussed above, MCollective provides a well-designed infrastructure for orchestrating changes on large numbers of systems simultaneously. Here are a few reasons to use it:

  1. MCollective uses a decentralized collective implemented with Publish-Subscribe Middleware to avoid the resource problems associated with centralized master/slave environments.
  2. MCollective allows you to filter operations based on customizable criteria: not just hostname, operating system, other common criteria, but anything that can be defined in a custom module.
  3. MCollective agents implement host-specific routines internally, allowing you to issue the same command to different operating systems without being concerned about the differences between them.
  4. MCollective agents report back success, failure, and specific return codes or data types for the entire process initiated.
  5. There are MCollective agents to control, reuse, and interact cleanly with Puppet and Chef. I’ve heard people discuss agents for CFengine, Ansible, and Salt as well.
  6. MCollective has replaced Puppet Kick for controlling Puppet Agents.


If you have used puppet kick in the past, you are likely aware that Puppet Labs has deprecated puppet kick and will be removing support for it in a future release. MCollective replaces Kick in both the community and Puppet Enterprise product lines, and provides significantly more features and functionality.

While MCollective plays very well with configuration management systems, it works above and outside of them. I’ve used MCollective to control and manage systems in hundreds of locations around the world without any configuration management available. I’ve seen MCollective used for multi-continent distributed data collection.

Don’t get tied up thinking of the control MCollective provides you as only puppets dancing on your strings. Consider a fishing model where the marionette holds the strings cautiously, waiting for the strings to go taut. I’ve built auto-healing components which listen passively to server inputs and take action to correct a problem without any human involvement.

There are far more ways to use MCollective than I can make marionette and string metaphors for. After reading this book you’ll likely have thought of a way to use it even the developers didn’t imagine. You’ll find that MCollective’s framework not only supports but encourages creativity.

How to Fail with MCollective

Some sites don’t succeed at using MCollective.

Wow. You may be surprised to a statement this strong at the front of this book. However I have found examining sites which fail with MCollective to be instructive in how to succeed with MCollective. So we’re going to evaluate some reasons I have seen MCollective not be widely deployed at sites.

  1. MCollective is installed. How do I make it work? What does it do for me?

    MCollective is not a software package that provides a singular feature set out of the box. MCollective is a framework for orchestration. As such, MCollective doesn’t do anything until you install agents to answer requests and process actions for you.

    Immediately after Chapter 2 you’ll move directly to install a baseline set of plugins which provide valuable and useful features. By the end of Part I you’ll have a feature rich set of tools for your evaluation of MCollective’s power.

  2. MCollective kept timing out in our Network.

    In a standard configuration MCollective will work in a variety of small and large networks, but any given environment may require tuning. MCollective and ActiveMQ contain hundreds of tuning options capable of supporting almost any global environment.

    In Chapter 8 we review in depth the configuration options, and discuss the changes necessary in large-scale or specialized environments.

  3. I went to the mailing list or IRC channel and nobody answered my questions.

    MCollective has active support by both Puppet Labs staff and friendly users and developers. However all of them are busy people, and none of them are mind readers. A question without a clear meaning may get overlooked. The best way to get help is to:

    1. Phrase your question clearly. Instead of what you see, “MCollective doesn’t work.”, tell the list what you did and what errors you received. Specific queries like “The agents on one node won’t respond. The logs from that server say…” are more likely to get helpful responses.
    2. Show the testing you have done. Provide the relevant configuration and log files when posting to the mailing list, or use a service like Gist (preferred) or Pastebin when posting to the #mcollective IRC Channel.

Posting to the help channels with specific information like this allows people to quickly determine if they can help you, and whether they have seen the problem before. Even a busy person might be able to point you in the right direction.

In summary, MCollective provides a flexible framework for server orchestration. If it isn’t doing what you expect, read through this book and see if your question is already answered. Reach out to the support resources provided in the book with specific questions about what you are trying to accomplish. Other people may have solved problems just like yours.

Developing new functionality with MCollective is a creative endeavor. If nothing available today meets your needs, this book provides you with the technical bits necessary to create your own plugins. When you are done reading this book, you’ll have all of the tools at your disposal. You’ll only fail if you don’t reach out and use them.

Time to Get Started

As we proceed, this book will show you how MCollective can help you do more, do it faster, and yet more precisely than ever before. You’ll learn how to extend MCollective to meet your specific needs:

  1. You’ll install MCollective and get it working seamlessly to control files, packages, services, and the Puppet daemon.
  2. You’ll learn the knobs available to tune in the middleware, allowing you to extend your MCollective environment across the campus or around the globe.
  3. You’ll tour through the security plugins available to cryptographically validate every request in your MCollective environment.
  4. You’ll discover an active community of MCollective developers who develop agents, clients, and other MCollective plugins on GitHub.
  5. You’ll build your own custom agent and client. You’ll test the agent using raw RPC calls, then build a native Ruby script to invoke MCollective features.

By the time you finish this book you will understand not just how powerful MCollective is, but you’ll know exactly how MCollective works. You’ll have the knowledge and understanding to debug problems within any part of the infrastructure. You’ll know what to tune as your collective grows. You’ll have a resource to return to as your knowledge and experience expands.

Let’s get moving. Your servers are marionettes waiting to dance for you. Time for you to take hold of the strings.

Get Learning MCollective now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.