MCollective provides a framework for server orchestration or parallel job execution. It is commonly used to synchronize changes across clusters of servers in near realtime. It is not entirely inaccurate to imagine the classic marionette controller with puppets dancing on strings. (Yes it’s a pun, but it is more apt than you may realize.)
MCollective is an adjunct tool in your toolbox that cooperates and enhances the capabilities of configuration management tools like Puppet, Chef, and Salt. Whereas these tools ensure configuration consistency, MCollective provides orchestration for parallel action.
Let’s talk about the difference between MCollective and using Chef, Puppet, Capistrano, Salt Overstack, or hand-built tools for orchestration. You may have used or built some home-built command-and-control systems before. You may have built or improved one yourself. And you know their limitations well - any or all of the following:
MCollective differs from these tools in a wide variety of ways:
Let’s talk about how MCollective’s design makes this possible.
All of us have spent time trying to make something happen at the same time on a number of systems. How many times have you done something like this?
$ for host in
bunch of hostsdo scp
/some/pathssh $host "service apache restart" done
You tell yourself that this is happening all at once, but you know better. Even with more complex parallel execution processors like
Fabric the sequential ordering ensures a delay that creates drift between hosts.
If you have more than a few hosts, the output from the commands drifts off the screen. Did you notice that the 15th host command failed? The better parallel SSH processors will keep the output from each host, but you need to examine it for errors. None of them can identify an error in the middle of a multi-command sequence and bring it to your attention.
When it’s time to do something on a lot of hosts, you want it to happen fast, and you need to know that it succeeded.
Puppet users might want to stand up right now saying, “But I can do this! Puppet can ensure that these things happen on a whole bunch of systems all at once!”
You’re right, Puppet is a great tool for making changes on systems, and ensuring that those changes happen. But Puppet is not intended to make this happen on many systems at the same time. Puppet is intended for eventual compliance over time. Each agent checks in over a period of time, allowing the
puppetmaster to process only a few at a time.
In very few environments could every puppet client get a catalogue from the
puppetmaster and execute it at the same moment in time. Additionally, each puppet client would take differing times to process their catalogues. MCollective is the perfect complement for Puppet, designed specifically to do things that Puppet wasn’t intended for.
MCollective was designed from the ground up to achieve true parallel execution with consistent and repeatable results. MCollective avoids the use of a centralized master for command and control, thus avoiding centralized resource problems. It also doesn’t reach out to the clients in an ordered loop, thus avoiding drift between each of the systems.
MCollective uses Publish-Subscribe Middleware to transport requests to servers. Any node you want to control by MCollective is an MCollective server running
mcollectived. Any system which has the
mco command installed is an MCollective client which can issue commands.
On each server
mcollectived registers with the middleware broker and remains in a listening or
IDLE state. Whenever a client sends a request to the middleware, each server receives and evaluates the request immediately and independently. If
mcollectived has an agent installed capable of process the request, it executes it immediately and sends back the results. In this model you can have a command execute on tens, hundreds, or thousands of hosts at exactly the same time. This Publish-Subscribe infrastructure delivers a scalable and fast parallel execution environment.
Now you might be thinking to yourself, “What if I only want the command executed on a subset of hosts?” MCollective provides a rich language for describing which systems should execute the commands. You can send filters based on hostname, operating system, packages installed, processes running, and many other criteria. Best of all, new criteria custom to your environment will be available when you create your own
If you are familiar with IP networking and are thinking to yourself that this looks like multicast, then yes it shares a lot of the same benefits. The sending host submits a single message, thus consuming very few resources. Each host which receives the message determines if the message applies to itself and either acts on or discards the message. Like multicast, IP latency is the only factor which influences drift between systems around the world.
As we have discussed above, MCollective provides a well-designed infrastructure for orchestrating changes on large numbers of systems simultaneously. Here are a few reasons to use it:
If you have used
puppet kick in the past, you are likely aware that Puppet Labs has deprecated
puppet kick and will be removing support for it in a future release. MCollective replaces Kick in both the community and Puppet Enterprise product lines, and provides significantly more features and functionality.
While MCollective plays very well with configuration management systems, it works above and outside of them. I’ve used MCollective to control and manage systems in hundreds of locations around the world without any configuration management available. I’ve seen MCollective used for multi-continent distributed data collection.
Don’t get tied up thinking of the control MCollective provides you as only puppets dancing on your strings. Consider a fishing model where the marionette holds the strings cautiously, waiting for the strings to go taut. I’ve built auto-healing components which listen passively to server inputs and take action to correct a problem without any human involvement.
There are far more ways to use MCollective than I can make marionette and string metaphors for. After reading this book you’ll likely have thought of a way to use it even the developers didn’t imagine. You’ll find that MCollective’s framework not only supports but encourages creativity.
Some sites don’t succeed at using MCollective.
Wow. You may be surprised to a statement this strong at the front of this book. However I have found examining sites which fail with MCollective to be instructive in how to succeed with MCollective. So we’re going to evaluate some reasons I have seen MCollective not be widely deployed at sites.
MCollective is installed. How do I make it work? What does it do for me?
MCollective is not a software package that provides a singular feature set out of the box. MCollective is a framework for orchestration. As such, MCollective doesn’t do anything until you install agents to answer requests and process actions for you.
Immediately after Chapter 2 you’ll move directly to install a baseline set of plugins which provide valuable and useful features. By the end of Part I you’ll have a feature rich set of tools for your evaluation of MCollective’s power.
MCollective kept timing out in our Network.
In a standard configuration MCollective will work in a variety of small and large networks, but any given environment may require tuning. MCollective and ActiveMQ contain hundreds of tuning options capable of supporting almost any global environment.
In Chapter 8 we review in depth the configuration options, and discuss the changes necessary in large-scale or specialized environments.
I went to the mailing list or IRC channel and nobody answered my questions.
MCollective has active support by both Puppet Labs staff and friendly users and developers. However all of them are busy people, and none of them are mind readers. A question without a clear meaning may get overlooked. The best way to get help is to:
Posting to the help channels with specific information like this allows people to quickly determine if they can help you, and whether they have seen the problem before. Even a busy person might be able to point you in the right direction.
In summary, MCollective provides a flexible framework for server orchestration. If it isn’t doing what you expect, read through this book and see if your question is already answered. Reach out to the support resources provided in the book with specific questions about what you are trying to accomplish. Other people may have solved problems just like yours.
Developing new functionality with MCollective is a creative endeavor. If nothing available today meets your needs, this book provides you with the technical bits necessary to create your own plugins. When you are done reading this book, you’ll have all of the tools at your disposal. You’ll only fail if you don’t reach out and use them.
As we proceed, this book will show you how MCollective can help you do more, do it faster, and yet more precisely than ever before. You’ll learn how to extend MCollective to meet your specific needs:
By the time you finish this book you will understand not just how powerful MCollective is, but you’ll know exactly how MCollective works. You’ll have the knowledge and understanding to debug problems within any part of the infrastructure. You’ll know what to tune as your collective grows. You’ll have a resource to return to as your knowledge and experience expands.
Let’s get moving. Your servers are marionettes waiting to dance for you. Time for you to take hold of the strings.