Learning MCollective

Chapter 6. Maintenance

Now that you have MCollective working, and have an idea of how powerful MCollective can be, let’s go over some of the steps involved in maintaining and debugging MCollective.

Time Sync

Many of the weirder problems end up being due to the clients and servers having a different idea of what time it is. Before you take any other debugging steps, ensure that your systems have a consistent view of the time: 

client$ date +%s
1396310400
server$ date +%s
1396310402

Allowing for the difference in time taken for you to run the commands on these two systems, they should be within a few seconds. In modern NTP time-sync, 1/100 of a second is a considerable gap, so most systems should be easily within the same second. 

The reason this is important is due to how messages are constructed. Every MCollective message sent out contains the current timestamp and a ttl to indicate how long the message is valid:

{
  :msgtime     => 1396310400,
  :ttl         => 60,
  :filter    => {
                   "fact"   => [{:fact=>"operatingsystem", :value=>"Debian"}],
                   "agent"  => ["package"]
                 },
 :senderid    => "desktop.example.net",
 :msgtarget   => "/topic/mcollective.discovery.command",
 :agent:      => 'discovery',
 :collective' => 'mcollective',
 :body        => body,
 :hash        => "88dd360f13614b7db83616ba49deb130",
 :requestid   => "70141ca8a465954706a51ef6a7d4914e"
}

In the situation described by this packet, the request is valid from 1396310400 until 1396310460. If your server receives a request from a client too far in the past, then the request ...