Monitoring Windows with Nagios and Exported Resources in Puppet

Suppose you would like to monitor your Windows systems alongside your Linux systems in Nagios, but want to avoid configuring every single one manually. Also, you want these systems to be removed from Nagios automatically when you decommission them. Puppet, NSClient++, and Chocolatey provide an excellent means for accomplishing this. In this post I will explain how I got this up and running on my CentOS 6 Nagios server.

This article assumes that you’re already using PuppetDB with exported resources in your Puppet environment. As this is outside the scope of this guide, I’m not going to go into how to set this up. However, it is fairly simple to do, especially with the puppetlabs/puppetdb Forge module. If you’re using Puppet Enterprise, you’re likely already using it. This also assumes that you’re using Puppet to manage your Nagios server. In the class that manages your Nagios server, you will need these two lines, to collect the exported resources from your nodes:

  Nagios_host    <<||>> { notify => Service['nagios'] }
  Nagios_service <<||>> { notify => Service['nagios'] }

You will need to ensure that you have Chocolatey available as a package provider for your Windows nodes. My previous post explains how to configure Chocolatey with Puppet. Chocolatey will be used to install NSClient++, which is a monitoring agent for Windows that includes a NRPE server with which Nagios can interface. The documentation on NSClient++ can be found here.

To prepare your Nagios server for managing Windows hosts, first ensure that you have a Nagios command that can initiate NRPE commands against Windows clients. Because the check_nrpe plugin will throw SSL errors against NSClient++, I’ve defined a nagios_command resource like below for Windows systems that does not use SSL:

  nagios_command { 'check_nrpe_win':
    command_name => 'check_nrpe_win',
    command_line => '$USER1$/check_nrpe -H $HOSTADDRESS$ -n -c $ARG1$ -t 30',
  }

Next, define a standard Nagios hostgroup for Windows that includes basic items to check, such as disk space. You may need to alter this for your own environment. I’ve put this under its own class, nagios::server::hostgroup_windows.

class nagios::server::hostgroup_windows {

  nagios_hostgroup { 'windows_hosts':
    alias => 'Windows Hosts',
  }

  nagios_service { 'check_win_cpu':
    check_command       => 'check_nrpe_win!check_cpu',
    use                 => 'generic-service',
    hostgroup_name      => 'windows_hosts',
    notification_period => '24x7',
    service_description => 'CPU Load',
    max_check_attempts  => 3,
  }

  nagios_service { 'check_win_mem':
    check_command       => 'check_nrpe_win!alias_memory',
    use                 => 'generic-service',
    hostgroup_name      => 'windows_hosts',
    notification_period => '24x7',
    service_description => 'Memory Usage',
    max_check_attempts  => 3,
  }

  nagios_service { 'check_win_drives':
    check_command       => 'check_nrpe_win!alias_space',
    use                 => 'generic-service',
    hostgroup_name      => 'windows_hosts',
    notification_period => '24x7',
    service_description => 'Disk Usage',
    max_check_attempts  => 3,
  }

  nagios_service { 'check_rdp':
    check_command       => 'check_rdp',
    use                 => 'generic-service',
    hostgroup_name      => 'windows_hosts',
    notification_period => '24x7',
    service_description => 'RDP',
    max_check_attempts  => 3,
  }

}

Next, create a template for nsclient.ini, which is the main configuration file for NSClient++. The one I’ve created for this guide is relatively simple; you can refer to the NSClient++ documentation for more options. The template, in my case, is located at nagios/templates/nsclient.ini.erb.

[/modules]
CheckSystem=enabled
CheckDisk=enabled
CheckExternalScripts=enabled
NRPEServer=enabled

[/settings/default]
allowed hosts = your_nagios_server_IP

[/settings/NRPE/server]
use ssl = false
allow arguments = true
allow nasty characters = false
port = 5666

[/settings/external scripts/alias]
alias_memory = check_memory "warn=free < 10%" "crit=free < 5%"
alias_space = check_drivesize "warn=free < 10%" "crit=free < 5%" drive=*

Finally, create a class that installs NSClient++ and manages the service on the Windows systems, and exports the nagios_host resource to the Nagios server.

class nagios::windows {

  package { 'nscp':
    ensure   => present,
  }

  service { 'nscp':
    ensure  => running,
    enable  => true,
    require => Package['nscp'],
  }

  file { 'C:/Program Files/NSClient++/nsclient.ini':
    ensure  => file,
    content => template('nagios/nsclient.ini.erb'),
    notify  => Service['nscp'],
  }

  @@nagios_host { $::fqdn:
    ensure             => present,
    alias              => $::hostname,
    address            => $::ipaddress,
    hostgroups         => 'windows_hosts',
    use                => 'generic-host',
    max_check_attempts => 3,
    check_command      => 'check_ping!100.0,20%!500.0,60%',
  }

}

Once you have committed these changes to your Puppet repository, you would then include nagios::windows in the catalog for your Windows host and trigger a Puppet run on it to install NSClient++, enable the service, and export the resource to your Nagios server. Then, execute a Puppet run on your Nagios server. The result should hopefully be like below.

Nagios Windows

And now your host should be available in Nagios for monitoring.

One additional note: if you include the Windows Hosts host group in the catalog of your Nagios server, you must then have at least one Windows nagios_host exported. Otherwise, Nagios will not start, as it does not allow empty host groups.

How I Configured Chocolatey with Puppet

Chocolatey is an apt-like package manager for Windows (https://chocolatey.org) that greatly simplifies the installation of software, especially with Puppet (versus having to call MSI packages with obscure switches that may or may not work). Many of my future tutorials that involve managing Windows with Puppet will require that Chocolatey be configured. Here I will explain how I’ve gotten Chocolatey up and running on Windows with Puppet.

This guide assumes that you have Puppet already installed on Windows. If you’re familiar with installing Puppet on Linux systems, it’s about the same for Windows. You would download and install the MSI package from the link here and afterwards sign the certificate request on your master server. You will also need to install the chocolatey/chocolatey and puppetlabs/powershell Forge modules. If you’re using R10K to manage your modules, just add the following to your Puppetfile:

mod 'puppetlabs/powershell'
mod 'chocolatey/chocolatey'

Otherwise, just install them using sudo puppet module install puppetlabs-powershell and sudo puppet module install chocolatey-chocolatey. Once these have been installed, I would then recommend defining some default parameters for the package and file resources at the top scope, in site.pp.

if $::kernel == 'windows' {
  File {
    owner              => undef,
    group              => undef,
    source_permissions => 'ignore',
  }

  Package {
    provider => 'chocolatey',h
  }
}

These tell Puppet not to attempt to apply *nix-style permissions to Windows file resources and to use Chocolatey as the default provider for packages. Now create a class that installs Chocolatey itself. Since the chocolatey/chocolatey module currently is not capable of installing Chocolatey, your class will need to install it using an exec resource. I’ve named my class windows::chocolatey and have created it under windows/manifests/chocolatey.pp.

class windows::chocolatey {

  exec { 'install_chocolatey':
    command  => "set-executionpolicy unrestricted -force -scope process; (iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1')))>\$null 2>&1",
    provider => 'powershell',
    creates  => 'C:/ProgramData/chocolatey',
  }

}

The above command for installing Chocolatey is from Chocolatey installation guide. It’s possible that this may change in the near future. Therefore, you should refer to that page before setting up your exec. If this is for a lab or evaluation environment, you may also want to have Puppet use Chocolatey to keep up to date with the latest Chocolatey release.

  package { 'chocolatey':
    ensure  => latest,
    require => Exec['install_chocolatey'],
  }

Once you have created this module and committed it to the repository containing your custom modules, you would then include the Chocolatey class (windows::chocolatey) in the catalog for your Windows node and initiate a Puppet run on it to apply the class. Now you can use Puppet to manage packages that have been made available by contributors to the Chocolatey project. A full listing can be found here. To manage a particular package with Puppet, include it the same way you would as with a package for Linux:

class windows::git {

  package { 'git':
    ensure => installed,
  }

}

Why You Should Always Test Out New Puppet Forge Modules Before Upgrading

Image

Modules from the Puppet Forge are a great resource. They simplify getting your systems managed under Puppet and save you from having to start from scratch in many cases. I have the utmost gratitude to the individuals who write or contribute to these modules. With that said, the people who contribute to them will sometimes introduce new features that diverge significantly from previous way, causing things to break. Because of this, in my lab environment I set most of the modules in my Puppetfile to ensure that the latest version is installed at all times, so that I can spot any potential any issues. DO NOT DO THIS IN A PRODUCTION ENVIRONMENT, HOWEVER!

mod 'puppetlabs/concat', :latest
mod 'puppetlabs/firewall', :latest
mod 'puppetlabs/git', :latest

Prior to upgrading a version of a Forge module, you should review all of the release notes all the way back to the version you’re upgrading from. It should be also noted you should apply this level of caution to “Puppet Labs-supported” modules as well as those created by community members. One example I recently encountered was with the puppetlabs/puppetdb module. As noted in the change log, the latest version, 5.0.0, has the manage_pg_repo parameter set to true by default, so it will install a version of Postgres from the Postgres Yum repo, while the previous version, 4.3.0, had this off by default, and thus used CentOS’s Postgres package. A demonstration below in a Vagrant box running CentOS 6 shows the result of me blindly upgrading.

First, I create a Vagrantfile that builds the initial environment:

$shell = <<SHELL
puppet module install puppetlabs/puppetdb --version=4.3.0 \
--modulepath=/vagrant/modules
SHELL

Vagrant.configure('2') do |config|
  config.vm.box = 'puppetlabs/centos-6.6-64-puppet'
  
  config.vm.provision "shell", inline: $shell

  config.vm.provision "puppet" do |puppet|
    puppet.module_path = "modules"
  end

end

In my Vagrant project folder, I create a simple manifests/default.pp manifest that just includes the PuppetDB module:

node default {

  include puppetdb

}

Issue a vagrant up and watch while it installs a PuppetDB server running on the OS’s Postgres package. Once complete vagrant ssh in and start up PuppetDB. It should be functioning properly.

Vagrant1

Now I’m going to break PuppetDB with the updated module. First, run sudo puppet module install puppetlabs/puppetdb –modulepath=/etc/puppet/modules, so that it installs the module inside the environment. Then run a sudo puppet apply /vagrant/manifests/default.pp to apply the manifest you created when provisioning the VM.

PuppetDB Module Failures

Guess what? If this were a production system, you’re going to be fixing PuppetDB. The module installed Postgresql 9.4 alongside the OS’s 8.x Postgres, causing a conflict and preventing it from starting. It’s quite apparent that the individuals who put this in did not actually test for this scenario. They probably assume that everyone installing their module would be doing so with a clean system, which is often not the case.

The lesson here: Puppet Forge modules are an excellent resource, but you should research and, if possible, test the changes made in the latest versions before upgrading your production environment. Otherwise, there’s always the possibility you may break something even more critical than PuppetDB.

How to Setup an OpenVPN Server with Puppet

Recently I decided that I wanted to setup an OpenVPN server so that I could access my lab environment at home over an encrypted connection. In order to automate the configuration of this, I decided to use the luxflux/openvpn Forge module. These directions are for a CentOS/Enterprise Linux 6 system.

One thing I noticed in the directions on the GitHub page for the site is that it says that there is a class for the module called openvpn::servers, which supposedly allows you to setup the module using class parameters in Hiera. This class does not actually exist in the current version of the module, which I quickly learned after attempting to add it in my ENC. Instead, I had to create my own module with its own set of parameters that called the openvpn::server defined type to configure my VPN server.

First, I created a module called site_openvpn in my set of custom modules, with a params class that contains the list of parameters to be read in from Hiera.

class site_openvpn::params {

  $vpn_clients = hiera(vpn_clients)
  $vpn_lcl_net = hiera(vpn_lcl_net)
  $vpn_rem_ip = hiera(vpn_rem_ip, $::ipaddress)
  $vpn_subnet = hiera(vpn_subnet)

}

The meaning of the above parameters are:

  • $vpn_clients: this is an array of your VPN client names. Example: test_laptop.
  • $vpn_lcl_net: the subnet of your LAN, such as your home or office network
  • $vpn_rem_ip: the IP your clients will connect to. This may be different than the IP of your VPN server if it resides behind a NAT.
  • $vpn_subnet: the network your VPN clients will connect on.

Create a firewall manifest that opens the necessary ports for OpenVPN:

class site_openvpn::firewall (
$vpn_subnet = $site_openvpn::params::vpn_subnet,
) inherits site_openvpn::params {

  firewall { '120 allow 1194/TCP for OpenVPN':
    state  => 'NEW',
    dport  => '1194',
    proto  => 'tcp',
    action => 'accept',
  }

  firewall { '121 allow TUN connections':
    chain   => 'INPUT',
    proto   => 'all',
    iniface => 'tun+',
    action  => 'accept',
  }

  firewall { '122 forward TUN forward connections':
    chain   => 'FORWARD',
    proto   => 'all',
    iniface => 'tun+',
    action  => 'accept',
  }

  firewall { '123 tun+ to eth0':
    chain    => 'FORWARD',
    proto    => 'all',
    iniface  => 'tun+',
    outiface => 'eth0',
    state    => [ 'ESTABLISHED', 'RELATED' ],
    action   => 'accept',
  }

  firewall { '124 eth0 to tun+':
    chain    => 'FORWARD',
    proto    => 'all',
    iniface  => 'eth0',
    outiface => 'tun+',
    state    => [ 'ESTABLISHED', 'RELATED' ],
    action   => 'accept',
  }

  firewall { '125 POSTROUTING':
    table    => 'nat',
    proto    => 'all',
    chain    => 'POSTROUTING',
    source   => "${vpn_subnet}/24",
    outiface => 'eth0',
    jump     => 'MASQUERADE',
  }

}

Create a init.pp manifest that calls the luxflux/openvpn with your parameters. Note, for dhcp-option DNS, I am using the OpenVPN server itself as a caching/forwarding DNS server, because I’ve run into problems with the DNS traffic forwarding correctly. You can point this to another server, however.

class site_openvpn (
  $vpn_clients = $site_openvpn::params::vpn_clients,
  $vpn_lcl_net = $site_openvpn::params::vpn_lcl_net,
  $vpn_rem_ip  = $site_openvpn::params::vpn_rem_ip,
  $vpn_subnet  = $site_openvpn::params::vpn_subnet,
) inherits site_openvpn::params {

  include ::openvpn
  include site_openvpn::firewall

  ::openvpn::server { $::fqdn:
    country      => 'US',
    province     => 'NC',
    city         => 'Your City',
    organization => $::domain,
    email        => "root@${::domain}",
    server       => "$vpn_subnet 255.255.255.0",
    route        => [
      "${vpn_lcl_net} 255.255.255.0",
    ],
    push         => [
      "route ${vpn_lcl_net} 255.255.255.0",
      "redirect-gateway def1 bypass-dhcp",
      "dhcp-option DNS $::ipaddress_tun0",
    ],
    c2c          => true,
  }

  # Create the VPN client configs
  ::openvpn::client { $vpn_clients:
    server      => $::fqdn,
    remote_host => $vpn_rem_ip,
  }

}

You will also need to set net.ipv4.ip_forward to 1 in /etc/sysctl.conf, so that the traffic can be forwarded through your OpenVPN server to the local LAN. I have a separate class that manages this, but you can include the code in your site_openvpn manifest as well:

sysctl/manifests/init.pp:

class sysctl {

  file { '/etc/sysctl.conf':
    ensure => file,
  }

  exec { 'sysctl -p':
    command     => '/sbin/sysctl -p',
    refreshonly => true,
    subscribe   => File['/etc/sysctl.conf'],
  }

}

sysctl/manifests/ip_forward.pp

class sysctl::ip_forward {

  include sysctl

  augeas { 'sysctl_ip_forward':
    context => '/files/etc/sysctl.conf',
    onlyif  => "get net.ipv4.ip_forward == '0'",
    changes => "set net.ipv4.ip_forward '1'",
    notify  => Exec['sysctl -p'],
  }

}

Finally, create the hiera parameters for the host. I’m creating these in a .yaml file for the host:

---
vpn_rem_ip: '192.168.1.67'
vpn_subnet: '192.168.100.0'
vpn_lcl_net: '192.168.1.0'
vpn_clients:
  - 'iphone6'
  - 'macbook'

Include the site_openvpn and sysctl::ip_forward modules in your node definition or ENC if applicable, and run Puppet on the OpenVPN server. This will install OpenVPN, configure the server, and generate the client configs.

Once installation is complete, the .ovpn client config files will be accessible at /etc/openvpn//download-configs. You can SCP these individual files to import into a Mac (such as Tunnelblick), Windows, Linux, or mobile OpenVPN client.

Author’s note: in my previous post, I forgot to include “proto => ‘all'” in some of the firewall resources, which would cause issues with UDP traffic such as DNS forwarding correctly.