Creating Custom Template in pnp4nagios

Some time need to create custom template in pnp4nagios for some command which behave differently when execute on remote host as the appearance of graphs depends on the check command used. eg check_nrpe

Example:

define command {
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

This would lead to a call of the check_nrpe.php template even when the monitored host would use a completely different plugin which is called via NRPE, eg

In service definition

define service{
use local-service,srv-pnp
host_name SomeHost
service_description Current Load
check_command check_nrpe!check_load
}

This will lead to call to check_nrpe command and it will use the default.php PNP template which may or may not give the graph as required.

Solution: Custom Template

1. Define new command in nagios/commands.cfg

define command{
command_name check_nrpe_load
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

2. Modify the service command for the host to use the new command

define service{
use local-service,srv-pnp
host_name SomeHost
service_description Current Load
check_command check_nrpe_load!check_load
}

Reload Nagios so that it start using the new check command.

At this point everything will be working just as before,

Pnp4nagios configuration

1. Define a check command /usr/local/pnp4nagios/etc/check_commands/check_nrpe_load.cfg (with the same name as the check command in above example)

Tip: Just copy the sample command definition /usr/local/pnp4nagios/etc/check_commands/check_nwstat.cfg-sample eg

cd /usr/local/pnp4nagios/etc/check_commands
sudo cp check_nwstat.cfg-sample check_nrpe_load.cfg

For more detail on custom command refer to http://docs.pnp4nagios.org/pnp-0.6/tpl_custom

2. Define a customer template for check_nrpe_load

In the above example if check_nrpe_load.cfg have

CUSTOM_TEMPLATE = 0,1 the it will results in → "check_nrpe_load_check_load.php"

Create a custom template in /usr/local/pnp4nagios/share/templates (pnp4nagios first look for template in this direcotry then fallback to other directories http://docs.pnp4nagios.org/pnp-0.6/tpl)

Note: Or you can just create a symbolic link to existing template that come with pnp4nagios installation eg ...

ln -s ../templates.dist/check_load.php check_nrpe_load_check_load.php

Here is the sample template for check_load on remote hosts that will plot graph of 1M and 15 minute graph for a host,

cd /usr/local/pnp4nagios/share/templates

sudo vi check_nrpe_load_check_load.php (template name should match with the custom command definition in our example check_nrpe_load_check_load.php)

<?php
$opt[1] = "--vertical-label Load -l0  -u 1 --title \"CPU Load for $hostname / $servicedesc\" ";

$def[1] =  "DEF:var1=$rrdfile:$DS[1]:AVERAGE " ;
$def[1] .= "DEF:var2=$rrdfile:$DS[2]:AVERAGE " ;
$def[1] .= "DEF:var3=$rrdfile:$DS[3]:AVERAGE " ;
if ($WARN[1] != "") {
    $def[1] .= "HRULE:$WARN[1]#FFFF00 ";
}
if ($CRIT[1] != "") {
    $def[1] .= "HRULE:$CRIT[1]#FF0000 ";
}
$def[1] .= "AREA:var1#EACC00:\"Load average  1 min \" " ;
$def[1] .= "GPRINT:var1:LAST:\"%6.2lf last\" " ;
$def[1] .= "GPRINT:var1:AVERAGE:\"%6.2lf avg\" " ;
$def[1] .= "GPRINT:var1:MAX:\"%6.2lf max\\n\" ";
$def[1] .= "LINE:var3#FF0000:\"Load average 15 min \" " ;
$def[1] .= "GPRINT:var3:LAST:\"%6.2lf last\" " ;
$def[1] .= "GPRINT:var3:AVERAGE:\"%6.2lf avg\" " ;
$def[1] .= "GPRINT:var3:MAX:\"%6.2lf max\\n\" " ;
?>

Nagios addon - pnp4nagios

PNP4Nagios is a gem of an add-on; it allows a Nagios administrator to easily add RRD-style graphs and efficient long-term trending ­capabilities to Nagios. This graphing framework makes extensive use of templates and is easy to customize. It consists of a PHP-based front using Kohana framework.

Pnp4nagios uses kohana framework for web frontend and kohana requires php 5.2.x has to be installed. In my previous post I have describe how to upgrade to php 5.2.x on centos which comes with 5.1.x.


System Requirement

Perl >= 5.x
RRDtool
PHP >= 5.2. for the Webfrontend based on Kohana
PHP-extension zlib, session and GD.
Nagios >= 2.x

Installation

Joerg Linge have written excellent documentation for pnp4nagios and using it one can easily install and configure PNP in no time.

In my case I have used the following ./configure line to have rrd data under nagios directories and rest of pnp4nagios goes its own directories in filesystem.

./configure --with-perfdata-logfile=/var/nagios/perfdata.log --with-perfdata-dir=/var/nagios/perfdata --with-perfdata-spool-dir=/var/nagios/spool/perfdata

After installing follow the pnp docs for start generating graphs and web integration with Nagios.

Before going for production use, decide which mode is right for your Nagios installation.

In our case we go with Bulk Mode with NPCD as per docs...

Nagios again uses a temporary file to store the data and executes a command after expiration of a certain time. Instead of immediate processing by process_perfdata.pl the file is moved to a spool directory. As moving a file inside the same filesystem nearly takes no time nagios is able to execute crucial work immediately.
The NPCD daemon (Nagios Performance C Daemon) will monitor the directory for new files and will pass the names to process_perfdata.pl. Processing of performance data is decoupled completely from nagios. NPCD itself is able to start multiple thread for processing the data.

Pages

One of the cool feature of pnp4nagios to have separate pages of graphs for hosts/services on a single page.
Once again Joerg have an excellent docs for configuring pages.

Conclusion

Once installed and configured pnp4nagios provides trending and capacity planning which allows organizations to plan for infrastructure upgrades before outdated systems catch them by surprise.

Picasa 3 sucks

Today after upgrading to Picasa 3 on Fedora 12, it no longer let me upload my photos to web album, each time it gives the same error

Login failed - please try again later.

I have exhausted all the options googling, uninstall and reinstall but no luck.

Come on Google you should be watching your groups on which people crying for the issue of sign in after installing/upgrading to picasa 3.

Adding chunk to dbspace in IBM Informix (IDS) using 'onspaces' utility

Recently I got into problem when there was on space left on dbspace on one of our Informix database. It took me nearly 2 hours of Googling to figure out how to add CHUNK to existing dbspace.

I thinks it would be better to blog it, this might help some other folks or might I need to use it some day when the newly added CHUNK also get full :)

For my setup I do not have the additional hard drive and have a plenty of free space on Unix partition. Therefore I used the unix file as new Chunk.

'onmonitor', oncheck -pe, onstat -d commands can be used to view the status of dbspaces.

Here are the steps to add chunk.

1. First create a chunk file in Unix filesystem and set the permission so that informix user/group have the read/write permission on the file.


touch /export/home/dbspace/billdb_dg/lv_recorddbs11

chmod 660 /export/home/dbspace/billdb_dg/lv_recorddbs11

Note: I run the above commands with 'informix' user or the owner of informix db.

The following example adds a 2GB chunk to recordbs. An offset of 200 kilobytes is specified.

onspaces -a recorddbs -p /export/home/dbspace/billdb_dg/lv_recorddbs11 -o 200 -s 2093150

Here -s 2093150 is specified in KB

The newly added chunk is available immediately.

After you create the chunk, you must perform a level-0 backup of the root dbspace and the dbspace that contains the chunk.

Chat with Joerg Linge developer/maintainer of pnp4nagios

Today I got an opportunity to chat with developer/maintainer Joerg Linge of pnp4nagios.

Chat was mostly about the new feature of using multiple Data Sources for a service in pnp4nagos.

RRD_STORAE_TYPE = SINGLE means: Store every Datasource in one single RRD Database i.e. for check_icmp this is DS 'pl' and DS 'rta'

With RRD_STORAE_TYPE = MULTILPLE every DS is stored in its own RRD Database i.e. host/service_pl.rrd and host/service_rta.rrd

RRD_STORAE_TYPE = MULTILPLE should only used when a plugin is able to change the number of DS like check_disk does.

Me:  Is it ok to set  RRD_STORAE_TYPE = MULTIPLE globally in process_perfdata.cfg for new installation of Joerg? when one do not have any RRD data at all.
Me:  tomorrow i am going to deploy Joerg 0.6.x on our production nagios server.
Me:  thanks.
Joerg : it depends
Joerg : happy new year!
Me:  happy new year :)
Me:  ok,
Me:  going with MULTIPLE has no drawback ok?
Joerg : multiple mean more Disk I/O because of more RRDs to update

Joerg : on sytems with 10000+ rrds you have to look on your disk I/O clisely
Me:  well this really need very close monitoring of I/O which is hard to follow :s
Me:  better stick with SINGLE :)
Joerg : right
Joerg : there are some plugins with changing DS counts like check_disk
Joerg : in this case multiple is an option
Joerg : or check_oracle_health for tablespace monitoring
Me:  we are using check_disk extensively.
Me:  for checking servers partitions.
Joerg : one check for all filesystems?
Me:  well mostly separate check_disk for each fs, however we have some servers with lot of fs onthese servers we are using single check_disk to check multiple fs in one go.
Me:  with -p option
Me:  those are mostly HP-UX
Joerg : so check_disk can be used with multiple
Me:  ok to use multiple have to configure it globally?
Joerg : wenn you add a nw filesystem, rrdtool is not able to update the rrd databases
Joerg : lol german wenn
Joerg : english when
Joerg : sorry
Me:  np, i thought its typo :)
Me:  ok to use multiple have to configure it globally? or their is another way around?
Joerg : moment
Joerg : http://docs.Joerg.org/pnp-0.6/tpl_custom
Me:  looking.
Joerg : you can set RRD_STORAGE_TYPE per check_commands
Joerg : i.e. your command is check_disk
Joerg : create etc/check_commands/check_disk.cfg
Joerg : set RRD_STORAGE_TYPE = MULTIPLE within this file
Joerg : thats the same rrd_convert.pl --check_command=check_disk will do in its final version
Me:  ok but for most host we are doing the check_disk on remote host and then using check_nrpe on nagios host
Joerg : so your command is?
Joerg : check_nrpe?
Me:  yes
Joerg : create a new command nrpe_disk
Joerg : thats it
Me:  can you give me full example of nrpe_disk here?
Joerg : define command {
Joerg : command_name nrpe_disk
Joerg : command line $USER1$/check_nrpe -H $HOSTqADDRESS$ -c check_diks1
Joerg : }
Joerg : so your command is nrpe_disk and not check_nrpe
Joerg : and now you are able to define MULTIPLE for this command
Joerg : this is done by creating Joerg/etc/check_commands/nrpe_disk.cfg
Me: Thanks for your time very much appreciated.