Wednesday, December 22, 2010

Receiving SNMP Traps in Nagios

SNMP traps are alerts and notifications generated by SNMP-enabled devices. The traps con-tain information about the status or an event on an SNMP-enabled device. For example, an authentication event or the change in status of an interface on a router may generate an SNMP trap that is sent to a management station of some sort, such as HP OpenView, CiscoWorks, Nagios.

Pre-requisites:

1. Net-SNMP with snmptrapd configured.
2. SNMPTT, SNMP trap translator.
3. Nagios.
5. Mib definition files for the equipment or software you need to monitor.

Installing Net-SNMP packages:

The Net-SNMP package is available as a series of installable packages on many distributions. Indeed, it may already be installed on your system or you may be able to install it via your distribution’s package management system, such as yum, apt, or the like. On Red Hat, SuSE, Debian, and Mandrake
distributions, the required packages are called net-snmp, net-snmp-libs, and net-snmp-utils.

Installing Net-SNMP packages on Centos 5.5

# yum install net-snmp net-snmp-libs net-snmp-utils net-snmp-perl perl-Net-SNMP net-snmp-devel

Configuring and Running the snmptrapd Daemon
When incoming traps are received from the snmptrapd daemon, they are passed to the SNMPTT tool. The SNMPTT tool will then try to match the incoming trap against the collection of trap definitions that it has translated. If the trap matches, SNMPTT will see if the translated trap definition contains logic to output it to Nagios and execute that logic. The trap is then out-put to Nagios as a passive check result.

On Centos 5.5

# vi /etc/sysconfig/snmptrapd.options
#OPTIONS="-On -Lsd -c /etc/snmp/snmptrapd.conf -p /var/run/snmptrapd.pid"
OPTIONS="-On -Lsd -p /var/run/snmptrapd.pid"

Make sure to remove the -c /etc/snmp/snmptrapd.conf part, otherwise you will receive TRAP twice, as snmptrapd' is compiled with the default configuration file path being already set to '/etc/snmp/snmptrapd.conf'.

As quoted from SNMP Trap Translator documentation: "The -On is recommended. This will make snmptrapd pass OIDs in numeric form and prevent SNMPTT from having to translate the symbolic name to numerical form."

# vi /etc/snmp/snmptrapd.conf
traphandle default /usr/sbin/snmptthandler
disableAuthorization yes
#donotlogtraps  yes

The traphandle directive tells the snmptrapd daemon how to handle incoming traps and where to send them. Adding the default option tells the daemon that this is the default way to handle all incoming traps. All traps will be sent to the snmptthandler script located in the /usr/sbin directory, the "disableAuthorization yes" tells to accept SNMP traps from all you can configure it to do authentication for detail refer to snmptrapd.conf manual.

Installing SNMPTT (SNMP Trap Translator)
You can get the SNMPTT tool from Sourceforge at http://snmptt.sourceforge.net/. This line shows how to download and unpack the SNMPTT tool:
Download snmptt_1.3.tgz which the latest version/stable release.

tar -zxvf snmptt_1.3.tgz

The SNMPTT package has no installation script, so a number of manual installations steps need to take place. First, copy the SNMPTT binaries to a suitable directory and mark them as executable. I recommend using the /usr/sbin directory

# cp snmptt snmptthandler /usr/sbin/
# chmod +x /usr/sbin/snmptt /usr/sbin/snmptthandler

I specified the snmptthandler binary as the value of the traphandle option in the snmptrapd.conf configuration file in the previous section. When a trap is received, this binary is executed by default and the trap sent to the snmptt daemon

Next, copy the SNMPTT configuration file, snmptt.ini, to the /etc/snmp directory and snmpttconvertmib utility

# cp snmptt.ini /etc/snmp/
# cp snmpttconvertmib /usr/sbin/

Also needed are a user and group to run the SNMPTT daemon as.

# groupadd snmptt
# adduser -g snmptt snmptt

# chown snmptt:snmptt /etc/snmp/snmptt.ini

The SNMPTT tool also needs a spool directory to hold the incoming traps. I usually use the default directory of /var/spool/snmptt. It needs to be owned by the user and group that will run SNMPTT. Create and change the ownership of the directory like so

# mkdir /var/spool/snmptt
# chown snmptt:snmptt /var/spool/snmptt

Finally, in order to start the SNMPTT tool, you can either execute it from the command line or use the init script provided with the package. On the following line

SNMPTT started in daemon mode:
# /usr/sbin/snmptt -daemon

Or copy the init script provided with the package, you can then add it to your startup process.

# cp snmptt-init.d /etc/init.d/snmptt

To start/stop/reload you can do with ..

/etc/init.d/snmptt start/stop/reload

Configuring SNMPTT
The first is configuring the /etc/snmp/snmptt.ini file. The file contains quite a large number of directives, but I’ll only look at those relevant to the process of translating and transmitting the received traps to Nagios

mode = daemon
daemon_fork = 1
daemon_uid = snmptt
spool_directory = /var/spool/snmptt/
sleep = 5
dns_enable = 1
strip_domain = 1
log_enable = 1
syslog_enable = 0
exec_enable = 1
snmptt_conf_files = <
/etc/snmp/snmptt.conf
END


The sample snmptt.ini file contained in the SNMPTT package has detailed explanations of all the directives and options that you can specify. I recommend reading this file for further information and explanations about SNMPTT’s configuration options

Tip:  If you enable DNS resolution, I recommend you add all the hostnames that need to be resolved to the local /etc/hosts file on your host server. This prevents your DNS server from being a bottleneck or preventing SNMPTT from functioning if your DNS server is unavailable.

Compiling MIBs
You must gather all MIBs for monitored software, so you can feed SNMPTT with them. Compiling consists in extracting each OID of type "trap" and its associated comments, and generate a configuration file in SNMPTT format from these informations.

Run the following command on each of your MIB files:

snmpttconvertmib --in= --out=/etc/snmp/snmptt.conf. \
--exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1'

The resulting SNMPTT configuration file will contain blocks (one per selected OID)


Catchall Trap Definition
SNMPTT also has a regular expression–matching capability that allows you to use an EVENT line that matches multiple incoming traps, a catchall trap definition. This means you don’t need to define individual translated trap definitions for each possible incoming trap.

Catchall Trap Definition
EVENT CatchAll .1.* "SNMP Traps" Critical
FORMAT $D
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result "$r" 
"snmp_traps" 2 "$O: $1 $2 $3 $4 $5"

I could also be more selective and select OIDs from a particular vendor or class of trap either using a wildcard or regular expression pattern matching. I’ve added a category called SNMP Traps and severity of Warning.

For example here is example to catchall traps from a specific vendor OID.

EVENT CatchAll .1.3.6.1.4.1.20916.* "Status Events" Normal
FORMAT A room-alert-4e-snmp-trap indicates that an alarm $*
EXEC /usr/lib/nagios/plugins/eventhandlers/submit_check_result $r "snmp_traps" 1 "A room-alert-4e-snmp-trap indicates that an alarm $*"
SDESC
A room-alert-4e-snmp-trap indicates that an alarm
condition has occurred on the sensor indicated
by the alarmmessage variable.
Variables:
  1: alarmmessage
EDESC

When done, add to SNMPTT configuration file /etc/snmp/snmptt.ini the path to compiled configuration files:

[...]
snmptt_conf_files = <
/etc/snmp/snmptt.conf.
/etc/snmp/snmptt.conf.
END


Configuring Nagios
You will use passive checks to receive SNMP traps but they also will be volatiles. If ever two traps are received from the same host, the second one coming in before the first one was reset to OK, we want to be notified twice, although there is no state change. That's why we use a volatile service.

You might define (for example) a service template for SNMP traps, inheriting from a generic service template:

define service{
        name                            generic-service    
        active_checks_enabled           1                     
        passive_checks_enabled          1                       ; Passive service checks are enabled/accepted
        parallelize_check               1                     
        obsess_over_service             1                      
        check_freshness                 0                      
        notifications_enabled           1                       ; Service notifications are enabled
        event_handler_enabled           1                       ; Service event handler is enabled
        flap_detection_enabled          1                       ; Flap detection is enabled
        failure_prediction_enabled      1                       ; Failure prediction is enabled
        process_perf_data               1                       ; Process performance data
        retain_status_information       1                       ; Retain status information across program restarts
        retain_nonstatus_information    1                       ; Retain non-status information across program restarts
        is_volatile                     0                       ; The service is not volatile
        check_period                    24x7                    ; The service can be checked at any time of the day
        max_check_attempts              3                       ; Re-check the service up to 3 times in order to determine its final (hard) state
        normal_check_interval           10                      ; Check the service every 10 minutes under normal conditions
        retry_check_interval            2                       ; Re-check the service every two minutes until a hard state can be determined
        contact_groups                  admins                  ; Notifications get sent out to everyone in the 'admins' group
        notification_options            w,u,c,r                 ; Send notifications about warning, unknown, critical, and recovery events
        notification_interval           60                      ; Re-notify about service problems every hour
        notification_period             24x7                    ; Notifications can be sent out at any time
         register                        0                      ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!
        }

define service{
name                    trap-service
use                     generic-service
register                0
service_description     snmp_traps
is_volatile             1
check_command           check-host-alive    ;Used to reset the status to OK when 'Schedule an immediate check of this service' is selected.
flap_detection_enabled  0                               ; Flap detection is disabled
process_perf_data       0                               ; Do not Process performance data
max_check_attempts      1                    ; Leave as 1
normal_check_interval   1                    ; Leave as 1
retry_check_interval    1                    ; Leave as 1
passive_checks_enabled  1                    ; Enables passive checks
check_period            24x7
notification_interval   31536000                ; Notification interval.  Set to a very high number to prevent you from getting                                 pages of previously received traps (1 year - restart Nagios at least once a year! -                                     do not set to 0!).
active_checks_enabled   0                    ; Prevent active checks from occuring as we are only using passive checks.
notification_options    w,u,c                    ; Notify on warning, unknown and critical.
contact_groups          sysadmins
}

define service{
 host_name       AVT-Room-Alert ; hostname is define /etc/hosts file
 use             trap-service
 contact_groups sysadmins
}

TIP: You could also use a wildcard to create this service for all hosts or use the hostgroup_name directive to create the service for all members of a host group or groups.

I’ve defined the service as volatile and set the maximum check attempts to 1. This will cause Nagios to immediately set a HARD service state and trigger any configured notifications or event handlers. I’ve also configured it for passive checks only and disabled active checks.

Putting It All Together
The SNMPTT tool is called via the trap handler defined in the snmptrapd.conf configuration file I defined in the “Configuring and Running the snmptrapd Daemon” section. This trap handler calls the /usr/sbin/snmptthandler script. The script reads the trap and then writes it to the spool directory defined in the spool_directory directive from the snmptt.ini configuration file. The script then exits.

From here the SNMPTT daemon takes over. It reads the trap from the spool file and searches for a match in its trap definitions. If it finds a match, it executes the EXEC statement in the matching trap definition. This EXEC statement sends the passive check result to the Nagios server using the submit_check_result script. The daemon then sleeps for the period specified in the sleep directive in the snmptt.ini file and checks the spool directory for additional traps; if it finds matches, it processes them and sends the check results to Nagios.

The Nagios server has to have host objects defined for every host that generates SNMP traps. Additionally, you need to define service objects for those hosts to receive the service check results. You should configure them to receive passive check results and as volatile services.

Friday, October 29, 2010

Fixing Oracle SQL Developer on 64 bit Fedora 13

If getting error while trying to start SQL Developer on 64 bit Fedora

java.lang.UnsatisfiedLinkError: /usr/java/jdk1.6.0_07/jre/lib/i386/xawt/libmawt.so: libXtst.so.6: cannot open shared object file: No such file or directory

The installed libXtst is 64 bit and sqldeveloper will not work until 32bit version of the same package is installed.

Installing 32 bit package on Fedora

sudo yum install packagename.i686

Now to fix the Oracle SQL Developer problem install the 32 bit version of  libXtst

  sudo yum install libXtst.i686

Monday, August 30, 2010

Cache Only Local DNS Server using dnsmasq

dnsmasq is the most popular caching DNS choice for small-to-medium deployments is which is both a lightweight DNS server and a lightweight DHCP server. It is a standard package in modern Linux distributions. If your distro doesn't provide it, though, you can download and install the source without too much trouble; there are no major external package dependencies.

Installing dnsmasq on Fedora

# yum install dnsmasq

Configuration

Open the file /etc/dnsmasq.conf as root. There are several dozen configuration options -- all of them well-documented -- but it only takes a few to get set up for normal use.  Near the top, uncomment out the listen-address= directive and add the localhost address 127.0.0.1: listen-address=127.0.0.1. This tells dnsmasq to listen for DNS queries that originate on the local machine.
 

listen-address=127.0.0.1 
 
Edit /etc/resolv.conf, and add nameserver 127.0.0.1 to the top. The locally-running copy of dnsmasq will now receive all DNS queries first, and cache the results.


Starting dnsmasq Service
# /etc/init.d/dnsmasq start


Saturday, August 28, 2010

No Video with mplayer on Fedora 13

After installing Mplayer on Fedora 13 I was facing the problem of no video which was fixed after running the mplayer from command line with the following option

gmplayer -vo xv filename.avi

After running it from command line it start working fine.

Friday, June 18, 2010

Installing Google Command-Line Tool for Linux

Ever wanted to upload a folder full of photos to Picasa from a command prompt?' Google introduced today a new project, Google CL, that lets you do that and much more. It's a new command line tool for Linux that acts as an interface with Google services; you can upload videos to YouTube or maybe post a new blog post in Blogger in just one line


Installing Google CL:

DEPENDENCIES

GoogleCL requires Python 2.5 or 2.6 and the gdata python client library. You can get the library from the project homepage: http://code.google.com/p/gdata-python-client/

1. Download gdata-2.0.10.tar.gz or .zip
2. extract the source tar zxvf gdata-2.0.10.tar.gz
3. cd gdata-2.0.10
4. Run sudo ./setup.py install

I got this error...

/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'install_requires'
warnings.warn(msg)
running install
error: invalid Python installation: unable to open /usr/lib/python2.6/config/Makefile (No such file or directory)

In case of above error install python-setuptools-devel which will also install python-devel as dependency,

sudo yum install python-setuptools-devel

and do the step (4) again

Running Tests and Samples

./tests/run_data_tests.py

If this script runs correctly, you should see output on the command line like this:

Running all tests in module gdata_test
.....................
----------------------------------------------------------------------
Ran 21 tests in 0.028s

OK

Running all tests in module atom_test
...............................................
----------------------------------------------------------------------
Ran 47 tests in 0.038s

OK

....

Finally get the Google CL source from http://code.google.com/p/googlecl/

1. Extract the source
tar xvf googlecl-0.9.5.tar.gz

2. change to googlecl-0.9.5 directory and run
sudo python setup.py install

Now you are ready to play with Google CL and starting posting blogs, videos to youtube etc right from your Linux command Line,

One first run for every google service you will get ...

Did not find config / preferences file at /home/myhome/.googlecl/config
... making new one.
Please specify user: myusername
(Hint: You can automatically launch your browser by adding "auth_browser = " to your config file under the GENERAL section, or define the BROWSER environment variable.)
Please log in and/or grant access via your browser at https://www.google.com/accounts/OAuthAuthorizeToken?oauth_token=4%2FglE62XHHpOLmP6jhGGZumt6Tm2re then hit enter.


Specify the username and put the following in the ~/yourhome/.googlecl/config GENERAL section.

auth_browser = firefox

And when browser open Auth Token page click "Grant Acces"

Examples:

1. Post a blog
$ google blogger post --tags "GoogleCL, awesome" --title "Test Post" "I'm posting from the command line"

2. To post a blog from .txt file

$ google blogger post ~/Desktop/googlecl.txt

3. To delete a post
google blogger delete --title "Test Post"
Are you SURE you want to delete post "Test Post"? (y/N): y

For detail examples refer to GooglCL project web

Friday, June 11, 2010

Installing Nagios-plugins and NRPE on Centos 5.x

To monitor remote hosts CPU load, disk partitions, processes etc with Nagios, requires to install NRPE and nagios-plugins on the remote host.

This is how I do it...

Nagios-plugins and NRPE is not available from Centos official repositories, so first of all we need to configure RPMforge> repo from where to install the required packages.

Read my previous article before configuring and installing anything from third party repo.

Installing RPMforge on Centos 5.x

Download the rpmforge-release package. Choose one of the two links below, selecting to match your host's architecture. If you are unsure of which one to use you can check your architecture with the command uname -i

x386
wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.i386.rpm

x86_64
wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm

Install the RPMForget key

rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt

If key didn't install no worries yum will ask when installing a package from rpmforge repo.

Verify the package you have downloaded
# rpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpm

Install the package
# rpm -ivh rpmforge-release-0.5.1-1.el5.rf.*.rpm

Now everything is setup now you can install nagios-plugins and nrpe packages from RPMforge repo

# yum install nagios-plugins nagios-nrpe

Running NRPE under xinetd

Edit /etc/xinetd.d/nrpe for

disable         = no
only_from       = 127.0.0.1 NagiosServerIP

Restart xinetd service

# /etc/init.d/xinetd restart

Before using third party repositories on Centos

Before configuring and installing anything from third party repo, it is recommended to make sure to use Priorities by installing yum-priorities package for your version of Centos

CentOS-4 or CentOS-6:
# yum install yum-plugin-priorities

CentOS-5:
# yum install yum-priorities

After the plugin is installed, make sure that it is enabled when you decide to use a given archive. You can do this by editing the /etc/yum/pluginconf.d/priorities.conf file, and ensuring that it contains the following lines:

[main]
enabled=1
check_obsoletes=1

Set the priorities of Repositories

With the plugin enabled, you may add priorities to repositories by adding the line:

priority=N

where N is an integer from 1 to 99. The default priority for repositories is 99. The repositories with the lowest numerical priority number have the highest priority. Usually, it is best to give at least the CentOS base and update repositories a very high priority.

Give CentOS base,update,addons,extras repositories the highest possible priority (1):

vi /etc/yum.repos.d/CentOS-Base.repo
priority=1

And give centosplus, and contrib priority 2
priority=2

Give low priority = 10 to all the third party repositories by setting repos files in /etc/yum.conf.d directory

For detail refer to the Centos wiki

Thursday, June 10, 2010

Upgrading to Fedora 13 with Preupgrade

Preupgrade is an application users run on a existing release, that resolves and downloads packages required to upgrade to a newer release of Fedora. While Preupgrade downloads the necessary packages, users are free to continue using their systems. This gives an experience similar to a live upgrade. For additional information

While preupgrade provides a generally hassle free upgrade experience. The following steps are recommended before proceeding.

   1. Backup - Before performing any system maintenance, it is recommended that you back up all important data before proceeding.
   2. Update - Apply available updates Fedora updates before proceeding. As the root user, issue the following command.

# yum update

2. If preupgrade package not already install then install it.

# yum install preupgrade

Perform the upgrade

1. As the root user, run the command preupgrade to start the Preupgrade application. If you prefer a command line application, the command preupgrade-cli is available.

2. On the Choose desired release screen, select the Fedora release (in my case my home laptop running Fedora 12, I have only one option ie Fedora 13) you want to upgrade to , and click the Apply button.

# preupgrade -cli

Checking for new repos for mirrors
 * preupgrade-updates: mirror.yandex.ru
Downloading 947.6MB
Available disk space for /var/cache/yum/preupgrade: 2.8GB


relax and do your work while packages being downloaded which will take quit sometime, depending on your Internet connection.

3. When all of the packages have downloaded, reboot your system to start the Fedora installer and upgrade to the next release.



Common post-upgrade tasks.

Refer to the Fedora wiki

Troubleshooting

Not enough space in /boot

Fedora 13 and above has a 500 MB default boot partition. The default /boot filesystem size of 200MB for previous releases can be a problem for users upgrading from that release.

Follow the instructions on Fedora wiki to handle the space problem on /boot partition.

Note: If preupgrade give you message "Not enough space in /boot/upgrade to download install.img." but it can download it after reboot if you have a wired connection to the network, you can click continue for that.


VLC not working after upgrade to Fedora 13.

Fix: If you are getting the invalid pointer error, try running VLC the first time with:

MALLOC_CHECK_=1 vlc

After setting my preferences and quitting, I was then able to run it normally on F13 without issue.

VirtualBox: Networking between Linux Host and Windows XP Guest

In this article we will learn how to configure networking between Linux host OS and Windows Guest (VM) using Virtualbox

1. First of all download and install Virtualbox on your Linux host and install the guest operating system

2. Now configure network on Guest OS, open VM settings and click Network

3. Select Host-Only Adapter from "Attached to" combo



The VM will create the default network interface named by "vboxnet0" on the host OS,

#ifconfig
vboxnet0  Link encap:Ethernet  HWaddr 0A:00:27:00:00:00 
          inet addr:192.168.56.1  Bcast:192.168.56.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3599 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:3896747 (3.7 MiB)

VirtualBox automatically assigned IP address 192.168.56.1 on the host to that network interface.

4. Launch the XP guest OS from virtualbox and configured the network upon VM successfully boot

 5. Make sure both Host and guest can responding to each other 'ping'
ping 192.168.56.10
PING 192.168.56.10 (192.168.56.10) 56(84) bytes of data.
64 bytes from 192.168.56.10: icmp_seq=1 ttl=128 time=1.83 ms
64 bytes from 192.168.56.10: icmp_seq=2 ttl=128 time=0.512 ms

Enabling Internet for VM.

Above steps was fine as far as networking between Host and VM is concerned, however if you want to let VM also access the Internet then you have to configure the following iptables rules and enable ipfowarding on Host OS.

# insert NAT rule
iptables -t nat -A POSTROUTING -o vboxnet0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# enable forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

Saturday, June 5, 2010

CountDown to 2010 FIFA World Cup South Africa

I wonders who will miss this amazing event :), I wish I could watch all the matches of this great event.

Italy will be defending its title which they  won  in the 2006 FWC in Germany against France.

Like all previous Football world cup Brazil will again my favourite team, Brazil is the only team which have qualified for all known FWC and also holding a record of wining FWC 5 times.

Wednesday, May 26, 2010

Face-Off with Facebook By Irfan Husain (The Dawn)

Recently Pakistan Government went on blocking Facebook (social networking) the site carrying the blasphemous drawings of the Holy Prophet [PBUH].

Read the full article here.

Fedora 13 Is Out

Thanks to Fedora folks for releasing yet another release of one of the best Linux distro.

Fedora 13 includes some major features like automatic print driver installation, automatic language pack installation, redesigned user account tool, color management to calibrate monitors and scanners, experimental 3D support for NVIDIA video cards, and more in the release notes.


Currently I have Fedora 12 installed which will be supported for quit sometime (almost six months) so there is no need to rush for new release yet.

Whenever I need to upgrade I'll decide to use one of the following tool/option for upgrading my home and work laptops.

Preupgrade
Using boot images available from http://boot.fedoraproject.org/

Though last time upgrading from Fedora 11 'preupgrade' work like a charm for me, so there is a chance that this time I'll also go with preupgrade.

Friday, March 19, 2010

Vnstat a console based network traffic monitor

vnStat is a console-based network traffic monitor for Linux and BSD that keeps a log of network traffic for the selected interface(s). It uses the network interface statistics provided by the kernel as information source. This means that vnStat won't actually be sniffing any traffic and also ensures light use of system resources.

The beautiy of Vnstat is that its very simple to install and configure

Installing vnstat on Fedora

Thanks to Fedora folks vnstat can now be installed from Fedora repository using 'yum', please refer to your distro package management commands for installing vnstat on other distros.

yum install vnstat

Monitoring Interface traffic


The commmand forces a database update for interface or creates the database if it doesn’t exist. This is usually the  first  command used after a fresh install.

vnstat  -u  -i Interface

Replace with the Interface name eg, eth0, wlan0 etc

Displaying statistics:

After installation and creating a interface database vnstat can be used from command line to display network traffic statistics

$ vnstat --help

vnStat 1.6 by Teemu Toivola

         -q,  --query          query database
         -h,  --hours          show hours
         -d,  --days           show days
         -m,  --months         show months
         -w,  --weeks          show weeks
         -t,  --top10          show top10
         -s,  --short          use short output
         -u,  --update         update database
         -i,  --iface          select interface (default: eth0)
         -?,  --help           short help
         -v,  --version        show version
         -tr, --traffic        calculate traffic
         -l,  --live           show transfer rate in real time

Tuesday, March 2, 2010

Sending SMS Notifications From Nagios

In my last article I have discuses how to install Gnokii for sending/receiving SMS from your computer. Today I'll explain how we are using Gnokii + Nagios for sending SMS notifications to our cell phones. Its a great way to get notify of the problems while on road.

I assume that you have working Nagios and its monitoring the devices in your infrastructure and sending notifications via Email and you are looking how to get these problem notifications on your phones.

Gnokii is also working and you can send SMS from CLI.

Lets cut it short and back to actual business.

In my setup we have Nagios and Gnokii install on same host running Centos 5.4, but it can easily be followed for any other Linux distro or even with setup where Gnokii is install on separate host.

1. Make sure you can send SMS from CLI with "gnokii --sendsms +92xxxxx" using root or the user under which Nagios process is running normally its 'nagios' user, sending under nagios user requires to add nagios to groups which have permission to access the device files.

a) So add nagios to 'uucp' group (you can do this with usermod command)

Gnokii also acquire a lock under /var/lock

b) So add nagios user to 'lock' group also.

su to nagios user and send sms from CLI using gnokii --sendsms, when it works move forward for defining commands.

2. Define command for send notification via SMS in commands.cfg

# 'notify-service-by-sms' command definition
define command{
command_name notify-service-by-sms
command_line /usr/bin/printf "%.120s" "*** Nagios Alert*** $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$" | /usr/local/bin/gnokii --sendsms $CONTACTPAGER$
}

# 'notify-host-by-sms' command definition
define command{
command_name notify-host-by-sms
command_line /usr/bin/printf "%.120s" "*** Nagios Alert*** $NOTIFICATIONTYPE$ : Host $HOSTALIAS$ is $HOSTSTATE$" | /usr/local/bin/gnokii --sendsms $CONTACTPAGER$
}

3. Modify contacts.cfg and add or modify a contact by calling new commands

define contact{
contact_name askarali
use generic-contact
alias Askar Ali Khan
email emailaddress
pager +92xxxxxx
service_notification_commands notify-service-by-email,notify-service-by-sms
host_notification_commands notify-host-by-email,notify-host-by-sms
}

The key in the contact detail is the service/host notifications commands

service_notification_commands notify-service-by-email,notify-service-by-sms
host_notification_commands notify-host-by-email,notify-host-by-sms

I have configured a contact so that he can receive notifications via Email 'notify-service-by-email' as well as via SMS 'notify-service-by-sms'


That's all, finally reload nagios, before reload better to run syntax check

'nagios -v PathTo nagios.cfg'

and then reload

/etc/init.d/nagios reload

Now Nagios will send SMS notifications on your phone whenever there is problem with any host/service which being monitor with Nagios.

I hope this could help.

Bash Scripting - Adding pipe "|" to each line of text file

Here is a simple script which will add a pipe symbol '|' to the start and end of every line in a text file.

Well do not ask why I need to do this, it was needed for some nasty text files which we have to deal with in our day to day works.

Someone will argue why the hell we need a script for this simple task when we can do the same with 'vi', well I must say chill mate...yes there are more then one way to do the work done, and using the script way you don't have to open a file or what if you have to work with more then one txt file? its easy to run a script "add_pipe" and sit back instead of vi file1;vi file2. :)

Script can be easily modified to accept the "SOURCEDIR" from command line or adding any other symbol, currently I put all such text files in /tmp/manual before running the script.

The Script:

#!/bin/sh
# Purpose: To add pipe '|' to begin and end of line in a txt file.
# Date: 2010-02-25
# add_pipes

# Directory to where files need to process for adding pipe
SOURCEDIR="/tmp/manual"

# Change to source directory
if [ -d "$SOURCEDIR" ];then
 cd "$SOURCEDIR"
else
 exit 1
fi

# First convert the files to Unix (if imported from M$ files carries the carrige return
CR='\015'  # Carriage return.
           # 015 is octal ASCII code for CR.
           # Lines in a DOS text file end in CR-LF.
           # Lines in a UNIX text file end in LF only.

for file in $(ls)
do
 if [ ! -d "$file" ];then
  tr -d $CR < $file >newname
  mv newname $file
 fi
done

# Finally add pipe by loop through files
for file in $(ls)
do
 if [ ! -d "$file" ];then
  #sed -e 's/^/|/g' -e 's/\r$/|/g' < $file >newname
  sed -e 's/^/|/g' -e 's/$/|/g' < $file >newname
  mv newname $file
 fi
done
# End

Friday, February 19, 2010

Installing and configuring caching nameserver (named) on Linux (Fedora 12)

To configure caching-nameserver on local machine in my case on my laptop running Fedora 12, local caching nameserver which will use "cascade" to other functional DNS server in this case my ISP DNS servers.

Advantage: Reduces the delay in domain name resolution drastically as the requests for frequently accessed websites are served from cache. Google for cache nameserver to learn more.

Installing caching-nameserver:

# yum install caching-nameserver

Configuring caching-nameserver:

The main configuration file reside in /etc/named.conf

Not much need to be changed in this file, however if you want to cascade your ISP DNS servers as forwarder, then edit the named.conf and add forwarder directive under the 'options' section.

options {
        listen-on port 53 { 127.0.0.1; };
        listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-query     { localhost; };
        recursion yes;
    // Replace the IPs with the DNS of your ISP
    forwarders { 192.168.36.204; 192.168.36.210; };
        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside . trust-anchor dlv.isc.org.;
};

And if you do not want to use 'dnssec' then disable it by...

dnssec-enable no;
dnssec-validation no;


Starting caching-nameserver:

# service named start

Or

# /etc/init.d/named start

To make named start every time you reboot your machine

# chkconfig named on

Or

# setup


Go to "System services" and check named

Using caching-nameserver:

To use your caching-nameserver, add the following line to /etc/resolv.conf

nameserver 127.0.0.1

Now your system will use your own nameserver (in caching mode) for resolving all domain names

$ dig askarali.org
Query time: 8 msec

First time the response time will be little high, next time the DNS query response will be served from local cache and will takes very little time

$ dig askarali.org
$ Query time: 0 msec

Friday, February 12, 2010

Sending SNMP Traps from Nagios

As well as receiving SNMP traps in Nagios, you can send SNMP traps from Nagios to a remote SNMP management station like HP OpenView (NNM) or the like. The easiest way to do this is to create a notification command that generates an SNMP trap. This way, you can use an SNMP management station as a notification destination. This will mean that the trap will be sent every time a notification is scheduled.


Prerequisites:

net-snmp, net-snmp-utils should be installed on Monitoring host running Nagios

Connectivity between management station and nagios has been stablished and nagios can connect and send snmptraps to snmptrap 162/udp port on management station.


1. First of all define notification command, below is the commands for service and hosts in commands.cfg

# 'send-service-trap' command definition
define command{
      command_name send-service-trap
      command_line /usr/local/bin/send-service-trap manager public "$HOSTNAME$" "$SERVICEDESC$" $SERVICESTATEID$ "$SERVICEOUTPUT$"
}

# 'send-host-trap' command definition
define command{
                command_name send-host-trap
                command_line /usr/local/bin/send-host-trap manager public "$HOSTNAME$" $HOSTSTATEID$ "$HOSTOUTPUT$"
}

Both the commands call the shell scripts send-service-trap and send-host-trap in /usr/local/bin/. Passing the hostname or IP address of the destination management station and the target community string, manager and public, respectively.

A number of macros are also passed to the shell script representing the hostname, the service description, the service/host state in numeric form, and the output of the host/service check.

The snmptrap tool generates SNMP traps and can send them to remote management stations. The traps are generated according to a Nagios
MIB available from the same Sourceforge project as the Nagios plug-ins. You can download the MIB from http://prdownloads.sourceforge.net/nagiosplug/nagiosmib-1.0.0.tar.gz?download.

The package contains two MIB files: a root MIB file containing basic definitions for Nagios OIDs called NAGIOS-ROOT-MIB and an MIB containing events and traps called NAGIOS-NOTIFY-MIB. You will need to install both MIB files into your SNMP management device; for example, if your trap daemon is the snmptrapd daemon, you would generally copy these files to the /usr/share/snmp/mibs directory. When you start the snmptrapd daemon, you load the MIB files using the -m
and -M options:

Also copy the MIBs files on localhost where Nagios is running under snmp/mibs

/usr/sbin/snmptrapd -m ALL -M /usr/share/snmp/mibs -Lf /var/log/snmptrapd.log


Tip:  There is an excellent HOWTO on how to use snmptrap to send traps here.

Note: Your own SNMP management station, such as HP OpenView or the like, will have its own method of loading MIB files. You should refer to its documentation for that.

2. Define a contact for Management station

Need to define a contact for management station in the same way we create for other contact person, to receive SNMP traps from Nagios whereever Nagios send notificaiton for host/service

define contact{
        contact_name                             managementstation
        use                                              generic-contact
        alias                                            Management Station
        service_notification_options    w,u,c,r
        host_notification_options         d,u,r
        service_notification_commands   send-service-trap
        host_notification_commands      send-host-trap
        }


3. Add the new conatct to contactgroup for hosts/services for which you want to send SNMP traps

define contactgroup{
        contactgroup_name       linuxadmins
        alias                                Linux Servers Administrators
        members                         me, managementstation
}

Now whenever there is a alert for linux servers Nagios will notify contact "managementstation" which in turn call the commands send-service-trap/send-host-trap accordingly depends on host or service alert.

4. send-service-trap/send-host-trap Shell Script

===/usr/local/bin/send-service-trap ====
# Arguments:
# $1 = Management Station
# $2 = Community String
# $3 = host_name
# $4 = service_description (Description of the service)
# $5 = return_code (An integer that determines the state
#       of the service check, 0=OK, 1=WARNING, 2=CRITICAL,
#       3=UNKNOWN).
# $6 = plugin_output (A text string that should be used
#       as the plugin output for the service check)
#
#
/usr/bin/snmptrap -v 2c -c $2 $1 '' NAGIOS-NOTIFY-MIB::nSvcEvent nSvcHostname s "$3" nSvcDesc s "$4" nSvcStateID i $5 nSvcOutput s "$6"


===/usr/local/bin/send-host-trap=======
# Arguments:
# $1 = Management Station
# $2 = Community String
# $3 = host_name
# $4 = HostStatID A number that corresponds to the current state of the host: 0=UP, 1=DOWN, 2=UNREACHABLE.
# $5 = HOSTOUTPUT The first line of text output from the last host check (i.e. "Ping OK").
#
#
/usr/bin/snmptrap -v 2c -c $2 $1 '' NAGIOS-NOTIFY-MIB::nHostEvent nHostname s "$3" nHostStateID i $4 nHostOutput s "$5"


The shell script receives all the incoming variables and passes them to the snmptrap command

-v = snmp version
-c = community string
$1 = Management Station IP address or hostname

The next two single quotes are special characters representing the uptime portion of a trap. They will be replaced with the current uptime of the system generating the trap when the snmptrap command is executed.

The enterprise OID is specified next; in this case it is NAGIOS-NOTIFY-MIB::nSvcEvent and NAGIOS-NOTIFY-MIB::nHostEvent. The NAGIOS-NOTIFY-MIB is the name of the MIB module being referenced, and the nSvcEvent/nHostEvent is the trap defined for sending service events. Together these form the enterprise OID.

Next specified a list of individual OIDs and their variables that I’m passing as part of the trap. They are, in order, the hostname, the service description, the host/service status ID in numerical form, and the output of the host/service check.

Enclosing the values ones that might contain multiword data in quotation marks so that they are passed cleanly to the command.

Tip:  You can find these and other OIDs that you can use with the traps and notifications defined in the NAGIOS-ROOT-MIB and NAGIOS-NOTIFY-MIB MIB files.

Friday, January 15, 2010

Creating Custom Template in pnp4nagios

Some time need to create custom template in pnp4nagios for some command which behave differently when execute on remote host as the appearance of graphs depends on the check command used. eg check_nrpe

Example:

define command {
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

This would lead to a call of the check_nrpe.php template even when the monitored host would use a completely different plugin which is called via NRPE, eg

In service definition

define service{
use local-service,srv-pnp
host_name SomeHost
service_description Current Load
check_command check_nrpe!check_load
}

This will lead to call to check_nrpe command and it will use the default.php PNP template which may or may not give the graph as required.

Solution: Custom Template

1. Define new command in nagios/commands.cfg

define command{
command_name check_nrpe_load
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

2. Modify the service command for the host to use the new command

define service{
use local-service,srv-pnp
host_name SomeHost
service_description Current Load
check_command check_nrpe_load!check_load
}

Reload Nagios so that it start using the new check command.

At this point everything will be working just as before,

Pnp4nagios configuration

1. Define a check command /usr/local/pnp4nagios/etc/check_commands/check_nrpe_load.cfg (with the same name as the check command in above example)

Tip: Just copy the sample command definition /usr/local/pnp4nagios/etc/check_commands/check_nwstat.cfg-sample eg

cd /usr/local/pnp4nagios/etc/check_commands
sudo cp check_nwstat.cfg-sample check_nrpe_load.cfg

For more detail on custom command refer to http://docs.pnp4nagios.org/pnp-0.6/tpl_custom

2. Define a customer template for check_nrpe_load

In the above example if check_nrpe_load.cfg have

CUSTOM_TEMPLATE = 0,1 the it will results in → "check_nrpe_load_check_load.php"

Create a custom template in /usr/local/pnp4nagios/share/templates (pnp4nagios first look for template in this direcotry then fallback to other directories http://docs.pnp4nagios.org/pnp-0.6/tpl)

Note: Or you can just create a symbolic link to existing template that come with pnp4nagios installation eg ...

ln -s ../templates.dist/check_load.php check_nrpe_load_check_load.php

Here is the sample template for check_load on remote hosts that will plot graph of 1M and 15 minute graph for a host,

cd /usr/local/pnp4nagios/share/templates

sudo vi check_nrpe_load_check_load.php (template name should match with the custom command definition in our example check_nrpe_load_check_load.php)

<?php
$opt[1] = "--vertical-label Load -l0  -u 1 --title \"CPU Load for $hostname / $servicedesc\" ";

$def[1] =  "DEF:var1=$rrdfile:$DS[1]:AVERAGE " ;
$def[1] .= "DEF:var2=$rrdfile:$DS[2]:AVERAGE " ;
$def[1] .= "DEF:var3=$rrdfile:$DS[3]:AVERAGE " ;
if ($WARN[1] != "") {
    $def[1] .= "HRULE:$WARN[1]#FFFF00 ";
}
if ($CRIT[1] != "") {
    $def[1] .= "HRULE:$CRIT[1]#FF0000 ";
}
$def[1] .= "AREA:var1#EACC00:\"Load average  1 min \" " ;
$def[1] .= "GPRINT:var1:LAST:\"%6.2lf last\" " ;
$def[1] .= "GPRINT:var1:AVERAGE:\"%6.2lf avg\" " ;
$def[1] .= "GPRINT:var1:MAX:\"%6.2lf max\\n\" ";
$def[1] .= "LINE:var3#FF0000:\"Load average 15 min \" " ;
$def[1] .= "GPRINT:var3:LAST:\"%6.2lf last\" " ;
$def[1] .= "GPRINT:var3:AVERAGE:\"%6.2lf avg\" " ;
$def[1] .= "GPRINT:var3:MAX:\"%6.2lf max\\n\" " ;
?>

Sunday, January 10, 2010

Nagios addon - pnp4nagios

PNP4Nagios is a gem of an add-on; it allows a Nagios administrator to easily add RRD-style graphs and efficient long-term trending ­capabilities to Nagios. This graphing framework makes extensive use of templates and is easy to customize. It consists of a PHP-based front using Kohana framework.



Pnp4nagios uses kohana framework for web frontend and kohana requires php 5.2.x has to be installed. In my previous post I have describe how to upgrade to php 5.2.x on centos which comes with 5.1.x.


System Requirement

Perl >= 5.x
RRDtool
PHP >= 5.2. for the Webfrontend based on Kohana
PHP-extension zlib, session and GD.
Nagios >= 2.x

Installation

Joerg Linge have written excellent documentation for pnp4nagios and using it one can easily install and configure PNP in no time.

In my case I have used the following ./configure line to have rrd data under nagios directories and rest of pnp4nagios goes its own directories in filesystem.

./configure --with-perfdata-logfile=/var/nagios/perfdata.log --with-perfdata-dir=/var/nagios/perfdata --with-perfdata-spool-dir=/var/nagios/spool/perfdata

After installing follow the pnp docs for start generating graphs and web integration with Nagios.

Before going for production use, decide which mode is right for your Nagios installation.

In our case we go with Bulk Mode with NPCD as per docs...

Nagios again uses a temporary file to store the data and executes a command after expiration of a certain time. Instead of immediate processing by process_perfdata.pl the file is moved to a spool directory. As moving a file inside the same filesystem nearly takes no time nagios is able to execute crucial work immediately.
The NPCD daemon (Nagios Performance C Daemon) will monitor the directory for new files and will pass the names to process_perfdata.pl. Processing of performance data is decoupled completely from nagios. NPCD itself is able to start multiple thread for processing the data.

Pages

One of the cool feature of pnp4nagios to have separate pages of graphs for hosts/services on a single page.
Once again Joerg have an excellent docs for configuring pages.

Conclusion


Once installed and configured pnp4nagios provides trending and capacity planning which allows organizations to plan for infrastructure upgrades before outdated systems catch them by surprise.

Picasa 3 sucks

Today after upgrading to Picasa 3 on Fedora 12, it no longer let me upload my photos to web album, each time it gives the same error

Login failed - please try again later.

I have exhausted all the options googling, uninstall and reinstall but no luck.

Come on Google you should be watching your groups on which people crying for the issue of sign in after installing/upgrading to picasa 3.

Sunday, January 3, 2010

Adding chunk to dbspace in IBM Informix (IDS) using 'onspaces' utility

Recently I got into problem when there was on space left on dbspace on one of our Informix database. It took me nearly 2 hours of Googling to figure out how to add CHUNK to existing dbspace.

I thinks it would be better to blog it, this might help some other folks or might I need to use it some day when the newly added CHUNK also get full :)

For my setup I do not have the additional hard drive and have a plenty of free space on Unix partition. Therefore I used the unix file as new Chunk.

'onmonitor', oncheck -pe, onstat -d commands can be used to view the status of dbspaces.

Here are the steps to add chunk.

1. First create a chunk file in Unix filesystem and set the permission so that informix user/group have the read/write permission on the file.


touch /export/home/dbspace/billdb_dg/lv_recorddbs11

chmod 660 /export/home/dbspace/billdb_dg/lv_recorddbs11

Note: I run the above commands with 'informix' user or the owner of informix db.

The following example adds a 2GB chunk to recordbs. An offset of 200 kilobytes is specified.

onspaces -a recorddbs -p /export/home/dbspace/billdb_dg/lv_recorddbs11 -o 200 -s 2093150

Here -s 2093150 is specified in KB

The newly added chunk is available immediately.

After you create the chunk, you must perform a level-0 backup of the root dbspace and the dbspace that contains the chunk.

Friday, January 1, 2010

Chat with Joerg Linge developer/maintainer of pnp4nagios

Today I got an opportunity to chat with developer/maintainer Joerg Linge of pnp4nagios.

Chat was mostly about the new feature of using multiple Data Sources for a service in pnp4nagos.

RRD_STORAE_TYPE = SINGLE means: Store every Datasource in one single RRD Database i.e. for check_icmp this is DS 'pl' and DS 'rta'

With RRD_STORAE_TYPE = MULTILPLE every DS is stored in its own RRD Database i.e. host/service_pl.rrd and host/service_rta.rrd

RRD_STORAE_TYPE = MULTILPLE should only used when a plugin is able to change the number of DS like check_disk does.

Me:  Is it ok to set  RRD_STORAE_TYPE = MULTIPLE globally in process_perfdata.cfg for new installation of Joerg? when one do not have any RRD data at all.
Me:  tomorrow i am going to deploy Joerg 0.6.x on our production nagios server.
Me:  thanks.
Joerg : it depends
Joerg : happy new year!
Me:  happy new year :)
Me:  ok,
Me:  going with MULTIPLE has no drawback ok?
Joerg : multiple mean more Disk I/O because of more RRDs to update

Joerg : on sytems with 10000+ rrds you have to look on your disk I/O clisely
Me:  well this really need very close monitoring of I/O which is hard to follow :s
Me:  better stick with SINGLE :)
Joerg : right
Joerg : there are some plugins with changing DS counts like check_disk
Joerg : in this case multiple is an option
Joerg : or check_oracle_health for tablespace monitoring
Me:  we are using check_disk extensively.
Me:  for checking servers partitions.
Joerg : one check for all filesystems?
Me:  well mostly separate check_disk for each fs, however we have some servers with lot of fs onthese servers we are using single check_disk to check multiple fs in one go.
Me:  with -p option
Me:  those are mostly HP-UX
Joerg : so check_disk can be used with multiple
Me:  ok to use multiple have to configure it globally?
Joerg : wenn you add a nw filesystem, rrdtool is not able to update the rrd databases
Joerg : lol german wenn
Joerg : english when
Joerg : sorry
Me:  np, i thought its typo :)
Me:  ok to use multiple have to configure it globally? or their is another way around?
Joerg : moment
Joerg : http://docs.Joerg.org/pnp-0.6/tpl_custom
Me:  looking.
Joerg : you can set RRD_STORAGE_TYPE per check_commands
Joerg : i.e. your command is check_disk
Joerg : create etc/check_commands/check_disk.cfg
Joerg : set RRD_STORAGE_TYPE = MULTIPLE within this file
Joerg : thats the same rrd_convert.pl --check_command=check_disk will do in its final version
Me:  ok but for most host we are doing the check_disk on remote host and then using check_nrpe on nagios host
Joerg : so your command is?
Joerg : check_nrpe?
Me:  yes
Joerg : create a new command nrpe_disk
Joerg : thats it
Me:  can you give me full example of nrpe_disk here?
Joerg : define command {
Joerg : command_name nrpe_disk
Joerg : command line $USER1$/check_nrpe -H $HOSTqADDRESS$ -c check_diks1
Joerg : }
Joerg : so your command is nrpe_disk and not check_nrpe
Joerg : and now you are able to define MULTIPLE for this command
Joerg : this is done by creating Joerg/etc/check_commands/nrpe_disk.cfg
Me: Thanks for your time very much appreciated.