Wednesday, December 22, 2010

Receiving SNMP Traps in Nagios

SNMP traps are alerts and notifications generated by SNMP-enabled devices. The traps con-tain information about the status or an event on an SNMP-enabled device. For example, an authentication event or the change in status of an interface on a router may generate an SNMP trap that is sent to a management station of some sort, such as HP OpenView, CiscoWorks, Nagios.

Pre-requisites:

1. Net-SNMP with snmptrapd configured.
2. SNMPTT, SNMP trap translator.
3. Nagios.
5. Mib definition files for the equipment or software you need to monitor.

Installing Net-SNMP packages:

The Net-SNMP package is available as a series of installable packages on many distributions. Indeed, it may already be installed on your system or you may be able to install it via your distribution’s package management system, such as yum, apt, or the like. On Red Hat, SuSE, Debian, and Mandrake
distributions, the required packages are called net-snmp, net-snmp-libs, and net-snmp-utils.

Installing Net-SNMP packages on Centos 5.5

# yum install net-snmp net-snmp-libs net-snmp-utils net-snmp-perl perl-Net-SNMP net-snmp-devel

Configuring and Running the snmptrapd Daemon
When incoming traps are received from the snmptrapd daemon, they are passed to the SNMPTT tool. The SNMPTT tool will then try to match the incoming trap against the collection of trap definitions that it has translated. If the trap matches, SNMPTT will see if the translated trap definition contains logic to output it to Nagios and execute that logic. The trap is then out-put to Nagios as a passive check result.

On Centos 5.5

# vi /etc/sysconfig/snmptrapd.options
#OPTIONS="-On -Lsd -c /etc/snmp/snmptrapd.conf -p /var/run/snmptrapd.pid"
OPTIONS="-On -Lsd -p /var/run/snmptrapd.pid"

Make sure to remove the -c /etc/snmp/snmptrapd.conf part, otherwise you will receive TRAP twice, as snmptrapd' is compiled with the default configuration file path being already set to '/etc/snmp/snmptrapd.conf'.

As quoted from SNMP Trap Translator documentation: "The -On is recommended. This will make snmptrapd pass OIDs in numeric form and prevent SNMPTT from having to translate the symbolic name to numerical form."

# vi /etc/snmp/snmptrapd.conf
traphandle default /usr/sbin/snmptthandler
disableAuthorization yes
#donotlogtraps  yes

The traphandle directive tells the snmptrapd daemon how to handle incoming traps and where to send them. Adding the default option tells the daemon that this is the default way to handle all incoming traps. All traps will be sent to the snmptthandler script located in the /usr/sbin directory, the "disableAuthorization yes" tells to accept SNMP traps from all you can configure it to do authentication for detail refer to snmptrapd.conf manual.

Installing SNMPTT (SNMP Trap Translator)
You can get the SNMPTT tool from Sourceforge at http://snmptt.sourceforge.net/. This line shows how to download and unpack the SNMPTT tool:
Download snmptt_1.3.tgz which the latest version/stable release.

tar -zxvf snmptt_1.3.tgz

The SNMPTT package has no installation script, so a number of manual installations steps need to take place. First, copy the SNMPTT binaries to a suitable directory and mark them as executable. I recommend using the /usr/sbin directory

# cp snmptt snmptthandler /usr/sbin/
# chmod +x /usr/sbin/snmptt /usr/sbin/snmptthandler

I specified the snmptthandler binary as the value of the traphandle option in the snmptrapd.conf configuration file in the previous section. When a trap is received, this binary is executed by default and the trap sent to the snmptt daemon

Next, copy the SNMPTT configuration file, snmptt.ini, to the /etc/snmp directory and snmpttconvertmib utility

# cp snmptt.ini /etc/snmp/
# cp snmpttconvertmib /usr/sbin/

Also needed are a user and group to run the SNMPTT daemon as.

# groupadd snmptt
# adduser -g snmptt snmptt

# chown snmptt:snmptt /etc/snmp/snmptt.ini

The SNMPTT tool also needs a spool directory to hold the incoming traps. I usually use the default directory of /var/spool/snmptt. It needs to be owned by the user and group that will run SNMPTT. Create and change the ownership of the directory like so

# mkdir /var/spool/snmptt
# chown snmptt:snmptt /var/spool/snmptt

Finally, in order to start the SNMPTT tool, you can either execute it from the command line or use the init script provided with the package. On the following line

SNMPTT started in daemon mode:
# /usr/sbin/snmptt -daemon

Or copy the init script provided with the package, you can then add it to your startup process.

# cp snmptt-init.d /etc/init.d/snmptt

To start/stop/reload you can do with ..

/etc/init.d/snmptt start/stop/reload

Configuring SNMPTT
The first is configuring the /etc/snmp/snmptt.ini file. The file contains quite a large number of directives, but I’ll only look at those relevant to the process of translating and transmitting the received traps to Nagios

mode = daemon
daemon_fork = 1
daemon_uid = snmptt
spool_directory = /var/spool/snmptt/
sleep = 5
dns_enable = 1
strip_domain = 1
log_enable = 1
syslog_enable = 0
exec_enable = 1
snmptt_conf_files = <
/etc/snmp/snmptt.conf
END


The sample snmptt.ini file contained in the SNMPTT package has detailed explanations of all the directives and options that you can specify. I recommend reading this file for further information and explanations about SNMPTT’s configuration options

Tip:  If you enable DNS resolution, I recommend you add all the hostnames that need to be resolved to the local /etc/hosts file on your host server. This prevents your DNS server from being a bottleneck or preventing SNMPTT from functioning if your DNS server is unavailable.

Compiling MIBs
You must gather all MIBs for monitored software, so you can feed SNMPTT with them. Compiling consists in extracting each OID of type "trap" and its associated comments, and generate a configuration file in SNMPTT format from these informations.

Run the following command on each of your MIB files:

snmpttconvertmib --in= --out=/etc/snmp/snmptt.conf. \
--exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1'

The resulting SNMPTT configuration file will contain blocks (one per selected OID)


Catchall Trap Definition
SNMPTT also has a regular expression–matching capability that allows you to use an EVENT line that matches multiple incoming traps, a catchall trap definition. This means you don’t need to define individual translated trap definitions for each possible incoming trap.

Catchall Trap Definition
EVENT CatchAll .1.* "SNMP Traps" Critical
FORMAT $D
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result "$r" 
"snmp_traps" 2 "$O: $1 $2 $3 $4 $5"

I could also be more selective and select OIDs from a particular vendor or class of trap either using a wildcard or regular expression pattern matching. I’ve added a category called SNMP Traps and severity of Warning.

For example here is example to catchall traps from a specific vendor OID.

EVENT CatchAll .1.3.6.1.4.1.20916.* "Status Events" Normal
FORMAT A room-alert-4e-snmp-trap indicates that an alarm $*
EXEC /usr/lib/nagios/plugins/eventhandlers/submit_check_result $r "snmp_traps" 1 "A room-alert-4e-snmp-trap indicates that an alarm $*"
SDESC
A room-alert-4e-snmp-trap indicates that an alarm
condition has occurred on the sensor indicated
by the alarmmessage variable.
Variables:
  1: alarmmessage
EDESC

When done, add to SNMPTT configuration file /etc/snmp/snmptt.ini the path to compiled configuration files:

[...]
snmptt_conf_files = <
/etc/snmp/snmptt.conf.
/etc/snmp/snmptt.conf.
END


Configuring Nagios
You will use passive checks to receive SNMP traps but they also will be volatiles. If ever two traps are received from the same host, the second one coming in before the first one was reset to OK, we want to be notified twice, although there is no state change. That's why we use a volatile service.

You might define (for example) a service template for SNMP traps, inheriting from a generic service template:

define service{
        name                            generic-service    
        active_checks_enabled           1                     
        passive_checks_enabled          1                       ; Passive service checks are enabled/accepted
        parallelize_check               1                     
        obsess_over_service             1                      
        check_freshness                 0                      
        notifications_enabled           1                       ; Service notifications are enabled
        event_handler_enabled           1                       ; Service event handler is enabled
        flap_detection_enabled          1                       ; Flap detection is enabled
        failure_prediction_enabled      1                       ; Failure prediction is enabled
        process_perf_data               1                       ; Process performance data
        retain_status_information       1                       ; Retain status information across program restarts
        retain_nonstatus_information    1                       ; Retain non-status information across program restarts
        is_volatile                     0                       ; The service is not volatile
        check_period                    24x7                    ; The service can be checked at any time of the day
        max_check_attempts              3                       ; Re-check the service up to 3 times in order to determine its final (hard) state
        normal_check_interval           10                      ; Check the service every 10 minutes under normal conditions
        retry_check_interval            2                       ; Re-check the service every two minutes until a hard state can be determined
        contact_groups                  admins                  ; Notifications get sent out to everyone in the 'admins' group
        notification_options            w,u,c,r                 ; Send notifications about warning, unknown, critical, and recovery events
        notification_interval           60                      ; Re-notify about service problems every hour
        notification_period             24x7                    ; Notifications can be sent out at any time
         register                        0                      ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!
        }

define service{
name                    trap-service
use                     generic-service
register                0
service_description     snmp_traps
is_volatile             1
check_command           check-host-alive    ;Used to reset the status to OK when 'Schedule an immediate check of this service' is selected.
flap_detection_enabled  0                               ; Flap detection is disabled
process_perf_data       0                               ; Do not Process performance data
max_check_attempts      1                    ; Leave as 1
normal_check_interval   1                    ; Leave as 1
retry_check_interval    1                    ; Leave as 1
passive_checks_enabled  1                    ; Enables passive checks
check_period            24x7
notification_interval   31536000                ; Notification interval.  Set to a very high number to prevent you from getting                                 pages of previously received traps (1 year - restart Nagios at least once a year! -                                     do not set to 0!).
active_checks_enabled   0                    ; Prevent active checks from occuring as we are only using passive checks.
notification_options    w,u,c                    ; Notify on warning, unknown and critical.
contact_groups          sysadmins
}

define service{
 host_name       AVT-Room-Alert ; hostname is define /etc/hosts file
 use             trap-service
 contact_groups sysadmins
}

TIP: You could also use a wildcard to create this service for all hosts or use the hostgroup_name directive to create the service for all members of a host group or groups.

I’ve defined the service as volatile and set the maximum check attempts to 1. This will cause Nagios to immediately set a HARD service state and trigger any configured notifications or event handlers. I’ve also configured it for passive checks only and disabled active checks.

Putting It All Together
The SNMPTT tool is called via the trap handler defined in the snmptrapd.conf configuration file I defined in the “Configuring and Running the snmptrapd Daemon” section. This trap handler calls the /usr/sbin/snmptthandler script. The script reads the trap and then writes it to the spool directory defined in the spool_directory directive from the snmptt.ini configuration file. The script then exits.

From here the SNMPTT daemon takes over. It reads the trap from the spool file and searches for a match in its trap definitions. If it finds a match, it executes the EXEC statement in the matching trap definition. This EXEC statement sends the passive check result to the Nagios server using the submit_check_result script. The daemon then sleeps for the period specified in the sleep directive in the snmptt.ini file and checks the spool directory for additional traps; if it finds matches, it processes them and sends the check results to Nagios.

The Nagios server has to have host objects defined for every host that generates SNMP traps. Additionally, you need to define service objects for those hosts to receive the service check results. You should configure them to receive passive check results and as volatile services.

81 comments:

  1. awesome tutorial, one of the best I found on the net.
    Thanks!

    ReplyDelete
  2. Why do you use the same line twice her:
    [...]
    snmptt_conf_files = <
    /etc/snmp/snmptt.conf.
    /etc/snmp/snmptt.conf.
    END
    ?

    ReplyDelete
  3. Great tutorial, thanks a lot. SNMPTT is not well packaged in my opinion and can easily have an install script, but this tutorial was very clear and easy to follow. I had some package dependencies issue and had to install perl-Config-IniFiles to make it work. It might be a good idea to add the required perl packages to the Pre-requisites. Running #perl -MCPAN -e 'install Config::IniFiles' seems to resolve the compilation errors as well. This is the error I got when tried to start the daemon:
    Can't locate Config/IniFiles.pm in @INC (@INC c... at /usr/sbin/snmptt line 3894

    ReplyDelete
    Replies
    1. Thanks for liking it this is why i decided at first place to put online my work of integrating Nagios + SNMP trap online due to the lack of easy to follow howto for this task.

      Delete
  4. Hi

    I am still puzzling which config file is needed to modify or created for the nagios, possible to share?

    - paul

    ReplyDelete
    Replies
    1. You must know where your cfg files are.

      You can find this out by finding your which nagios.cfg/icinga.cfg file your nagios/icinga instance are referring to and greping or cfg_dir or cfg_file.

      You must then include the template and service definition within the cfg file.

      You can also add a cfg_dir definition to your nagios.cfg or icinga.cfg:

      mkdir -p /usr/local/nagios/snmptrapcfg
      echo "cfg_dir=/usr/local/nagios/snmptrapcfg" >> /usr/local/nagios/etc/nagios.cfg
      curl -k https://raw.github.com/gist/3503677/9965139383ed4014d47d25d0a32cab16f8d36173 > /usr/local/nagios/snmptrapcfg/snmptrap_template.cfg
      curl -k https://raw.github.com/gist/3540126/1e637d701707f13f4af53b13955719b50011ea8d > /usr/local/nagios/snmptrapcfg/ex_service.cfg
      vim ex_service.cfg

      Edit and copy the ex_service.cfg and assign your hosts.

      Delete
  5. Hi,

    I found your guide after several unsuccessful attempts to get snmptt working. I get as far as starting the snmptt service but get the below error.

    "Can't locate List/MoreUtils.pm in @INC (@INC contains: /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.8/i386-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /usr/lib/perl5/vendor_perl/5.8.8/Config/IniFiles.pm line 14.
    BEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/Config/IniFiles.pm line 14.
    Compilation failed in require at /usr/local/sbin/snmptt line 4026.
    BEGIN failed--compilation aborted at /usr/local/sbin/snmptt line 4026."

    Any ideas?

    ReplyDelete
    Replies
    1. On CentOS/Redhat/Fedora...

      # yum whatprovides */MoreUtils.pm

      perl-List-MoreUtils-0.22-10.el6.i686 : Provide the stuff missing in List::Util
      Repo : base
      Matched from:
      Filename : /usr/lib/perl5/List/MoreUtils.pm


      Install this package:
      yum -y install perl-List-MoreUtils

      Delete
  6. Greatings,

    I am also tyring to start and getting :

    /etc/rc.d/init.d/snmptt start
    Starting snmptt: Can't locate Config/IniFiles.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/sbin/snmptt line 4026.
    BEGIN failed--compilation aborted at /usr/sbin/snmptt line 4026.


    Dont know what to install next ? Centos 6

    ReplyDelete
    Replies
    1. install perl-Config-IniFiles using "yum install perl-Config-IniFiles"
      and try it

      Delete
    2. I tried successfuly with this lines:

      perl -MCPAN -e 'install Test::Harness'
      perl -MCPAN -e 'install Config::IniFiles'

      Delete
  7. Hi,

    It is indeed very sad to see how blatanly you copied James Turnbull's Pro Nagios 2.0 book (2006) without stating it.

    Shame on you dude...

    ReplyDelete
  8. Oh no! and I copied this guy, so now I'll have to cite James also!

    ReplyDelete
  9. How do you specify if the trap receives is a warning/critical/etc ?

    ReplyDelete
  10. I am a new guy in perl,in my place I have to made Perl script.. I never did it before. The application owner is the Nagios and ondot equipments, The SNMP Traps are coming from Nagios. We have to make the script to pars e them in to NSM server for going to WALLBOARD.
    They have MIB, OID (we will provide), they will monitors cPU, Diskapce, RT, DNS, network, servers etc. Can anybody help me her, as its the challenging project for me. Thanks

    Sifte277@gmail.com

    SIFULLAH
    Atlanta

    ReplyDelete
  11. Hi Mate,

    In the section 'Compiling MIBs' when I run below command:

    snmpttconvertmib --in=/CISCO-C2900-MIB.mib --out=/etc/snmp/snmptt.conf --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1'

    I am getting following error. See last few lines of output:

    Cannot adopt OID in CISCO-C2900-MIB: c2900BandwidthUsageCurrent ::= { c2900BandwidthUsage 1 }
    Cannot adopt OID in CISCO-C2900-MIB: c2900PortEntry ::= { c2900PortTable 1 }
    Unknown object identifier: CISCO-C2900-MIB::c2900RpsFailed
    OID:

    Done

    Total translations: 3
    Successful translations: 0
    Failed translations: 3

    Could someone please help me, if any idea what's wrong?

    Cheers !!

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Realize this is a really old post, but I ran into this problem as well. I had to copy all of the .my files from the MIB to /usr/share/snmp/mibs to resolve it.

      Delete
  12. Hi,

    I have been able to receive traps in /var/log/snmptt/snmptt.log file. But I am struggling to make it to nagios-xi. Can anybody guide me on how to add this to nagios-xi?

    ReplyDelete
    Replies
    1. Hi mahendra how to receive traps in snmptt.log file? I need read traps for save in mysql database

      Delete
  13. This is a fantastic tutorial, one of the best I found. Thank you!

    (Haven't seen inside the book, but I may also owe a thanks to James Turnbull)

    Although all of the important steps are covered here, I would still encourage others to spend some time going through the documentation that comes with snmptt as it has a lot of useful options.

    I have just set up Nagios, snmptrapd and snmptt on CentOS 5.8 to monitor Veeam backup jobs and I found the snmptt REGEX directive especially useful for translating messages from Veeam about job status into Nagios status codes.

    When installing Perl dependencies, I used the list supplied with snmptt. To find Perl modules already installed I use a script similar to the one posted by dland on Stack Overflow (http://stackoverflow.com/questions/115425/how-do-i-get-a-list-of-installed-cpan-modules). I found most of the deps in the Rpmforge CentOS repo.

    ReplyDelete
  14. Great Tutorial! Could you please provide detailed procedure on how to compile Mibs files, specificaly cisco ones.
    Thanks.

    ReplyDelete
  15. SNMP protocol Analysis
    https://www.youtube.com/watch?v=_A2mPw7iSfY

    ReplyDelete
  16. Hi Ali,

    I could not find the file name MiB for transfer. Have a sample file name or how can I find?

    ReplyDelete
    Replies
    1. root@icingamon:/usr/share/mibs/netsnmp# ls /usr/share/mibs/
      iana ietf netsnmp
      root@icingamon:/usr/share/mibs/netsnmp# ls /var/lib/mibs
      iana ietf
      root@icingamon:/usr/share/mibs/netsnmp#

      Delete
  17. This comment has been removed by the author.

    ReplyDelete
  18. how to receive snmp trap in order to save the ip, the description of failure in my mysql database??

    ReplyDelete
  19. Hi

    This may well be termed as a very stupid question but still I will ask it since I am very new to this. Where should the SNMPTT be configured? Is it in the monitoring server where the Nagios has been configured or in the host(or server) monitored by Nagios.

    Thanks
    D

    ReplyDelete
    Replies
    1. On the monitoring server where Nagios is running.

      Delete
    2. Okay, thanks. I followed each of your steps here but when I am sending a SNMP TRAP from my monitored server, I don't see any warning in my monitoring server, the TRAP service remains OK and doesn't change to WARNING. What do you think I should look for in my configuration to make it work and receive the traps?

      Thanks
      D

      Delete
    3. Sorry can't help much not in touch with Nagios from LONG time (doing different job) :-s, please try to Google, or check in nagios mailing list , or #nagios on irc.freenode.net.

      Delete
  20. Hi!
    Btw, great tutorial. Im a little confused. I need to add the service "name generic-service" and "name trap-service" to templates.cfg?
    the last service you mentioned in your post must go in the host.cfg in which I like to monitor? In my case a power generator.
    Im correct?

    I'll appreciate.

    ReplyDelete
  21. This comment has been removed by the author.

    ReplyDelete
  22. Hi,

    When we try to start the snmptt service we are getting following error. Any idea why ?

    Starting snmptt: Error in config file - please check the syntax in the config file

    ReplyDelete
    Replies
    1. HI Rajeev

      Can you try the following update in your `/etc/snmp/snmptt.ini` file

      [...]
      snmptt_conf_files = <<END
      /etc/snmp/snmptt.conf.
      END

      Delete
  23. I'm struggling because I just can't recieve the traps, I'm on the handler mode.. but I don't know what is wrong on my system:
    1.- I generate snmp events ./simulate_events_tf.sh (on my monitored host)
    2. I do tail of /var/log/messages
    Sep 17 17:09:04 gdl-storage snmptrapd[32621]: No access configuration - dropping trap.
    Sep 17 17:09:04 gdl-storage snmptrapd[32621]: No access configuration - dropping trap.
    Sep 17 17:09:04 gdl-storage snmptrapd[32621]: No access configuration - dropping trap.
    Sep 17 17:09:04 gdl-storage snmptrapd[32621]: No access configuration - dropping trap.
    Sep 17 17:09:04 gdl-storage snmptrapd[32621]: No access configuration - dropping trap.
    Sep 17 17:09:04 gdl-storage snmptrapd[32621]: No access configuration - dropping trap.
    Sep 17 17:09:04 gdl-storage snmptrapd[32621]: No access configuration - dropping trap.
    Sep 17 17:09:04 gdl-storage snmptrapd[32621]: No access configuration - dropping trap.
    Sep 17 17:09:04 gdl-storage snmptrapd[32621]: No access configuration - dropping trap.
    Sep 17 17:09:18 gdl-storage snmptt-sys[4869]: Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0

    3. The content of /etc/snmp/snmptrap.conf
    # Example configuration file for snmptrapd
    #
    # No traps are handled by default, you must edit this file!
    #
    disableAuthorization yes
    #authcommunity log,execute,net public
    #traphandle default /usr/sbin/snmptt
    traphandle default /usr/sbin/snmptthandler
    # traphandle SNMPv2-MIB::coldStart /usr/bin/bin/my_great_script col

    ReplyDelete
    Replies
    1. Hey Juan, were you able to make snmp trap working on nagios core?

      Delete
  24. Hi there, hope you can help out on this.
    I'm stuck on the step of converting the mibs.

    when I run the command:


    snmpttconvertmib --in==/root/Downloads/CISCO-CONFIG-MAN-MIB.my --out=/etc/snmp/snmptt.conf. \
    --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1'

    I get this:

    ***** UCD-SNMP / NET-SNMP Perl module enabled *****



    ***** Processing MIB file *****

    snmptranslate version: NET-SNMP version: 5.7.2
    severity: Normal

    File to load is: /root/Downloads/CISCO-CONFIG-MAN-MIB.my
    File to APPEND TO: /etc/snmp/snmptt.conf

    MIBS environment var: /root/Downloads/CISCO-CONFIG-MAN-MIB.my
    mib name: CISCO-CONFIG-MAN-MIB
    MIB search path: /root/.snmp/mibs:/usr/share/snmp/mibs
    Cannot find module (CISCO-TC): At line 42 in /root/Downloads/CISCO-CONFIG-MAN-MIB.my
    Cannot find module (CISCO-SMI): At line 44 in /root/Downloads/CISCO-CONFIG-MAN-MIB.my
    Did not find 'Unsigned64' in module #-1 (/root/Downloads/CISCO-CONFIG-MAN-MIB.my)
    Did not find 'ciscoMgmt' in module #-1 (/root/Downloads/CISCO-CONFIG-MAN-MIB.my)
    Unlinked OID in CISCO-CONFIG-MAN-MIB: ciscoConfigManMIB ::= { ciscoMgmt 43 }
    Undefined identifier: ciscoMgmt near line 47 of /root/Downloads/CISCO-CONFIG-MAN-MIB.my
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ciscoConfigManMIBGroups ::= { ciscoConfigManMIBConformance 2 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ciscoConfigManMIBCompliances ::= { ciscoConfigManMIBConformance 1 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ciscoConfigManMIBComplianceRev4 ::= { ciscoConfigManMIBCompliances 4 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ciscoConfigManMIBComplianceRev3 ::= { ciscoConfigManMIBCompliances 3 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ciscoConfigManMIBComplianceRev2 ::= { ciscoConfigManMIBCompliances 2 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ciscoConfigManMIBCompliance ::= { ciscoConfigManMIBCompliances 1 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ccmCLIHistoryCommand ::= { ccmCLIHistoryCommandEntry 2 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ccmCLIHistoryCommandIndex ::= { ccmCLIHistoryCommandEntry 1 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ciscoConfigManMIBNotifications ::= { ciscoConfigManMIBNotificationPrefix 0 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ccmHistoryEventServerAddrRev1 ::= { ccmHistoryEventEntry 19 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ccmHistoryEventServerAddrType ::= { ccmHistoryEventEntry 18 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ccmHistoryEventCommandSourceAddrRev1 ::= { ccmHistoryEventEntry 17 }
    Cannot adopt OID in CISCO-CONFIG-MAN-MIB: ccmHistoryEventCommandSourceAddrType ::= { ccmHistoryEventEntry 16 }

    ***** Using UCD-SNMP / NET-SNMP Perl module *****



    Done

    Total translations: 3
    Successful translations: 0
    Failed translations: 3


    ReplyDelete
    Replies
    1. You have to copy all the .my files for the MIB you're trying to convert to /usr/share/snmp/mibs

      Delete
    2. Thank you for the reply, I copied all the mibs and now I'm getting the right output
      Total translations: 3
      Successful translations: 3
      Failed translations: 0

      Following the mib translation page, comes the step of catchall Trap Definition. is this a configuration changes ? and where we change this?

      Catchall Trap Definition
      SNMPTT also has a regular expression–matching capability that allows you to use an EVENT line that matches multiple incoming traps, a catcha.......

      Delete
    3. If you were to add it you would add it to /etc/snmp/snmptt.conf. However, depending on the MIB you imported you don't really have to worry about it. The catch all is there for any traps received which aren't covered by the trap definitions imported.

      Delete
    4. thank you for the reply Grant.
      I've done all the steps but I'm not getting the traps.

      how can I test and see If I'm receiving the traps on my nagios server ?

      thank you

      Delete
    5. Make sure that the client its sending the traps, after that you should take a look at the snmptt.log and see whats inside it, if its working you should get a message like this in the log:

      Tue Aug 25 12:07:43 2015 .1.3.6.1.4.1.41385.1.1.2.0.1 Normal "Status Events" xxx.xxx.xxx.xxx - Variables of the dseEventTrap
      3 2 51 1440504240 258854 Fuel Monitoring Event 0 1st

      If not please make sure if your firewalld or iptables can accept incoming traffic at port 162 or check the snmptt.ini file

      Delete
    6. I'm able to walk the snmp mibs on the switch from the nagios core

      my firewall rules are open on port 162.

      [root@Server snmptt]# iptables-save | grep 162
      -A IN_public_allow -p tcp -m tcp --dport 162 -m conntrack --ctstate NEW -j ACCEPT
      -A IN_public_allow -p udp -m udp --dport 162 -m conntrack --ctstate NEW -j ACCEPT
      [root@Server snmptt]#

      this is my config on a cisco switch.

      snmp-server engineID local 1234567891
      snmp-server group G1 v3 auth
      snmp-server group G1 v3 priv access 15
      snmp-server host 10.100.1.14 version 3 auth U1

      User name: U1
      Engine ID: 1234567891
      storage-type: nonvolatile active
      Authentication Protocol: SHA
      Privacy Protocol: AES128
      Group-name: G1

      Delete
  25. Hi,

    Can Nagios handle 1600 traps per second? Any idea on maximum traps per second that Nagios can handle?

    --Ravi

    ReplyDelete
  26. This comment has been removed by the author.

    ReplyDelete
  27. Hi

    Thanks for this.

    I wonder why it shows "Pending" in the nagios GUI all the time?

    Forwarded SNMP Traps

    Active checks of the service have been disabled - only passive checks are being accepted

    PENDING N/A 0d 0h 9m 23s+ 1/1 Service is not scheduled to be checked...

    :-)

    BR Maali

    ReplyDelete
    Replies
    1. Can you paste your nagios configurations? host.cfg and the template.

      Delete
    2. Can you paste your nagios configurations? host.cfg and the template.

      Delete
  28. templates.cfg:
    ###########
    define service{
    name trap-service
    use generic-service
    register 0
    service_description snmp_traps
    is_volatile 1
    check_command check-host-alive ; Used to reset the status to OK when 'Schedule an immediate check of this service' is selected.
    flap_detection_enabled 0 ; Flap detection is disabled
    process_perf_data 0 ; Do not Process performance data
    max_check_attempts 1 ; Leave as 1
    normal_check_interval 1 ; Leave as 1
    retry_check_interval 1 ; Leave as 1
    passive_checks_enabled 1 ; Enables passive checks
    check_period 24x7
    notification_interval 31536000 ; Notification interval.
    active_checks_enabled 0 ; Prevent active checks from occuring as we are only using passive checks.
    notification_options w,u,c ; Notify on warning, unknown and critical.
    contact_groups ngin
    }
    services.cfg:
    ##########
    define service{
    host_name sde-sb1-pe001.net
    use trap-service
    register 1
    }
    hosts.cfg
    ########
    define host{
    host_name sde-sb1-pe001.net
    alias Load Balancer
    address xx.71.xxx.xxx
    hostgroups TeMIP
    use 24x7-host-template
    }

    /etc/hosts
    xx.71.xxx.xxx sde-sb1-pe001.net

    and smmptt and snmptrapd are both running OK.

    snmp traps arrives to the Nagios server ok and are shown in the snmptt.log

    What I am missing is that on Nagios GUI: It looks like below:

    sde-sb1-pe001.net


    snmp_traps

    Active checks of the service have been disabled - only passive checks are being accepted
    PENDING N/A 0d 0h 44m 7s+ 1/1 Service is not scheduled to be checked...


    BR
    Maali


    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Your .cfg files seems to be fine. I think the pending states its because its a passive check and waits for a trap in order for the state to change. Try running the next command, it just changes the PENDING state.

      /usr/local/nagios/libexec/submit_check_result sde-sb1-pe001.net TRAP 2 "TRAP INFO $*"

      The first part is the path of your submit_check_result plugin. the second is the name of the host, the third is the state of your service, 2 is for CRITICAL, I think 1 is for OK. and the last one is a string, usually with information about your trap.
      Did you have your traps working? your nagios server receives traps?

      Delete
    3. I tried, but it did not change state !!!

      The alarm arraives to nagios like below:
      ###########################################################################
      Agent Address: 0.0.0.0
      Agent Hostname: xxx-int.vip.skanova.net
      Date: 12 - 31 - 36 - 25 - 6 - 4461752
      Enterprise OID: .
      Trap Type: Cold Start
      Trap Sub-Type: 0
      Community/Infosec Context: TRAP2, SNMP v2c, community public
      Uptime: 0
      Description: Cold Start
      PDU Attribute/Value Pair Array:
      DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (15972882) 1 day, 20:22:08.82
      SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.140.625.0.65
      SNMPv2-SMI::enterprises.140.625.100.5 = STRING: "Thu Nov 19 09:26:45 UTC 2015"
      SNMPv2-SMI::enterprises.140.625.100.10 = STRING: "xxx-int.vip.skanova.net"

      Delete
  29. The below changes the state to OK on Nagios GUI.
    /usr/local/nagios/libexec/eventhandlers/submit_check_result xxx-int.vip.skanova.net "snmp_traps" 0 "State changed to ok"

    ReplyDelete
    Replies
    1. Maybe I pick the wrong host. But nice to hear that. So, now you have your alarms working when you receive a trap?

      Delete
    2. This comment has been removed by the author.

      Delete
    3. Yes, it seems that you receive the trap. If you receive the trap its because your mib file have imported succesfully using the snmpttconvertmib command. Try to open the output file of the snmpttconvertmib command and paste it here to see if it is OK. This file looks like:

      EVENT dseEventTrap .1.3.6.1.4.1.41385.1.1.2.0.1 "Status Events" Normal
      FORMAT Variables of the dseEventTrap $*
      EXEC /usr/local/nagios/libexec/submit_check_result Generador-Edificio TRAP 2 "dseEventTrap $*"
      SDESC
      Variables of the dseEventTrap
      Variables:
      1: eventType
      Syntax="INTEGER"
      Descr="The Event Type"
      2: eventSubtype
      Syntax="INTEGER"
      Descr="The Event Subtype"
      3: eventParameters
      Syntax="INTEGER"
      Descr="Parameters of the event"
      4: eventTimestamp
      Syntax="INTEGER"
      Descr="Timestamp in UNIX time"
      5: eventHoursRun
      Syntax="INTEGER"
      Descr="Hours run in seconds"
      6: eventDescription
      Syntax=""
      Descr="Brief description of the event"
      7: moduleID
      Syntax="INTEGER"
      Descr="Index of the module that caused the trap"
      8: moduleDescription
      Syntax=""
      Descr="Description of the module that caused the trap"
      EDESC

      Delete
  30. What is the name of the output file? Do you mean snmptt.conf?
    If yes, then it looks like below:

    EVENT sS7PointCodeStatusChangedNotification .1.3.6.1.4.1.111.18.2.1.2 "Status Events" Normal
    FORMAT This notification is sent when the SS7 Remote Pointcode Status Changed $*
    EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "This notification is sent when the SS7 Remote Pointcode Status Changed $*"
    SDESC
    This notification is sent when the SS7 Remote Pointcode Status Changed
    Variables:
    EDESC

    ReplyDelete
  31. another example:

    EVENT wlsWatchNotification .1.3.6.1.4.1.140.625.0.85 "Status Events" Normal
    FORMAT This trap is generated when a watch rule evaluates to true. $*
    EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "This trap is generated when a watch rule evaluates to true. $*"
    SDESC
    This trap is generated when a watch rule evaluates to true.
    Variables:
    1: trapTime
    2: trapDomainName
    3: trapServerName
    4: trapWatchName
    5: trapWatchRuleType
    6: trapWatchRule
    7: trapWatchSeverity
    8: trapWatchData
    9: trapWatchAlarmType
    10: trapWatchAlarmResetPeriod
    11: trapWatchSNMPNotificationName
    EDESC

    ReplyDelete
  32. It works fine now:
    I noticed that I have a mistake in the compiling command :-)
    snmpttconvertmib --in=/CISCO-C2900-MIB.mib --out=/etc/snmp/snmptt.conf --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1'

    The word TRAP is the "Service Name" which is used on Nagios

    ReplyDelete
  33. and the $r parameter is the host. It must match the same host and service. Nice to hear its working!

    ReplyDelete
  34. Yes, and thanks for your very nice guide :-)

    ReplyDelete
  35. Hi,

    I have a problem with sending SNMP traps to Nagios.
    In particular, when I run the next command locally from the server it works ok and I receive the alert to Nagios
    /opt/nagios-4.0.7/contrib/eventhandlers/submit_check_result A6B-switch TRAP 2 “TESTING2”
    When I generate an SNMPv2 alert from a Cisco switch this alert is not displayed to the Nagios although is received to the nagios server (tcpdump the nic card).

    I have load the MIB file by running the next command
    snmpttconvertmib --in=/usr/share/snmp/mibs/IF-MIB.txt --out=/etc/snmp/snmptt.conf --exec='/opt/nagios-4.0.7/contrib/eventhandlers/submit_check_result $r TRAP 2'

    I have to use a different file path from the one described to this web site although i followed these instructions since the submit_check result is located there.

    Could you please help me?

    ReplyDelete
  36. I just executed the first yum step and do not see a vi /etc/sysconfig/snmptrapd.options file.

    ReplyDelete
  37. Hi,

    I want to disable Cold Start SNMP trap from my LINUX Server. Is it possible to do the same.

    Could you please help me??

    ReplyDelete
  38. Is there an updated version of these tutorial? and also is it the same with standalone config (not running in daemon)?

    ReplyDelete
  39. Note that if the EngineID of the sending host is seen in hex format like "80 00 05 55", your snmptrapd.config should have that as "800555" (80 0 5 55).
    This drove me crazy for a week, not knowing why adding the verbose hex engineID to my config wasnt working!

    ReplyDelete
  40. I encountered an issue with a trap from a new host (StoreOnce Server). The values in the incoming got out of order when written out to the trap file. I needed this article [https://support.nagios.com/kb/article.php?id=557] which showed me how to change this line in snmptrapd.conf:
    traphandle default /usr/sbin/snmptthandler
    to...
    perl do /usr/sbin/snmptthandler-embedded
    (Obviously need to move snmptthandler-embedded from the snmpt package to /usr/sbin)

    ReplyDelete
  41. Hi have no idea if this post is still active and it's a great doc. Although I'm having the following issue where nagios is not finding the service:

    EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;centralindia-netscaler02;trap-srv;This trap indicates that the netscaler is now operating in the primary mode. ;
    [1610120533] Warning: Passive check result was received for service 'trap-srv' on host 'centralindia-netscaler02', but the service could not be found!
    [1610120533] Error: External command failed -> PROCESS_SERVICE_CHECK_RESULT;centralindia-netscaler02;trap-srv;This trap indicates that the netscaler is now operating in the primary mode. ;

    ReplyDelete
    Replies
    1. I have the following service setup as per the doc:

      define service {
      host_name centralindia-netscaler02
      use trap-srv
      # hostgroup_name daas-netscalers
      service_description Netscaler::Traps::Alarms
      #notifications_enabled 0
      #contacts cloud-desktop-alerts
      contact_groups daas
      }

      Delete
    2. Hi, I am using Nagiosql so I am not that familiar with defining host n services through cli, but I think you should have the same string in service description and in the exec command.
      for example, my EXEC command in snmptt.conf is:
      /opt/nagios-4.4.6/.../submit_check_result $ar TRAP 1
      and my service name is check-trap,but description of the service is TRAP
      so I believe you should change service description to be trap-srv

      Delete
    3. Hi, many thanks for your reply. It is greatly appreciated.

      I'm pretty certain you don't need a cmd string for this passive check... But your recommendation of using the "service description" rather than the "service name" worked a treat :-) You legend :-)

      So, I can now run manually the EXEC cmd from the /etc/snmp/snmptt.conf which I can see triggers the nagios alert and see it in the nagios log. BUT, when the event comes in which I can see in the snmptt.log file so it has been passed from snmptrapd, it never triggers the nagios EXEC cmd. So, I'm guessing it's not being matched within the /etc/snmp/snmptt.conf!

      What part is it actually matching? do we know

      Delete
    4. Quick update. The snmptt.conf must be correct and it is matching or I wouldn't get the entry in the snmptt.log. In fact I tested it and if I remove the file and restart snmpttd it no longer logs the entry. So this all look correct, however, not sure why it doesn't trigger the EXEC nagios cmd!?

      Delete
  42. Another update: Sorry.

    All resolved it was the following that needed to be changed in the nagios submit_check_result script. Interesting there was nothing in the snmptt debug log to point to this!

    #`$echocmd $cmdline >> $CommandFile`
    sg nagcmd -c "echo '$cmdline' >> $CommandFile"

    Anyway all working now :-)

    ReplyDelete
  43. I eventually got this working on Rocky Linux (a RHEL8 clone) using the Nagios packages from the epel repository.

    In this environment there are a few issues or items that are not clear in this tutorial, some of which have been noticed by other readers.

    1) Snmptrapd was relatively painless. It needed the net-snmp and net-snmp-perl packages installed. I set snmptrapd options to "-On -Lsd"

    2) There were a few missing perl modules needed by snmptt. These will vary depending on your system. I found all of them using 'dnf whatprovides *missingfilename'. Make sure you have enabled the PowerTools repository as well as epel.

    3) /var/log/snmptt directory needed manually creating with write permissions for the snmptt user

    4) Be aware that snmptt.ini may need to be in /etc/snmp/ or at least have a link to it's actual location from there, even if you specify a different location when running the daemon.

    5) Unless I set group write permissions for root on /var/spool/snmptt the trap notifications were not deleted and so were processed infinitely.

    6) When running snmpttconvertmib you need to substitute the value of service_description, from you nagios configuration's service definition for TRAP. In the tutorial this is 'snmp_traps'.

    Also make sure you use the correct path to the event handler, in my case using the file from epel's nagios-contrib package it was /usr/lib64/nagios/plugins/eventhandlers/submit_check_result

    My final command was

    # /usr/local/sbin/snmpttconvertmib \
    --in=/usr/share/snmp/mibs/mymib \
    --out=/etc/snmp/snmptt.conf \
    --exec='/usr/lib64/nagios/plugins/eventhandlers/submit_check_result $r snmp_traps 1'

    7) The event handler, submit_check_result, itself needed modifying. Make sure the CommandFile option is set as follows:
    CommandFile="/var/spool/nagios/cmd/nagios.cmd"

    8) I needed to give group write permissions to the snmptt user on the named pipe /var/spool/nagios/cmd/nagios.cmd to allow the snmptt daemon to submit notifications to Nagios.

    ReplyDelete