logIt » snmp

Zabbix Proxy: distributed NMS monitoring via embedded Linux (Ubuntu on BeagleBoard)

Arif — Thu, 19 Jul 2012 07:26:15 +0000

As already mentioned in previous post’s introduction, proof of concept on how Zabbix Proxy works under unreliable communication is what this next post about. The idea is to have these scenarios tested:

independent SNMP data polling by individual NMS proxy (embedded system)
intermittent connection between main/master NMS server with its proxy

Of course we’ll do them in implementation context of Zabbix Server as main/master NMS server and our BeagleBoard xM as its proxy.

Zabbix Proxy - distributed NMS over embedded Linux

In the technical details, I already followed pretty much the same steps showed in installation wiki (with the exception that I didn’t put the process into startup init.d script as I would start it manually as needed).

The first test is to make our Ubuntu ARM board accepts configuration done in Zabbix Server web-GUI, retrieves SNMP data to it, and sends collected data to server. Mode passive is chosen so that the server will be the one initiating contact with the board. After enabling a certain SNMP host, i.e. host-aaa, as monitored-by-proxy, the Zabbix Server log will show:

15677:20120601:022022.960 Sending configuration data to proxy 'beaglexm1'. Datalen 9205

Our board, ‘beaglexm1‘, will start to poll SNMP data from host-xxx to then store it in designated sqlite database.

975:20120531:212024.098 proxy #8 started [poller #5]
984:20120531:212024.128 proxy #17 started [discoverer #1]
968:20120531:212025.423 Received configuration data from server. Datalen 9205
971:20120531:212030.388 Enabling SNMP host [host-aaa]

From time to time as configured, our board will also run housekeeping to keep our board database file in small size.

1081:20120531:224513.922 Executing housekeeper
1081:20120531:224514.505 Deleted 4885 records from history [0.453765 seconds]

Under normal connection, in main Zabbix Server we’ll start seeing the collected data from host-aaa the same way as when the SNMP data is polled by the server directly. To emulate disruption to the connectivity, I choose to use simple iptables drop packet. This time the server will no longer seeing new data.

After the drop block is released, the server we’ll start seeing new data as well as data from the period of intermittent connection. It will take some time before all of that being sent to the server.

In general that proves it. I collected 44 OID values from the host-aaa every 30 seconds and in this case BeagleBoard utilization was very low as this wasn’t a stress test to see its performance.

Compile Zabbix Proxy in Ubuntu ARM (BeagleBoard)

Arif — Tue, 17 Jul 2012 08:28:57 +0000

I needed to tackle practical limitation of SNMP monitoring under unreliable communication, a serious consideration was made for Zabbix Proxy. It was an option said to be ready for embedded hardware. I already had BeagleBoard xM Rev C running Ubuntu 11.10 Oneiric and needed to proof that it would port functionally to this Linux ARM board. There existed zabbix-proxy-mysql in Oneiric repository, but sqlite seemed to be a better scale option for embedded deployment.

Zabbix Proxy Distributed NMS

An example set out for Zabbix Proxy on Debian appears to be pretty straightforward to follow. With the ARM board already installed with build-essential, snmp, and snmpd, the following needs to be added:

$ sudo apt-get install libsnmp-dev libcurl4-openssl-dev fping curl sqlite libsqlite3-dev ntpdate

then start the pre-compilation configure (in this case Zabbix 1.8.7 is used):

cd zabbix-1.8.7
./configure --enable-proxy --enable-agent --with-sqlite3 --with-net-snmp --with-libcurl

It will go smooth. However, during make there are complains about undefined references to ‘sqlite3_open‘, ‘sqlite3_close‘, etc.:

../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_connect':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:331: undefined reference to `sqlite3_open'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:343: undefined reference to `sqlite3_busy_timeout'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:334: undefined reference to `sqlite3_errmsg'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:335: undefined reference to `sqlite3_close'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_init':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:385: undefined reference to `sqlite3_open'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:387: undefined reference to `sqlite3_errmsg'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_close':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:446: undefined reference to `sqlite3_close'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_vexecute':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:827: undefined reference to `sqlite3_exec'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:852: undefined reference to `sqlite3_changes'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:833: undefined reference to `sqlite3_free'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `SQ_DBfree_result':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:1363: undefined reference to `sqlite3_free_table'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_vselect':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:1121: undefined reference to `sqlite3_get_table'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:1127: undefined reference to `sqlite3_free'

this is obviously from the database related code compilation. I run the gcc manually for the db.c file inside its directory by adding the flag ‘-lsqlite3‘ in the tail of ‘../../src/libs/zbxdb/libzbxdb.a‘ to make it work. Afterward by going to the initial directory first, running make again will then continue successfully.

$ cd src/zabbix_proxy
$ gcc -DZABBIX_DAEMON -g -O2     -L/usr/lib/arm-linux-gnueabi    -I/usr/local/include -I/usr/lib/perl/5.12/CORE -I. -I/usr/include      -I/usr/include -lsqlite3  -L/usr/lib/arm-linux-gnueabi -lcurl  -L/usr/lib -lnetsnmp -lcrypto  -L/usr/lib -lnetsnmp -lcrypto   -rdynamic    -o zabbix_proxy zabbix_proxy-servercomms.o zabbix_proxy-events.o zabbix_proxy-zlog.o zabbix_proxy-proxy.o heart/libzbxheart.a ../../src/zabbix_server/dbsyncer/libzbxdbsyncer.a ../../src/zabbix_server/discoverer/libzbxdiscoverer.a housekeeper/libzbxhousekeeper.a ../../src/zabbix_server/httppoller/libzbxhttppoller.a proxyconfig/libzbxproxyconfig.a ../../src/zabbix_server/pinger/libzbxpinger.a ../../src/zabbix_server/poller/libzbxpoller.a ../../src/zabbix_server/trapper/libzbxtrapper.a ../../src/zabbix_server/nodewatcher/libzbxnodewatcher.a datasender/libzbxdatasender.a ../../src/zabbix_server/selfmon/libzbxselfmon.a ../../src/libs/zbxsysinfo/libzbxserversysinfo.a ../../src/libs/zbxsysinfo/linux/libspecsysinfo.a ../../src/libs/zbxsysinfo/common/libcommonsysinfo.a ../../src/libs/zbxsysinfo/simple/libsimplesysinfo.a ../../src/libs/zbxlog/libzbxlog.a ../../src/libs/zbxdbcache/libzbxdbcache.a ../../src/libs/zbxmemory/libzbxmemory.a ../../src/libs/zbxalgo/libzbxalgo.a ../../src/libs/zbxnix/libzbxnix.a ../../src/libs/zbxsys/libzbxsys.a ../../src/libs/zbxconf/libzbxconf.a ../../src/libs/zbxcommon/libzbxcommon.a ../../src/libs/zbxcrypto/libzbxcrypto.a ../../src/libs/zbxcomms/libzbxcomms.a ../../src/libs/zbxcommshigh/libzbxcommshigh.a ../../src/libs/zbxjson/libzbxjson.a ../../src/libs/zbxexec/libzbxexec.a ../../src/libs/zbxself/libzbxself.a ../../src/libs/zbxserver/libzbxserver.a ../../src/libs/zbxicmpping/libzbxicmpping.a ../../src/libs/zbxdbhigh/libzbxdbhigh.a ../../src/libs/zbxdb/libzbxdb.a -lsqlite3 -lcurl  -lnetsnmp    -lm  -lresolv
$ cd ../../
$ make

SNMP: Mistake of Using Tabular OID in Practice

Arif — Tue, 20 Mar 2012 07:57:58 +0000

Recently, I retrieve rectifier-related SNMP data from Eltek’s Smartpack Controller. This one has the following rear type where we can connect to the ethernet plug (see the purple area).

Eltek Smartpack Controller rear view (type that comes with SNMP support)

It is presumably in its default configuration and I can snmpwalk through its corporate specific OIDs (under 12148 tree) to find i.e. output voltage, current, battery temperature, etc. Doing it without translating against the MIB will retrieve all the values perfectly fine. However, I want to show you a mistake in practice related to how the MIB is written and put to place, which is a version 4 by the name of ELTEK-DISTRIBUTED-PLANTV4-MIB. It has tabular part as clipped in the following tree (see previous post on how to write such part in your own MIB):

            |     +--eltek(12148)
            |        |
            |        +--eltekDistributedPlantV4(9)
...
            |           |
            |           |     |
            |           |     +--rectifierStatusTable(2)
            |           |        |
            |           |        +--rectifierStatusEntry(1)
            |           |           |  Index: rectifierStatusID
            |           |           |
            |           |           +-- -R-- Integer32 rectifierStatusID(1)
            |           |           |        Range: 1..100
...
            |           |           +-- -R-- Integer32 rectifierStatusOutputVoltage(4)
            |           |           |        Range: 0..65535

Looking at the above rectifierStatusEntry branch, it marks a tabular entry expected to range from 1 to 100 as defined by rectifierStatusID. Hence, putting 0 as one of the output voltage index will confuse the OID reported by GET as

belong to the scalar type (a misconception that is)
belong to the tabular type but out of index range

GET against the MIB will return the following error

$ sudo snmpget -m +ELTEK-DISTRIBUTED-PLANTV4-MIB -v 1 -c public raddlex ELTEK-DISTRIBUTED-PLANTV4-MIB::rectifierStatusOutputCurrent.0
ELTEK-DISTRIBUTED-PLANTV4-MIB::rectifierStatusOutputCurrent.0: Unknown Object Identifier (Index out of range: 0 (rectifierStatusID))

while the item (output current) actually has value in it as found below

$ sudo snmpget -v 1 -c public raddlex .1.3.6.1.4.1.12148.9.5.5.2.1.3.0
SNMPv2-SMI::enterprises.12148.9.5.5.2.1.3.0 = INTEGER: 1

By this time the MIB is already the latest to be used with firmware version 4.3.

Write Your Own MIB File (Tabular)

Arif — Fri, 09 Sep 2011 11:23:19 +0000

When it comes to writing our own MIB, a practical choice might be to write the whole objects as of scalar type (RFC-1212). That means, let’s say we have 2 battery statuses, each of them must be explicitly written individually within the MIB file. Example of calling their voltages:

$ snmpwalk -m +REKTRONIK-MIB -v 1 -c public monitored-host .1.3.6.1.4.1.38610
...
REKTRONIK-MIB::battery1Voltage.0 = INTEGER: 9
REKTRONIK-MIB::battery2Voltage.0 = INTEGER: 11
...

If additional 3rd battery is to be monitored, REKTRONIK-MIB.txt in this example must be modified. In small scale we don’t need consistent MIB as changes are manageable, but if a whole bunch of different people must reference to this, a better way is to have tabular object where the above batteries are indexed instead of ended with “.0” scalar identifier.

Quick approach by checking established LM-SENSORS-MIB.txt shows at least we need three object definitions inside the MIB file:

Components of writing tabular MIB

Its stemmed MIB tree to show the (1) Table, (2) Entry, and (3) Index:

...
            |     |  |     +--lmFanSensorsTable(3)
            |     |  |     |  |
            |     |  |     |  +--lmFanSensorsEntry(1)
            |     |  |     |     |  Index: lmFanSensorsIndex
            |     |  |     |     |
            |     |  |     |     +-- -R-- Integer32 lmFanSensorsIndex(1)
            |     |  |     |     |        Range: 0..65535
...

Now the writing part of those three types to our MIB, plus the batteryVoltage which is the actual object to be indexed in the final SNMP monitoring.

batteryTable OBJECT-TYPE
    SYNTAX     SEQUENCE OF BatteryEntry
    MAX-ACCESS not-accessible
    STATUS     current
    DESCRIPTION
        "The (conceptual) table of batteries contained by the
        module."
    ::= { monitor 1 }
 
batteryEntry OBJECT-TYPE
    SYNTAX     BatteryEntry
    MAX-ACCESS not-accessible
    STATUS     current
    DESCRIPTION
        "A (conceptual) entry for one battery contained by
        the module.  The batteryIndex in the index represents
        the entry in the batteryTable that corresponds to the
        batteryEntry.
 
        As an example of how objects in this table are named,
        an instance of the batteryVoltage object might be
        named batteryVoltage.3"
    INDEX { batteryIndex }
    ::= { batteryTable 1 }
 
batteryIndex OBJECT-TYPE
    SYNTAX      DeviceIndex
    MAX-ACCESS  read-only
    STATUS      current
    DESCRIPTION
            "A unique value, greater than zero, for each battery. 
            It is recommended that values are assigned contiguously
            starting from 1."
    ::= { batteryEntry 1 }
 
batteryVoltage    OBJECT-TYPE 
   SYNTAX Integer32 
   ACCESS read-only 
   STATUS current 
   DESCRIPTION 
      "Voltage A/D value" 
   ::= { batteryEntry 2 }

There are some header parts of course, check the resulting REKTRONIK-MIB.txt here.

The tree view becomes:

            +--private(4)
            |  |
            |  +--enterprises(1)
            |     |
            |     +--rektronik(39559)
...
            |        |
            |        +--monitor(2)
            |        |  |
            |        |  +--batteryTable(1)
            |        |  |  |
            |        |  |  +--batteryEntry(1)
            |        |  |     |  Index: batteryIndex
            |        |  |     |
            |        |  |     +-- -R-- Integer32 batteryIndex(1)
            |        |  |     |        Textual Convention: DeviceIndex
            |        |  |     |        Range: 1..2147483647
            |        |  |     +-- -R-- Integer32 batteryVoltage(2)
            |        |  |     +-- -R-- Integer32 batteryCurrent(3)
            |        |  |
            |        |  +--environment(2)
            |        |  |  |
            |        |  |  +-- -R-- Integer32 temperature(1)
            |        |  |  +-- -R-- Integer32 humidity(2)
...

and example of getting SNMP is

$ snmpwalk -m +REKTRONIK-MIB -v 1 -c public monitored-host .1.3.6.1.4.1.38610
REKTRONIK-MIB::name.0 = STRING: "RekMini"
REKTRONIK-MIB::version.0 = STRING: "1.0"
REKTRONIK-MIB::date.0 = STRING: "2011-09-08"
REKTRONIK-MIB::batteryIndex.22 = INTEGER: 22
REKTRONIK-MIB::batteryIndex.23 = INTEGER: 23
REKTRONIK-MIB::batteryVoltage.22 = INTEGER: 9
REKTRONIK-MIB::batteryVoltage.23 = INTEGER: 11
REKTRONIK-MIB::batteryCurrent.22 = INTEGER: 2
REKTRONIK-MIB::batteryCurrent.23 = INTEGER: 2
...

in the above example any additional 3rd battery voltage insertion will be OID “REKTRONIK-MIB::batteryVoltage.24“

Running Raddle (2): snmpd & Replay Custom Private Enterprise MIB

Arif — Thu, 18 Aug 2011 03:13:44 +0000

Net-SNMP wiki gives tutorial on extending snmpd using Perl. In basic run of Raddle (previous post) this involves r1.pl called by r1.conf fed to snmpd. Private enterprise MIB i.e. Microchip’s in this example can be replayed without really ever captured the real device’s SNMP data. This is necessary when the actual device is yet existed; normal situation when developing. There are two simple ways of replaying this artificial SNMP data over which either way requires correct OID.

First put the Microchip.TXT (the MIB file) under /usr/share/snmp/mibs to easily exchange OID canonical form and name vice versa while making dummy. In this case I’ve modify the MIB to have some additional object i.e. name:
$ snmptranslate -m +Microchip -On Microchip::name .1.3.6.1.4.1.17095.1.1
so we can then use this correct OID for our dummy.

How?

use SetMIBValue() in r1.pl

$agent->SetMIBValue( '.1.3.6.1.4.1.17095.1.1', ASN_OCTET_STR, "Microchip");
$agent->SetMIBValue( '.1.3.6.1.4.1.17095.3.13', ASN_INTEGER, 82);

Feed r1.snmp for playing as in

$agent->ParseDataFile( '/usr/local/etc/snmp-emulator/r1.snmp', 0 )

with the content of

.1.3.6.1.4.1.17095.1.1 = STRING: Microchip
.1.3.6.1.4.1.17095.3.13 = INTEGER: 82

In all above two examples we can only GET for Raddle to return an answer for each OID. GETNEXT as in bulk snmpwalk won’t give output because they are not sequenced.

A sample of sequenced data inside a real snmpwalk dump is:

$ snmpwalk -v 1 -c public -On localhost
 
.1.3.6.1.2.1.1.1.0 = STRING: Linux xp-racy 2.6.38-10-generic #46~lucid1-Ubuntu SMP Wed Jul 6 18:40:11 UTC 2011 i686
.1.3.6.1.2.1.1.2.0 = OID: .1.3.6.1.4.1.8072.3.2.10
.1.3.6.1.2.1.1.3.0 = Timeticks: (496371) 1:22:43.71
.1.3.6.1.2.1.1.4.0 = STRING: Root  (configure /etc/snmp/snmpd.local.conf)
.1.3.6.1.2.1.1.5.0 = STRING:
.1.3.6.1.2.1.1.6.0 = STRING: Unknown (configure /etc/snmp/snmpd.local.conf)
.1.3.6.1.2.1.1.8.0 = Timeticks: (0) 0:00:00.00
.1.3.6.1.2.1.1.9.1.2.1 = OID: .1.3.6.1.6.3.10.3.1.1
.1.3.6.1.2.1.1.9.1.2.2 = OID: .1.3.6.1.6.3.11.3.1.1

thus, we can only

$ snmpget -m +Microchip -v 1 -c public localhost Microchip::control.13
Microchip::control.13 = INTEGER: 82

or from other system with no Microchip MIB use

$ snmpget -v 1 -c public 192.168.40.105 .1.3.6.1.4.1.17095.3.13
SNMPv2-SMI::enterprises.17095.3.13 = INTEGER: 82

The tree below may visually explains sequenced data for GETNEXT where our Microchip is branched at private.enterprises.microchip :

$ snmptranslate -m +Microchip -Tp
 
+--iso(1)
   |
   +--org(3)
      |
      +--dod(6)
         |
         +--internet(1)
            |
            +--directory(1)
            |
            +--mgmt(2)
            |  |
            |  +--mib-2(1)
            |     |
            |     +--system(1)
            |     |  |
...
            +--private(4)
            |  |
...
            |  +--enterprises(1)
            |     |
...
            |     +--microchip(17095)
            |        |
            |        +--product(1)
            |        |  |

How GET and GETNEXT appeared in SNMP can be viewed by running snmpd in debug mode with this options:

$ snmpd -m +Microchip -f -L -V -C -I vacm_vars -c /usr/local/etc/snmp-emulator/r1.conf
NET-SNMP version 5.3.1
Connection from UDP: [127.0.0.1]:32770
Received SNMP packet(s) from UDP: [127.0.0.1]:32770
  GET message
    -- control.13

Running Raddle in RHEL

Arif — Mon, 15 Aug 2011 07:43:05 +0000

Raddle is a great SNMP player written in Perl. I say player to this emulator application, meaning that I can append or dump snmpwalk output from a device and then replay that from Raddle. Installing it can be both simple and tedious tasks. I used Red Hat Enterprise Linux 5.2 inside a virtual machine (VirtualBox) to make it compact without interfering my own host snmpd setting.

net-SNMP packages are required for Raddle to work on top of primarily snmpd.

yum install net-snmp net-snmp-utils

Prior to installation some rpm packages must be downloaded separately as my local repo (created from original RHEL CD images) may not contain them. They are installed as follow:

rpm -i perl-Carp-Clan-6.00-1.el4.rf.noarch.rpm
rpm -i perl-Bit-Vector-6.4-2.el5.rf.i386.rpm
rpm -i perl-Crypt-DES-2.05-3.2.el5.rf.i386.rpm
rpm -i perl-Digest-SHA1-2.11-1.el5.rf.i386.rpm
rpm -i perl-Digest-HMAC-1.01-2.2.el5.rf.noarch.rpm
rpm -i perl-Socket6-0.20-1.el5.rf.i386.rpm
rpm -i perl-Net-SNMP-5.2.0-1.2.el5.rf.noarch.rpm
rpm -i perl-Date-Pcalc-1.2-1.2.el5.rf.noarch.rpm
rpm -i perl-Date-Calc-5.4-1.el5.rf.i386.rpm

continued by installing net-snmp-perl from original repo

yum install net-snmp-perl

Uncompressing Raddle and successful make test will mean installation is good and can be finished by make install

cd Net-Raddle-0.08
perl Makefile.PL
make test
make install

Replaying SNMP Packets

The idea is to run snmpd which replays a specific prepared-SNMP-packets instead of our own real time system SNMP data (in this case the condition of RHEL 5.2 host). There are three files to do that, let’s name them r1.*:

r1.snmp
A capture from i.e.
snmpwalk -v 1 -c -OneU > r1.snmp
r1.conf
snmpd will run this instead of the original OS default:
snmpd -C -I vacm_vars -c /usr/local/etc/snmp-emulator/r1.conf
r1.pl
the perl script where the most basic is to just replay the above r1.snmp

An official documentation to replay SNMP can be found here. I got hold back when trying to listen from other host, it turns out that I must remove the localhost section in r1.conf which will make it to use standard port 161 as well

# Listen on port 9501
#agentaddress    udp:localhost:9501,tcp:localhost:9501

It is also wise to always cut the r1.snmp first to see if the perl works (test with perl -w /usr/local/etc/snmp-emulator/r1.pl) either with warning or clean, meaning no problem with just some of the OIDs found inside the file.

Configuring Cacti

Arif — Wed, 02 Sep 2009 05:55:04 +0000

A successful installation in a LAMP system may require more library. I added php5-cli to enable php through command line i.e $ /usr/bin/php , libphp-adodb, php5-snmp.

In localhost snmpd is installed and configured /etc/default/snmpd:

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid'

and edit /etc/snmp/snmpd.conf to have:

com2sec readonly  localhost         <some community string>

Install snmp, then I can test snmpwalk:

$ snmpwalk -Os -c <some community string> -v 1 localhost system

Cacti requires RRDTool and the correct version to be set in Settings menu of the Cacti web console. In my case, RRDTool version is 1.3x. In Cacti 0.8.6.x and above poller.php is called by cron instead of cmd.php, put this trough crontab in Cacti user (enter this Cacti user crontab from crontab -e -u or embed the user in crontab by root instead of individually edited).

*/2 * * * * /usr/bin/php /var/www/cacti/poller.php

It will generate file under rra directory of the web /var/www/cacti/rra/.rrd. RRD tool will create image based on this polling result file.