If this than that recipes alert for Commuter Train trouble

The ingredient is tweet, Twitter Search use case is no longer just sentiment analysis, its function extends to alert, even close to real time–I’m pretty optimistic given the number of Commuter Line users joining the conversation. Now, the no-code part of our recipe, a Twitter search query, that is

(to:krlmania OR @krlmania OR from:CommuterLine)
AND (gangguan OR pantograf OR wesel OR patah)
-semoga -smoga -moga

and a brief explanation is that it queries for some keywords for potential trouble within mentions/replies to @krlmania and official announcements of @CommuterLine. Exclusion added to the search, the sentiment part of tweets, expressing wishes to shoo trouble away–I’m sorry, but keep praying online anyway.

When Twitter launched Twitter Stories in 2011, I read about Ravi Pina running the crowd sourced @Caltrain. Later now I see that official accounts like @SMRT_Singapore is also a model of service disruption info center. We can go deeper to better stats from the API web service that shows real time position (an app is out there showing that the location data API is working), but until its open data status is known, crowd sourcing stays as the most reliable solution, independent to regulation, and easiest to implement. Does the recipe work? During the first week of testing, alerts were flooding–within limit of course–as trouble really happened. So, cook your own ingredients or get my recipes served here.

Battery power bank, WiFi USB stick, and thumb drive

I then turn away to motor-less small storage, a 32 Gigs USB thumb drive so I can move on to other concern: on-the-fly encryption. Hence, I follow instructions to install True Crypt on Raspberry Pi. After unpacking of wxWidgets-2.8.12.tar.gz and TrueCrypt 7.1a Source.tar.gz in place, putting header files from pkcs-11-cryptoki2.20, and then install libfuse-dev, the following make will require long time:

$ export PKCS11_INC=/usr/local/src/truecrypt/pkcs/
 
$ make NOGUI=1 WX_ROOT=/usr/src/wxWidgets wxbuild
Configuring wxWidgets library...
Building wxWidgets library...
/usr/src/wxWidgets/src/common/string.cpp:84:39: warning: ‘wxEmptyString’ initialized and declared ‘extern’ [enabled by default]

$ make NOGUI=1 WX_ROOT=/usr/src/wxWidgets wxbuild
Compiling Buffer.cpp
Compiling Exception.cpp
Compiling Event.cpp
...
../Crypto/Aeskey.c:527:25: warning: operation on ‘ss[7]’ may be undefined [-Wsequence-point]
../Crypto/Aeskey.c:527:25: warning: operation on ‘ss[7]’ may be undefined [-Wsequence-point]
../Crypto/Aeskey.c:527:25: warning: operation on ‘ss[7]’ may be undefined [-Wsequence-point]
...
Converting Language.xml
Compiling Resources.cpp
Linking truecrypt

I created the TrueCrypt volume separately via its desktop GUI with ext3 file system to then mount it in Pi to a configured Samba share. As pointed out in a post, the following changes are added to smb.conf

...
wins support = yes
...
[pitruecrypt]
   comment= Pi Truecrypt Volume
   path=<the mount path of the USB thumb drive TrueCrypt volume>
   browseable=Yes
   writeable=Yes
   only guest=no
   create mask=0777
   directory mask=0777
   public=no

and then user-password are entered via interactive command.

For mobility, I already had the Pi as WiFi access point using hostapd (check these steps) and power bank, so it’s now matter of performance. In the case of rsync, initial sync of some 1,500 items totaling in 1 Gig size elapses in approximately the same 12 minutes of time compared to one bulk file of the same size. Of course, over the next incremental sync, it only takes less than a minute for the thousand items to just update slight differences.

Security

Back to security, there surely risk by opening Samba share to mounted TrueCrypt volume. But, for me it would be practically manageable (cross my finger). There is more concern to the fate of TrueCrypt after it is being closed in such a weird way, given that last audit finds nothing severe. Anyway, I found brute force tool, but no critical attack exists currently, unless e.g. it stays powered on and mounted, the person gain physical access. Beats me again.

A matrix keypad instance is defined by:

1. Actual GPIO pins used forming the row and column of the m x n matrix
2. Individual character in-use as symbol for each button

Hence, I instantiate and call method like

...
QPad  = matrixQPi(keyPad=keyPad,row=row,col=col)
print QPad.scanQ()

to print the character being pressed. Some examples pushed to my github explains how the above keyPad, row, and col are defined to scan-read pressed button of 2×2, 2×3, and 4×3 matrix keypads (or have it 3×4 matrix keypads in other words).

Illustration of any m-by-n matrix: 4x3, 2x2, and 2x3 keypad with different button symbols & GPIO combinations

I used deprecated Wiring-Pi Python (they already moved to 2.x version) without problem. However, you’ll fail building from latest commit and must use combination of older commits as described by my updated part of an old-post. By the way, there’s I/O expander support for WiringPi2-Python which is good, considering:

GPIO is expensive and for the sake of a keypad, you should not spend all.

(A friend told me that once)

First put the Microchip.TXT (the MIB file) under /usr/share/snmp/mibs to easily exchange OID canonical form and name vice versa while making dummy. In this case I’ve modify the MIB to have some additional object i.e. name:
$ snmptranslate -m +Microchip -On Microchip::name
.1.3.6.1.4.1.17095.1.1
so we can then use this correct OID for our dummy.

How?

1. use SetMIBValue() in r1.pl

   $agent->SetMIBValue( '.1.3.6.1.4.1.17095.1.1', ASN_OCTET_STR, "Microchip");
   $agent->SetMIBValue( '.1.3.6.1.4.1.17095.3.13', ASN_INTEGER, 82);

2. Feed r1.snmp for playing as in

   $agent->ParseDataFile( '/usr/local/etc/snmp-emulator/r1.snmp', 0 )

   with the content of

   .1.3.6.1.4.1.17095.1.1 = STRING: Microchip
   .1.3.6.1.4.1.17095.3.13 = INTEGER: 82

In all above two examples we can only GET for Raddle to return an answer for each OID. GETNEXT as in bulk snmpwalk won’t give output because they are not sequenced.

A sample of sequenced data inside a real snmpwalk dump is:

$ snmpwalk -v 1 -c public -On localhost
 
.1.3.6.1.2.1.1.1.0 = STRING: Linux xp-racy 2.6.38-10-generic #46~lucid1-Ubuntu SMP Wed Jul 6 18:40:11 UTC 2011 i686
.1.3.6.1.2.1.1.2.0 = OID: .1.3.6.1.4.1.8072.3.2.10
.1.3.6.1.2.1.1.3.0 = Timeticks: (496371) 1:22:43.71
.1.3.6.1.2.1.1.4.0 = STRING: Root  (configure /etc/snmp/snmpd.local.conf)
.1.3.6.1.2.1.1.5.0 = STRING:
.1.3.6.1.2.1.1.6.0 = STRING: Unknown (configure /etc/snmp/snmpd.local.conf)
.1.3.6.1.2.1.1.8.0 = Timeticks: (0) 0:00:00.00
.1.3.6.1.2.1.1.9.1.2.1 = OID: .1.3.6.1.6.3.10.3.1.1
.1.3.6.1.2.1.1.9.1.2.2 = OID: .1.3.6.1.6.3.11.3.1.1

thus, we can only

$ snmpget -m +Microchip -v 1 -c public localhost Microchip::control.13
Microchip::control.13 = INTEGER: 82

or from other system with no Microchip MIB use

$ snmpget -v 1 -c public 192.168.40.105 .1.3.6.1.4.1.17095.3.13
SNMPv2-SMI::enterprises.17095.3.13 = INTEGER: 82

The tree below may visually explains sequenced data for GETNEXT where our Microchip is branched at private.enterprises.microchip :

$ snmptranslate -m +Microchip -Tp
 
+--iso(1)
   |
   +--org(3)
      |
      +--dod(6)
         |
         +--internet(1)
            |
            +--directory(1)
            |
            +--mgmt(2)
            |  |
            |  +--mib-2(1)
            |     |
            |     +--system(1)
            |     |  |
...
            +--private(4)
            |  |
...
            |  +--enterprises(1)
            |     |
...
            |     +--microchip(17095)
            |        |
            |        +--product(1)
            |        |  |

How GET and GETNEXT appeared in SNMP can be viewed by running snmpd in debug mode with this options:

$ snmpd -m +Microchip -f -L -V -C -I vacm_vars -c /usr/local/etc/snmp-emulator/r1.conf
NET-SNMP version 5.3.1
Connection from UDP: [127.0.0.1]:32770
Received SNMP packet(s) from UDP: [127.0.0.1]:32770
  GET message
    -- control.13

net-SNMP packages are required for Raddle to work on top of primarily snmpd.

yum install net-snmp net-snmp-utils

Prior to installation some rpm packages must be downloaded separately as my local repo (created from original RHEL CD images) may not contain them. They are installed as follow:

rpm -i perl-Carp-Clan-6.00-1.el4.rf.noarch.rpm
rpm -i perl-Bit-Vector-6.4-2.el5.rf.i386.rpm
rpm -i perl-Crypt-DES-2.05-3.2.el5.rf.i386.rpm
rpm -i perl-Digest-SHA1-2.11-1.el5.rf.i386.rpm
rpm -i perl-Digest-HMAC-1.01-2.2.el5.rf.noarch.rpm
rpm -i perl-Socket6-0.20-1.el5.rf.i386.rpm
rpm -i perl-Net-SNMP-5.2.0-1.2.el5.rf.noarch.rpm
rpm -i perl-Date-Pcalc-1.2-1.2.el5.rf.noarch.rpm
rpm -i perl-Date-Calc-5.4-1.el5.rf.i386.rpm

continued by installing net-snmp-perl from original repo

yum install net-snmp-perl

Uncompressing Raddle and successful make test will mean installation is good and can be finished by make install

cd Net-Raddle-0.08
perl Makefile.PL
make test
make install

Replaying SNMP Packets

The idea is to run snmpd which replays a specific prepared-SNMP-packets instead of our own real time system SNMP data (in this case the condition of RHEL 5.2 host). There are three files to do that, let’s name them r1.*:

1. r1.snmp
   A capture from i.e.
   snmpwalk -v 1 -c <community string> -OneU <device> > r1.snmp
2. r1.conf
snmpd will run this instead of the original OS default:
   snmpd -C -I vacm_vars -c /usr/local/etc/snmp-emulator/r1.conf
3. r1.pl
   the perl script where the most basic is to just replay the above r1.snmp

An official documentation to replay SNMP can be found here. I got hold back when trying to listen from other host, it turns out that I must remove the localhost section in r1.conf which will make it to use standard port 161 as well

# Listen on port 9501
#agentaddress    udp:localhost:9501,tcp:localhost:9501

It is also wise to always cut the r1.snmp first to see if the perl works (test with perl -w /usr/local/etc/snmp-emulator/r1.pl) either with warning or clean, meaning no problem with just some of the OIDs found inside the file.

Both of course based on .htaccess manipulation (mod_rewrite of Apache must be enabled). I install Yourls in side http://my.web/somedir. To make my short URL to use something like http://my.web/xGf12 instead of http://my.web/somedir/xGf12, I run the regular install with that somedir as based URL, but then create .htaccess in http://my.web/.htaccess (instead of http://my.web/somedir/.htaccess) However, I intend to use prefix x for the shorten URL. Here is how the regular expression implemented:

# BEGIN YOURLS
 
RewriteEngine On
RewriteBase /somedir/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(x[0-9A-Za-z]+)/?$ /somedir/yourls-go.php?id=$1 [L]
RewriteRule ^(x[0-9A-Za-z]+)\+/?$ /somedir/yourls-infos.php?id=$1 [L]
RewriteRule ^(x[0-9A-Za-z]+)\+all/?$ /somedir/yourls-infos.php?id=$1&all=1 [L]
 
# END YOURLS

I decided to install in RedHat 5.2. Maybe it is easier in Ubuntu (Nagios 3 already listed in the repository, but I haven’t tried it). As pointed out by the quickstart (it is meant for Fedora by the way), requirements are:

1. php, httpd
2. gcc, glibc, glibc-common
3. gd, gd-devel

After getting those, the configure-make-install procedure runs well, except that I don’t check the warnings. It turns out that only some of Nagios core components are installed. It can be seen from the content of /usr/local/nagios/libexec directory. There is no check_snmp etc. Error also appears in statusmap page of Nagios.

Compile More Nagios Core Components

I run through ./configure once more, check out all warnings, and begin to evaluate requirements (found this text inside nagios-plugins-1.4.14 tarball)

MySQL

For check_mysql functionality, mysql (MySQL client for linux) must be installed. ./configure also test this client, by default config, to the local MySQL socket, hence mysql-server must also exist.

Radius Client

Client from freeradius package doesn’t meet Nagios requirements. It must be the ones from original radiusclient, I found RPMs for radiusclient-0.3.2-0.2.el5.rf.i386.rpm and radiusclient-devel-0.3.2-0.2.el5.rf.i386.rpm.

GD

It is quite tricky to get statusmap.c to be compiled for the .cgi of Nagios status map. Compilation needs gd library from http://www.libgd.org/. gd requires zlib, libjpeg, and libpng. On first attempt, I installed from RPMs. This appeared to be not working as I could still see warnings about gd2 during ./configure.

So, I choose to compile from the source gd-2.0.33.tar.bz2. This tarball was found in the older section of current gd2 web. I chose this version because the RPM used was gd-2.0.33-9.4.el5_1.1.rpm. I removed this RPM first which made me to remove also libmwf and ImageMagick.

After straightforward  configure-make-install the library will be installed under /usr/local/lib i.e libgd.so.2.0.0. Header files will also be available in /usr/local/include i.e gd.h. These are different from the situation when I installed from RPM where libgd.so.2.0.0 is under /usr/lib/ and no gd.h found under /usr/include.

So, I make devclean and rerun

./configure --with-gd-lib=/usr/local/lib --with-gd-inc=/usr/local/include --with-command-group=nagcmd

I configure the /usr/local/nagios/etc/cgi.conf but status map page is still empty. Next workaround is to install back  gd-2.0.33-9.4.el5_1.1.rpm. (which also bring back libmwf and ImageMagick) and point the gd library to the ones created from RPM. I make devclean and rerun:

./configure --with-gd-lib=/usr/lib --with-gd-inc=/usr/local/include --with-command-group=nagcmd

At this point I regret not to have Ubuntu implementation in the first place. Just a small regret.

Other library needed for check_smtp, check_http, etc. don’t seem to trouble me much, I need openssl, openldap, etc. and simply get them from RPMs.

Inside hijra-0.1.18 run $ sudo python setup.py install

It’d fail as it couldn’t find HijriApplet. Rename HijriApplet.py to HijriApplet to make it work. Now, it is in path (/usr/bin/) and can be called by HijriApplet command. We can copy hijra-autostart.desktop description to i.e. ~/.gnome2/panel2.d/default/launchers for shortcut launcher.

python-gnome2-extras must be installed or else it’d fail to launch saying can’t find egg.trayicon

$ sudo make install

After that I test tagsistant to organize /data/tags/

$ tagsistant /data/tags

The tagging works as expected but not too handy since no browser GUI is embedded. It is just like browsing through multiple OR and AND folders.

The installation process will also install tagman (a tag manager GUI) if libgtk2.0-dev and libglib2.0-dev are installed.

In localhost snmpd is installed and configured /etc/default/snmpd:

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid'

and edit /etc/snmp/snmpd.conf to have:

com2sec readonly  localhost         <some community string>

Install snmp, then I can test snmpwalk:

$ snmpwalk -Os -c <some community string> -v 1 localhost system

Cacti requires RRDTool and the correct version to be set in Settings menu of the Cacti web console. In my case, RRDTool version is 1.3x. In Cacti 0.8.6.x and above poller.php is called by cron instead of cmd.php, put this trough crontab in Cacti user (enter this Cacti user crontab from crontab -e -u or embed the user in crontab by root instead of individually edited).

*/2 * * * * /usr/bin/php /var/www/cacti/poller.php

It will generate file under rra directory of the web /var/www/cacti/rra/.rrd. RRD tool will create image based on this polling result file.