A quick adaptation of Nebri’s straightforward blog post is to first shortlist rain related codes into 15 of them and changing from forecast to latest measurement instead. Bridging Nebri and my smartwatch are Twitter and IFTTT recipe–the watch merely mirrored my phone, you don’t need it actually. Why (again) Twitter? Well, rather than the “then that” side–the watch enabler, the “if this” side of IFTTT for Twitter recipes is quite powerful given so many tweet filtering options.

Key-value pairs as trigger for event driven Nebri OS workflow

Many things already taken care by Nebri, hence unseasoned programmer should easily cope with writing simple Python rule and focus on the automation, nevertheless complex rule should also be accommodated. I was having trouble on using previous KVP from different PID though–moved them to comment in the script, but they seem to be working on it. Check their YWeather blog post on how the basics work and compare what I did in Github. I embed an example tweet from @twithujan below. Yahoo! Weather frequency of measurement and accuracy are of different topic by the way.

Bogor is Light Rain with temperature at 23 °C. Taken at Thu, 28 May 2015 7:00 pm WIT

— Twit Hujan (@twithujan) May 28, 2015

PS: “hujan” means “rain” in Indonesian

If this than that recipes alert for Commuter Train trouble

The ingredient is tweet, Twitter Search use case is no longer just sentiment analysis, its function extends to alert, even close to real time–I’m pretty optimistic given the number of Commuter Line users joining the conversation. Now, the no-code part of our recipe, a Twitter search query, that is

(to:krlmania OR @krlmania OR from:CommuterLine)
AND (gangguan OR pantograf OR wesel OR patah)
-semoga -smoga -moga

and a brief explanation is that it queries for some keywords for potential trouble within mentions/replies to @krlmania and official announcements of @CommuterLine. Exclusion added to the search, the sentiment part of tweets, expressing wishes to shoo trouble away–I’m sorry, but keep praying online anyway.

When Twitter launched Twitter Stories in 2011, I read about Ravi Pina running the crowd sourced @Caltrain. Later now I see that official accounts like @SMRT_Singapore is also a model of service disruption info center. We can go deeper to better stats from the API web service that shows real time position (an app is out there showing that the location data API is working), but until its open data status is known, crowd sourcing stays as the most reliable solution, independent to regulation, and easiest to implement. Does the recipe work? During the first week of testing, alerts were flooding–within limit of course–as trouble really happened. So, cook your own ingredients or get my recipes served here.

Battery power bank, WiFi USB stick, and thumb drive

I then turn away to motor-less small storage, a 32 Gigs USB thumb drive so I can move on to other concern: on-the-fly encryption. Hence, I follow instructions to install True Crypt on Raspberry Pi. After unpacking of wxWidgets-2.8.12.tar.gz and TrueCrypt 7.1a Source.tar.gz in place, putting header files from pkcs-11-cryptoki2.20, and then install libfuse-dev, the following make will require long time:

$ export PKCS11_INC=/usr/local/src/truecrypt/pkcs/
 
$ make NOGUI=1 WX_ROOT=/usr/src/wxWidgets wxbuild
Configuring wxWidgets library...
Building wxWidgets library...
/usr/src/wxWidgets/src/common/string.cpp:84:39: warning: ‘wxEmptyString’ initialized and declared ‘extern’ [enabled by default]

$ make NOGUI=1 WX_ROOT=/usr/src/wxWidgets wxbuild
Compiling Buffer.cpp
Compiling Exception.cpp
Compiling Event.cpp
...
../Crypto/Aeskey.c:527:25: warning: operation on ‘ss[7]’ may be undefined [-Wsequence-point]
../Crypto/Aeskey.c:527:25: warning: operation on ‘ss[7]’ may be undefined [-Wsequence-point]
../Crypto/Aeskey.c:527:25: warning: operation on ‘ss[7]’ may be undefined [-Wsequence-point]
...
Converting Language.xml
Compiling Resources.cpp
Linking truecrypt

I created the TrueCrypt volume separately via its desktop GUI with ext3 file system to then mount it in Pi to a configured Samba share. As pointed out in a post, the following changes are added to smb.conf

...
wins support = yes
...
[pitruecrypt]
   comment= Pi Truecrypt Volume
   path=<the mount path of the USB thumb drive TrueCrypt volume>
   browseable=Yes
   writeable=Yes
   only guest=no
   create mask=0777
   directory mask=0777
   public=no

and then user-password are entered via interactive command.

For mobility, I already had the Pi as WiFi access point using hostapd (check these steps) and power bank, so it’s now matter of performance. In the case of rsync, initial sync of some 1,500 items totaling in 1 Gig size elapses in approximately the same 12 minutes of time compared to one bulk file of the same size. Of course, over the next incremental sync, it only takes less than a minute for the thousand items to just update slight differences.

Security

Back to security, there surely risk by opening Samba share to mounted TrueCrypt volume. But, for me it would be practically manageable (cross my finger). There is more concern to the fate of TrueCrypt after it is being closed in such a weird way, given that last audit finds nothing severe. Anyway, I found brute force tool, but no critical attack exists currently, unless e.g. it stays powered on and mounted, the person gain physical access. Beats me again.

A matrix keypad instance is defined by:

1. Actual GPIO pins used forming the row and column of the m x n matrix
2. Individual character in-use as symbol for each button

Hence, I instantiate and call method like

...
QPad  = matrixQPi(keyPad=keyPad,row=row,col=col)
print QPad.scanQ()

to print the character being pressed. Some examples pushed to my github explains how the above keyPad, row, and col are defined to scan-read pressed button of 2×2, 2×3, and 4×3 matrix keypads (or have it 3×4 matrix keypads in other words).

Illustration of any m-by-n matrix: 4x3, 2x2, and 2x3 keypad with different button symbols & GPIO combinations

I used deprecated Wiring-Pi Python (they already moved to 2.x version) without problem. However, you’ll fail building from latest commit and must use combination of older commits as described by my updated part of an old-post. By the way, there’s I/O expander support for WiringPi2-Python which is good, considering:

GPIO is expensive and for the sake of a keypad, you should not spend all.

(A friend told me that once)

Low level view of WPAD interactions involving DHCP, DNS, and HTTP servers in packet capture (shared in CloudShark)

In DHCP discovery-offer-request-acknowledgement cycle, WPAD information is given in option 252 (check section 4.4.1 of the draft). I used dhcpd3-server package in Ubuntu and setup /ip dhcp-server option add code=252 ... in Mikrotik 5.20 for example variations. For both, trailing “\n” are added to wpad.dat URL value as most howtos recommend.

"http://wpad.some-company.net/wpad.dat\n"

In the above example value, wpad.dat is hosted in Apache2 HTTP server resolvable by the DNS to the actual host IP of wpad.some-company.net. When the client use all DHCP offered items, it gets domain name (option 15) some-company.net besides other things e.g. router (option 3), DNS (option 6), etc. In a case where the client use its own DNS (only use IP from DHCP), WPAD likely won’t work unless it is able to resolve the example some-company.net or wpad.some-company.net. Following the tail of /var/log/apache2/access.log, successful wpad.dat request will appear as

192.168.40.75 - - [01/May/2013:21:51:15 +0700] "GET /wpad.dat HTTP/1.1" 200 1070 "-" "-"

In the above tail, I enabled a Windows 7′s network adapter (from previously disabled) and it would soon try to HTTP GET the wpad.dat even when no browser opened yet.

From testing with Windows XP, Windows 7, and Ubuntu (with different browsers mostly), DHCP Inform‘s WPAD behavior can be seen with Windows 7 test by analyzing the packet going out after some time since the initial DHCP stream (check the packet capture for 7 seconds after DHCP Discover). It asked for WPAD in one of the Parameter Request List Item (apply the filter bootp.option.request_list_item == 252 for a closer look), but received no answer.

I failed to reproduce other samples of DHCP Inform packet asking for WPAD. Hence, for the rest, it’s the browser who is doing the GET to http://wpad.some-company.net/wpad.dat with DHCP only supplied domain name instead of complete WPAD (check another packet capture, this time for Ubuntu with Firefox 20). The GET attempt isn’t actually one shot attempt. When it fails to find the wpad.dat in http://wpad.some-company.net/wpad.dat, it should iteratively try to find within the higher domain e.g. http://some-company.net/wpad.dat.

Getting proxy setting (WPAD) from DHCP or directly by browser iterative decision

DHCP server configuration used, wpad.dat file example (I simply use symbolic link to proxy.pac), and Apache2 VirtualHost config are available in my github, an attempt to integrate Squid3 cache with WPAD, PAC, and DHCP.

History

For trailing character in WPAD value mentioned earlier, I once configured option 252 value with “wpad.dat\?“, “proxy.pa“, and some other likely wrong variations. I also configured the browser in Windows XP and 7 with these variations of wrong values before getting the browser to use proxy auto-discovery again. Somehow it kept the wrong GET with trailing character resulting in 404 (page not found):

1. Firefox

192.168.40.78 - - [30/Apr/2013:16:20:39 +0700] "GET /wpad.dat/ HTTP/1.1" 404 478 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Win32)"

2. IE 6

192.168.40.78 - - [30/Apr/2013:16:28:07 +0700] "GET /wpad.dat/ HTTP/1.1" 404 478 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Win32)"

3. IE 8

192.168.40.77 - - [01/May/2013:20:07:24 +0700] "GET /wpad.dat/? HTTP/1.1" 404 534 "-" "-"

I was able to force it to GET the correct WPAD value by adding new network adapter which received different IP lease from DHCP server. Afterwards, connecting with the old adapter will also keep correct behavior.

I still don’t know how they keep the previously incorrect behavior. In short we can’t rely on WPAD alone, but there is probability that some portions of the network clients will find the proxy this way, automatically without technical assistance.

$ sudo service squid3 {start|stop|restart}

To specify what to run, an upstart must have exec or script stanza. When the goal is to start two or more (multiple) instances of squid3, exec is meant to replace existing process image of /usr/sbin/squid3 executable, therefore will not start two instances.

Multiple squid3 instances started via upstart exec

As workaround we can have symbolic link to squid3 executable instead and add new upstart job configuration with exec call to the link. I need two instances running in the same machine due to the implementation design where the fist instance is HTTP proxy cache while the second one is accelerator/interceptor (reverse proxy) that serves default landing page telling user to use proxy (and how to do that). Hence, for the second instance I add:

lrwxrwxrwx 1 root root 6 May  3 09:25 /usr/sbin/squid3ins2 -> squid3
-rw-r--r-- 1 root root 1156 May  6 13:30 /etc/init/squid3ins2.conf

and for the first instance I add:

-rw-r--r-- 1 root root 298 May  3 10:13 /etc/init/squid3.override

squid3ins2 is our link and /etc/init/squid3ins2.conf contains exec call to it to run second squid3. For the first instance I choose to write an override (/etc/init/squid3.override) so that the original job configuration file is left intact. For the complete content of all files check my github v1.1 of the Squid integration. There, in details you’ll also find how both instances PID, log, etc. are differentiated by each instance config via the following directives:

http_port
cache_dir
pid_filename
cache_access_log
cache_log

In the Debian example, the wiki introduced NetInstall. Later I find out the mk_mmc.sh script shown there to be useful for (1) duplicating and (2) restoring working Linux backup to new or corrupted MMC/SD card. I break down that Robert C. Nelson’s mk_mmc.sh script at GitHub to small routines for setting up the card.

Typical boot-rootfs partitions of SD card to boot ARM-Linux boards

It’s an elegant script automating the process of preparing SD card for embedded Linux, but requires large downloading on the run (mostly root file system and then U-Boot binaries and loader config). Duplicating/restoring means we already have those and probably only need to do elementary process rather than full card preparation. Below are some processes that can be repeatedly used.

Checking Tool Versions

The tools to work with are common in Linux distribution (I used Ubuntu) and require root privilege. The script checked for fdisk and parted installed. My versions are

$ fdisk -v
fdisk (util-linux-ng 2.17.2)
$ parted -v
parted (GNU parted) 2.2

Originally mk_mmc.sh will exit If “GNU Fdisk” found instead. Also for the above parted version, “--align cylinder” argument will be used.

Creating Two Partitions

Setting up two partitions which are the boot image (FAT 16) and root file system (ext4). The SD card plugged in through USB card reader found as /dev/sdb in this example (we’re formatting be sure of this!). If you plug MMC card to laptop slot, it may be found as /dev/mmcblk0.

1. Format card into a single disk

$ parted --script /dev/sdb mklabel msdos
$ fdisk -l /dev/sdb

2. Create the boot partition (allocating 64MB)

$ fdisk /dev/sdb << END
n
p
1
1
+64M
t
e
p
w
END

then make sure it’s done.

$ sync

3. Flag it as boot

$ parted --script /dev/sdb set 1 boot on

4. Create the root file system by first determining space left after the first partition

$ END_BOOT=$(LC_ALL=C parted -s /dev/sdb unit mb print free | grep primary | awk '{print $3}' | cut -d "M" -f1)
$ echo $END_BOOT 
69.7
$ END_DEVICE=$(LC_ALL=C parted -s /dev/sdb unit mb print free | grep Free | tail -n 1 | awk '{print $2}' | cut -d "M" -f1)
$ echo $END_DEVICE 
3951

In this case 4GB SD card is used and the above calculation will stretch space left from 69.7 to 3951 of as rootfs partition.

$ parted --script --align cylinder /dev/sdb mkpart primary ext4 $END_BOOT $END_DEVICE

5. Format boot and rootfs according to each file system type with those naming also. They’re already accessible as /dev/sdb1 and /dev/sdb2 (previously didn’t exist) or relevant /dev/mmcblk0p(some index).

$ mkfs.vfat -F 16 /dev/sdb1 -n boot
$ mkfs.ext4 /dev/sdb2 -L rootfs

6. Finally we have something like the output of parted below

$ LC_ALL=C parted -s /dev/sdb unit mb print free 
Model: HUAWEI MMC Storage (scsi)
Disk /dev/sdb: 4023MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
 
Number  Start   End     Size    Type     File system  Flags
 1      0.03MB  70.9MB  70.8MB  primary  fat16        boot, lba
        70.9MB  4023MB  3953MB           Free Space

(yes I used Huawei modem as the microSD card reader)

Restoring Linux

For the boot partition content we can simply copy-paste the backup files, while for rootfs normally compressed file (preserving path information) is used for backup. Restore this by mounting rootfs and then unpacking the file (again /dev/sdb2 in this case).

[root@host]$ mkdir /media/rootfs
[root@host]$ mount -t ext4 /dev/sdb2 /media/rootfs
[root@host]$ pv armel-rootfs-201110131018.tar | sudo tar --numeric-owner --preserve-permissions -xf - -C /media/rootfs

Below is an example of how to backup rootfs from the mounted SD card (/media/rootfs).

[root@host]$ cd /media/rootfs
[root@host]$ /media/rootfs# tar --numeric-owner --preserve-permissions -cvf /home/arif/Documents/arm-linux/ubuntu/rootstock-rootfs/armel-rootfs-201202141421.tar ./*

(Check my other post related to creating rootfs)

Creating Swap

mk_mmc.sh also offers optional swap creation. It is a “file” inside rootfs partition created by

[root@host]$ dd if=/dev/zero of=/media/rootfs/SWAP.swap bs=1M count=250
[root@host]$ mkswap /media/rootfs/SWAP.swap

(the above will create 262MB swap inside rootfs SD card currently mounted to /media/rootfs of the laptop)

It will be mounted as swap by the ARM-Linux board via fstab setting configured below.

[root@host]$ echo "/SWAP.swap  none  swap  sw  0 0" >> /media/rootfs/etc/fstab

Synchronize and safely remove the card

[root@host]$ sync
[root@host]$ umount /media/rootfs

Now it’s almost automatically attached as modem after insertion and people have been contributing to list of device-and-target device after mode-switching (find it as /usr/share/usb_modeswitch/configPack.tar.gz). At least for Raspberry Pi (RPi), I have one device list from August 2012 when playing around with XBian 0.8.3 and one from May 2012 in when using Raspbian Wheezy (2012-08-16).

Huawei E220 requires no mode-switch from vendor-product ID 12d1:1003 to function as modem

There is still wrapper for udev in /lib/udev/rules.d/40-usb_modeswitch.rule and my Huawei E153 HSDPA stick recognized and switched successfully as shown:

usb_modeswitch: switching device 12d1:1446 on 001/005
...
logger: usb_modeswitch: switched to 12d1:14ac on 001/006

But I didn’t get there in the first place. Because of USB power problem (detailed in previous post), I used to plug USB devices before powering up so that I could use Y-cable injected by separate power source without powered USB-hub. It turned out that if I had plugged the USB before powering up the RPi the mode-switching didn’t occur.

The HSDPA USB modem used is Huawei E153 appearing in lsusb with vendor-product ID 12d1:1446. Using aforementioned XBian and target device 12d1:140c (suggested in Santinoli’s post), I arbitrarily succeeded to switch to modem and dial internet with pppd.

[root]$ usb_modeswitch -v 12d1 -p 1446 -P 140c

(I placed Santinoli’s config in /etc/usb_modeswitch.conf)
This wasn’t stable however (also note that XBian consumed CPU above 50% for xbmc).

Later I find out that it will automatically switch to 12d1:14ac instead when plugged in after RPi is up. However, since it also won’t switch if plugged before power-up, I also try to mode-switch manually using config file found inside /usr/share/usb_modeswitch/configPack.tar.gz archive named 12d1:1446.

[root]$ usb_modeswitch -v 12d1 -p 1446 -c 12d1\:1446

It doesn’t seem to work. Modification to something close to my previous config also fails. Hence, consistent result only occurs if the stick is plugged in after RPi up. Then I decide to use USB-hub (also detailed in previous post) to tackle power issue (it will reboot anyway if you plug in without adequate powering). However, after successful mode-switch what’s being reported in eLinux Wiki often happens: it slows down, try lsub command for example.

Currently I give up the idea of using that modem and choose older Huawei E220:

1. No usb_modeswitch. Driver attached directly by the kernel as option.ko device (this device has been on the list for long)
2. Stable. No slowing down

The E153 modem itself has been implemented to stream video using the higher ARM type BeagleBoard-xM without significant stability issue (check my YouTube demo). It mode-switched when powering up Ubuntu on the Beagle.

Y-cable for USB modem stick & various current rating-AC power adapters for smartphone/tablet to go with Raspberry Pi

eLinux Wiki lists market available USB-hubs reported to work. A decent brand (that will cost you) qualifies:

• Doesn’t leak power back to RPi via USB port (check this by unplugging the main power, then check the red LED indicator) as this will interfere with the reboot command (physical state vs software).
• Provides charging from one of its port for RPi, meaning no separate source for the exclusive 700 mA requirement. A single AC power adapter with high current rating will fit all (check single power connectivity sketch in this blog post)

An ugly situation when I plugged in WiFi stick and modem looks like:

kernel: [ 1836.871640] smsc95xx 1-1.1:1.0: eth0: Failed to read register index 0x00000118
...
kernel: [ 1803.780601] phy0 -> rt2x00usb_vendor_request: Error - Vendor Request 0x07 failed for offset 0x101c with error -110.

tail of the above syslog is saying error with the (1) ethernet (this is actually also a USB 2.0 but not appearing in lsusb command) and (2) the WiFi stick used (in this case with RT5370 chipset). Then, of course (3) doing things with the just plugged modem you’ll expect more instability.

I mentioned BeagleBoard-xM which also came with similar USB ethernet on-board. Unfortunately, Beagle was more stable when my case was to have it stream a zoom-camera input to internet (check my YouTube demo). The camera decoder and modem were both USB plugged without Y-cable or additional power supply. As a standalone system, part of the design was to reboot under trouble detected by the software. Again, OS command reboot is not achievable consistently with the RPi when power backfeeds through USB port.

(PS: Note that previous comparison solely points out what to expect when dealing with USB among other considerations e.g. price, different ARM, etc. which are entirely different.)

As an absolute requirement for me, I found a powered USB-hub available in Indonesian market as 7 ports XTec Go. Yes, it leaks power back as confirmed by the vendor-product ID 05e3:0608 of the chipset listed in eLinux Wiki lists with a bunch of different names (judging the picture, Hama is closest).

$ lsusb
...
Bus 001 Device 004: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 005: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
...

Three Musketeers: powered USB-hub (Genesys chipset), Huawei E220 HSDPA modem, and TL-WN727N WiFi stick

My el cheapo USB-hub, HDSPA modem, and WiFi stick are the matchmaking of brands shown in the above image. It achieves long uptime like 6 hours serving internet when RPi used as router. I still use separate AC power adapter on the go with me: regular 700 mA that comes with smartphone will do for the RPi, while for the USB-hub, I have 2A rating from tablet charger. However, I’ve also plugged separate 500 mA charger for the hub without problem and/or excessive heat (you need to touch the supply sometime to feel any nasty heat as some of them might have been low quality build that could blow up. Yes it did). My plan with mobility is to have power bank with dual USB charging output (this Hame has 1A and 2.1A) when no AC source nears.

Have I tested other USB modem stick? Check my other post related to usb_modeswitch.

Updated: for those who fail to build using deprecated WiringPi-Python, check this updated post to know which commit that build without error. There is now also a Python class for matrix keypad.

My idea of having the keypad is to make alternative input available under no keyboard presence nor shell access. List of things I can think of for instances, pressed key “0″ will make the Raspberry Pi (RPi) dial GPRS to a specific ISP and act as router to the USB WiFi stick, pressed key “7″ will convert it to a router that will bridge the ethernet to WiFi, etc. In short, those key readings will invoke subsequent scripts to run inside RPi.

The physical connection schematic drawings and code are gitified (visit v1.2 of the project on gitHub). A nice animated image on how the buttons connect pins forming a matrix can be found in hackyourmind.org.

Matrix keypad: alternative quick input for Raspberry Pi to start certain command

Some remarks over what the code does:

• It only read one pressed key at a time.
• It uses no interrupt. To read input it must wait and scan (this waiting experience is by far negligible to human sense).
• Debouncing doesn’t seem necessary as WiringPi already provided this by software (I suppose this timing method on their gitHub deals with debounce).

How it works? RPi forum thread gives a general idea that applies in my case:

1. Divide the 3×4 matrix as columns and rows. 4 GPIO pins as rows are pulled-up with 10k resistors and initialized as input.
2. Other 3 GPIO pins as columns are initialized as output low.
3. First loop will scan for one pressed key being read as one of the rows pulled-low
4. After the loop breaks, all columns are set as input, then the row pin found in the loop is set as output-high
5. Second loop will scan for column being pulled-high by that row pin. Between both loops, it is assumed that the key press is still in effect. In reality, normal human act of pressing this key elapses long enough for the software to run the scans in two loops.
6. Bingo! The code reads row-column combination of the pressed key.

Example of the code’s output by calling from shell:

$ while true; do /usr/local/bin/matrixQPi.py -i; done
3
6
9
8
#
*

(In the above example ^C will throw Python KeyboardInterrupt messages before breaking the bash loop)