Low level view of WPAD interactions involving DHCP, DNS, and HTTP servers in packet capture (shared in CloudShark)

In DHCP discovery-offer-request-acknowledgement cycle, WPAD information is given in option 252 (check section 4.4.1 of the draft). I used dhcpd3-server package in Ubuntu and setup /ip dhcp-server option add code=252 ... in Mikrotik 5.20 for example variations. For both, trailing “\n” are added to wpad.dat URL value as most howtos recommend.

"http://wpad.some-company.net/wpad.dat\n"

In the above example value, wpad.dat is hosted in Apache2 HTTP server resolvable by the DNS to the actual host IP of wpad.some-company.net. When the client use all DHCP offered items, it gets domain name (option 15) some-company.net besides other things e.g. router (option 3), DNS (option 6), etc. In a case where the client use its own DNS (only use IP from DHCP), WPAD likely won’t work unless it is able to resolve the example some-company.net or wpad.some-company.net. Following the tail of /var/log/apache2/access.log, successful wpad.dat request will appear as

192.168.40.75 - - [01/May/2013:21:51:15 +0700] "GET /wpad.dat HTTP/1.1" 200 1070 "-" "-"

In the above tail, I enabled a Windows 7′s network adapter (from previously disabled) and it would soon try to HTTP GET the wpad.dat even when no browser opened yet.

From testing with Windows XP, Windows 7, and Ubuntu (with different browsers mostly), DHCP Inform‘s WPAD behavior can be seen with Windows 7 test by analyzing the packet going out after some time since the initial DHCP stream (check the packet capture for 7 seconds after DHCP Discover). It asked for WPAD in one of the Parameter Request List Item (apply the filter bootp.option.request_list_item == 252 for a closer look), but received no answer.

I failed to reproduce other samples of DHCP Inform packet asking for WPAD. Hence, for the rest, it’s the browser who is doing the GET to http://wpad.some-company.net/wpad.dat with DHCP only supplied domain name instead of complete WPAD (check another packet capture, this time for Ubuntu with Firefox 20). The GET attempt isn’t actually one shot attempt. When it fails to find the wpad.dat in http://wpad.some-company.net/wpad.dat, it should iteratively try to find within the higher domain e.g. http://some-company.net/wpad.dat.

Getting proxy setting (WPAD) from DHCP or directly by browser iterative decision

DHCP server configuration used, wpad.dat file example (I simply use symbolic link to proxy.pac), and Apache2 VirtualHost config are available in my github, an attempt to integrate Squid3 cache with WPAD, PAC, and DHCP.

History

For trailing character in WPAD value mentioned earlier, I once configured option 252 value with “wpad.dat\?“, “proxy.pa“, and some other likely wrong variations. I also configured the browser in Windows XP and 7 with these variations of wrong values before getting the browser to use proxy auto-discovery again. Somehow it kept the wrong GET with trailing character resulting in 404 (page not found):

1. Firefox

192.168.40.78 - - [30/Apr/2013:16:20:39 +0700] "GET /wpad.dat/ HTTP/1.1" 404 478 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Win32)"

2. IE 6

192.168.40.78 - - [30/Apr/2013:16:28:07 +0700] "GET /wpad.dat/ HTTP/1.1" 404 478 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Win32)"

3. IE 8

192.168.40.77 - - [01/May/2013:20:07:24 +0700] "GET /wpad.dat/? HTTP/1.1" 404 534 "-" "-"

I was able to force it to GET the correct WPAD value by adding new network adapter which received different IP lease from DHCP server. Afterwards, connecting with the old adapter will also keep correct behavior.

I still don’t know how they keep the previously incorrect behavior. In short we can’t rely on WPAD alone, but there is probability that some portions of the network clients will find the proxy this way, automatically without technical assistance.

$ sudo service squid3 {start|stop|restart}

To specify what to run, an upstart must have exec or script stanza. When the goal is to start two or more (multiple) instances of squid3, exec is meant to replace existing process image of /usr/sbin/squid3 executable, therefore will not start two instances.

Multiple squid3 instances started via upstart exec

As workaround we can have symbolic link to squid3 executable instead and add new upstart job configuration with exec call to the link. I need two instances running in the same machine due to the implementation design where the fist instance is HTTP proxy cache while the second one is accelerator/interceptor (reverse proxy) that serves default landing page telling user to use proxy (and how to do that). Hence, for the second instance I add:

lrwxrwxrwx 1 root root 6 May  3 09:25 /usr/sbin/squid3ins2 -> squid3
-rw-r--r-- 1 root root 1156 May  6 13:30 /etc/init/squid3ins2.conf

and for the first instance I add:

-rw-r--r-- 1 root root 298 May  3 10:13 /etc/init/squid3.override

squid3ins2 is our link and /etc/init/squid3ins2.conf contains exec call to it to run second squid3. For the first instance I choose to write an override (/etc/init/squid3.override) so that the original job configuration file is left intact. For the complete content of all files check my github v1.1 of the Squid integration. There, in details you’ll also find how both instances PID, log, etc. are differentiated by each instance config via the following directives:

http_port
cache_dir
pid_filename
cache_access_log
cache_log

1. independent SNMP data polling by individual NMS proxy (embedded system)
2. intermittent connection between main/master NMS server with its proxy

Of course we’ll do them in implementation context of Zabbix Server as main/master NMS server and our BeagleBoard xM as its proxy.

Zabbix Proxy - distributed NMS over embedded Linux

In the technical details, I already followed pretty much the same steps showed in installation wiki (with the exception that I didn’t put the process into startup init.d script as I would start it manually as needed).

The first test is to make our Ubuntu ARM board accepts configuration done in Zabbix Server web-GUI, retrieves SNMP data to it, and sends collected data to server. Mode passive is chosen so that the server will be the one initiating contact with the board. After enabling a certain SNMP host, i.e. host-aaa, as monitored-by-proxy, the Zabbix Server log will show:

15677:20120601:022022.960 Sending configuration data to proxy 'beaglexm1'. Datalen 9205

Our board, ‘beaglexm1‘, will start to poll SNMP data from host-xxx to then store it in designated sqlite database.

975:20120531:212024.098 proxy #8 started [poller #5]
984:20120531:212024.128 proxy #17 started [discoverer #1]
968:20120531:212025.423 Received configuration data from server. Datalen 9205
971:20120531:212030.388 Enabling SNMP host [host-aaa]

From time to time as configured, our board will also run housekeeping to keep our board database file in small size.

1081:20120531:224513.922 Executing housekeeper
1081:20120531:224514.505 Deleted 4885 records from history [0.453765 seconds]

Under normal connection, in main Zabbix Server we’ll start seeing the collected data from host-aaa the same way as when the SNMP data is polled by the server directly. To emulate disruption to the connectivity, I choose to use simple iptables drop packet. This time the server will no longer seeing new data.

After the drop block is released, the server we’ll start seeing new data as well as data from the period of intermittent connection. It will take some time before all of that being sent to the server.

In general that proves it. I collected 44 OID values from the host-aaa every 30 seconds and in this case BeagleBoard utilization was very low as this wasn’t a stress test to see its performance.

Zabbix Proxy Distributed NMS

An example set out for Zabbix Proxy on Debian appears to be pretty straightforward to follow. With the ARM board already installed with build-essential, snmp, and snmpd, the following needs to be added:

$ sudo apt-get install libsnmp-dev libcurl4-openssl-dev fping curl sqlite libsqlite3-dev ntpdate

then start the pre-compilation configure (in this case Zabbix 1.8.7 is used):

cd zabbix-1.8.7
./configure --enable-proxy --enable-agent --with-sqlite3 --with-net-snmp --with-libcurl

It will go smooth. However, during make there are complains about undefined references to ‘sqlite3_open‘, ‘sqlite3_close‘, etc.:

../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_connect':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:331: undefined reference to `sqlite3_open'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:343: undefined reference to `sqlite3_busy_timeout'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:334: undefined reference to `sqlite3_errmsg'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:335: undefined reference to `sqlite3_close'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_init':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:385: undefined reference to `sqlite3_open'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:387: undefined reference to `sqlite3_errmsg'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_close':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:446: undefined reference to `sqlite3_close'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_vexecute':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:827: undefined reference to `sqlite3_exec'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:852: undefined reference to `sqlite3_changes'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:833: undefined reference to `sqlite3_free'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `SQ_DBfree_result':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:1363: undefined reference to `sqlite3_free_table'
../../src/libs/zbxdb/libzbxdb.a(db.o): In function `zbx_db_vselect':
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:1121: undefined reference to `sqlite3_get_table'
/home/zabbix/zabbix-1.8.7/src/libs/zbxdb/db.c:1127: undefined reference to `sqlite3_free'

this is obviously from the database related code compilation. I run the gcc manually for the db.c file inside its directory by adding the flag ‘-lsqlite3‘ in the tail of ‘../../src/libs/zbxdb/libzbxdb.a‘ to make it work. Afterward by going to the initial directory first, running make again will then continue successfully.

$ cd src/zabbix_proxy
$ gcc -DZABBIX_DAEMON -g -O2     -L/usr/lib/arm-linux-gnueabi    -I/usr/local/include -I/usr/lib/perl/5.12/CORE -I. -I/usr/include      -I/usr/include -lsqlite3  -L/usr/lib/arm-linux-gnueabi -lcurl  -L/usr/lib -lnetsnmp -lcrypto  -L/usr/lib -lnetsnmp -lcrypto   -rdynamic    -o zabbix_proxy zabbix_proxy-servercomms.o zabbix_proxy-events.o zabbix_proxy-zlog.o zabbix_proxy-proxy.o heart/libzbxheart.a ../../src/zabbix_server/dbsyncer/libzbxdbsyncer.a ../../src/zabbix_server/discoverer/libzbxdiscoverer.a housekeeper/libzbxhousekeeper.a ../../src/zabbix_server/httppoller/libzbxhttppoller.a proxyconfig/libzbxproxyconfig.a ../../src/zabbix_server/pinger/libzbxpinger.a ../../src/zabbix_server/poller/libzbxpoller.a ../../src/zabbix_server/trapper/libzbxtrapper.a ../../src/zabbix_server/nodewatcher/libzbxnodewatcher.a datasender/libzbxdatasender.a ../../src/zabbix_server/selfmon/libzbxselfmon.a ../../src/libs/zbxsysinfo/libzbxserversysinfo.a ../../src/libs/zbxsysinfo/linux/libspecsysinfo.a ../../src/libs/zbxsysinfo/common/libcommonsysinfo.a ../../src/libs/zbxsysinfo/simple/libsimplesysinfo.a ../../src/libs/zbxlog/libzbxlog.a ../../src/libs/zbxdbcache/libzbxdbcache.a ../../src/libs/zbxmemory/libzbxmemory.a ../../src/libs/zbxalgo/libzbxalgo.a ../../src/libs/zbxnix/libzbxnix.a ../../src/libs/zbxsys/libzbxsys.a ../../src/libs/zbxconf/libzbxconf.a ../../src/libs/zbxcommon/libzbxcommon.a ../../src/libs/zbxcrypto/libzbxcrypto.a ../../src/libs/zbxcomms/libzbxcomms.a ../../src/libs/zbxcommshigh/libzbxcommshigh.a ../../src/libs/zbxjson/libzbxjson.a ../../src/libs/zbxexec/libzbxexec.a ../../src/libs/zbxself/libzbxself.a ../../src/libs/zbxserver/libzbxserver.a ../../src/libs/zbxicmpping/libzbxicmpping.a ../../src/libs/zbxdbhigh/libzbxdbhigh.a ../../src/libs/zbxdb/libzbxdb.a -lsqlite3 -lcurl  -lnetsnmp    -lm  -lresolv
$ cd ../../
$ make

1
2
3
4
5
6
7
8

WAN=wan0
LAN=eth1
IPTABLES=/sbin/iptables
 
$IPTABLES --flush                         # Flush all the rules in filter and nat tables
$IPTABLES --table nat --flush
$IPTABLES --delete-chain                  # Delete all chains that are not in default filter and nat table
$IPTABLES --table nat --delete-chain

the real script is just two command line below:

1
2

$IPTABLES --table nat --append POSTROUTING --out-interface $WAN -j MASQUERADE
$IPTABLES --append FORWARD --in-interface $LAN -j ACCEPT

Default route gateway is wan0 and /etc/resolv.conf is edited with relevant DNS. Internet works perfectly and then after squid is installed, I use the following /etc/squid.conf :

1
2
3
4
5
6

visible_hostname mithrandir
http_port 3128
cache_dir ufs /var/spool/squid 1000 16 256
cache_access_log /var/log/squid/access.log
acl intranet src 192.168.40.0/24
http_access allow intranet

It is pretty straightforward and simple, a listening proxy 192.168.40.40 in port 3128. It is also a router to the internet, NAT, via wan0.