Automating HTTPWatch with PHP

March 5th, 2011. Tagged: HTTP, performance, php

HTTPWatch is a nice tool to inspect HTTP traffic in easy and convenient way and it works in both IE and FF now. Drawback - windows-only and paid. But the free version is good enough for many tasks.

HTTPWatch can be automated and scripted which is pretty cool for a number of monitoring-like tasks. Their site and help section lists C# and Ruby+Watir examples. So I was curious - what about PHP (and no Watir).

In general with PHP you can open/close/navigate IE using COM (whatever that is) which is nice, but you can't do that with Firefox as it doesn't expose a COM interface. But HTTPWatch fills the gap. K, let's see an example.


OS: Windows
Install: IE, Firefox, HTTPWatch, php (command-line is fine, no need for Apache, MySQL, etc)

Getting started

Create a file, say C:\http.php, open command prompt and go:

cd \
C:\>php http.php

Now all that's left is to put something worth executing in http.php :)

Instantiating HTTPWatch

$controller = new COM("HttpWatch.Controller");
if(!method_exists($controller, 'IE')) {
  throw new Exception('failed to enable HTTPWatch');

Opening a new Firefox window:

$plugin = $controller->Firefox->New();

BTW, it's the same for IE:

$plugin = $controller->IE->New();

Disabling any filters (filters defined in HTTPWatch that is)


Clear HTTPWatch's log (the list of requests), clear the browser cache and start recording traffic:

// clear log and cache
// start

Navigate to a URL and wait for it to complete - that means wait a bit after onload even

// browse
$controller->Wait($plugin, -1);

Stop monitoring traffic and quit the browser:


This is nice, we opened the browser, visited a URL and closed. Now we can even get some meaningful data out of the whole experience.

$plugin->Log->Entries is an object that has a list of all requests. It also has a property Summary. So we can see how many bytes we sent and how many received as a result of this visit to

$sum = $plugin->Log->Entries->Summary;
echo "in: {$sum->BytesReceived}, out: {$sum->BytesSent}";

Note: oh, you need to get your data before closing the browser, otherwise the Log object gets destroyed it seems

So the result:

C:\>php http.php
in: 89185, out: 7102


This may look like nothing, but is pretty impressive in an of itself. At least I know I was happy the first time it worked. Because, you see, any monitoring that doesn't use a real browser is kinda smelly, isn't that right? Plus this is awesome for performance tests, research and experiments. You can create page A and page B and go out for a walk. Meanwhile your script can load the pages 200 times in the two browsers (at least, because you can have FF+IE[678]), with empty and full cache... and you come back for the results! Tired of all the walking, not of hitting REFRESH.

Below you can see (HD!) video of a script that opens IE and FF, loads Google and then gives you the bytes in/out in the two browsers. This example uses a PHP class I created and will talk about later, but you can still see the idea.

A better experience in IE

One thing I don't like is that HTTPWatch won't let you control the browser very well. Two features I'm looking for: being able to see HTTPWatch's log while running (for testing) and then being able to completely hide the window (for "production"). Luckily IE let's you do that and HTTPWatch let's you "attach" an already running IE instance.

So. We open IE with its own COM interface:

$browser = new COM("InternetExplorer.Application");
if(!method_exists($browser, 'Navigate')) {
  throw new Exception('didn\'t create IE obj');
$browser->Visible = true;

As you can see - not very different. But there's Visibile which can be false if you so like. This way you can still work on something while tests are running in the background without windows popping up all the time.

Also if you open HTTPWatch manually and close the browser, then the next time (in your scripted runs) HTTPWatch will stay open and you can check what's up.

So, connecting HTTPWatch with the IE instance means instantiating HTTPWatch as before and passing the IE object to Attach() method (was New() before).

// watch this!
$controller = new COM("HttpWatch.Controller");
if(!method_exists($controller, 'IE')) {
  throw new Exception('failed to enable HTTPWatch');
// enable plugin
$plugin = $controller->IE->Attach($browser);

The rest is all the same.

There's more

The most interesting part is getting data back from HTTPWatch. Dunno about you, but I love just dumping whatever structure I have with print_r() or var_dump() and then deciding what I want from it and how to to go about getting it.

That doesn't happen here because these COM objects are Variants and you can't just dump'em. You have to read the API docs. That sucks. So I did a hack (next post) and also read the APIs ("Stoyan: reading the APIs so you don't have to!") to enable just dumping the httpwatch's log.



HTTPWatch can write you a HAR file with the log. Not everything is in there, but it's still a lot and it's easy. HAR is JSON so you json_decode() it and voila - a log!

$filename = tempnam('/tmp', 'watchmenowimgoindown');
$json = file_get_contents($filename);

If you're curious as to what that prints - here it is.

Want to see a HAR (from another run)? Here it is.

So here you go - much data can be extracted and dumped for inspection from the HAR output. For the full httpwatch data, there's the API.

(to be continued...)

Tell your friends about this post: Facebook, Twitter, Google+

12 Responses

  1. This is very nice, I’m also playing with some libs recently for automating browsers.

    Have a look at : & (sauce rc)

    This allow you to launch any selenium-compatible web browser and control it.

    Then all you have to do is plugging pagespeed and get the beacon url to talk to one of your script.

    Now I’m looking at video recording. This is hard. The best way I’ve found is to use a vncserver + a vnc recorder. It works and it seems to not stress too much the CPU. + tightvnc server (1.3, not 2)

    I do now that WPT does all of this already. But only for IE..

  2. [...] Stoyan's blog about (x)html, ajax, bookmarklets, browsers, css, firebug, javascript, json, mdb2, mysql, pear, performance, php, phpbb, tools, yslow, yui, writing, music,… life and everything. « Automating HTTPWatch with PHP [...]

  3. [...] first part is here, the second is here. This third post is more about PHP and COM, rather than HTTPWatch or monitoring [...]

  4. Watch out for the HAR decoding. HARs are UTF8 JSON files (not encoded ascii like php is expecting) so json_decode will fail on some HAR files depending on the site being tested.

  5. [...] Automating HTTPWatch with PHP Another good how-to from Stoyan. This is a three-part post about automating and scripting HTTPWatch to perform various “monitoring-like” tasks. [...]

  6. [...] …with PHP (and again and again, and response) [...]

  7. [...] …with PHP (and again and again, and response) [...]

  8. Hello,

    I am fairly new to programming and first time user of HttpWatch.

    I am trying to automate HttpWatch testing. I would like to get all the parameters and values on a page containing WebTrends tags (base tags and the ones developed in-house). I can get some info related to httpwatch entries, but don’t see any function to get all the parameters and values in that specific entry.

    Although I am using C#, your suggestions still might help.

    Thanks in advance…


  9. PHP Scripts…

    [...]Automating HTTPWatch with PHP / Stoyan’s[...]…

  10. [...] the first post he shows you how to get started with the tool – installation and using it in a sample script [...]

  11. Do it for about 40 sports injury management seconds.

    The work chronicled the adventures of journalist Christopher McDougall and
    his coach, James Li has been effective. In focusIn-depth
    coverage of the US has underlined just how vulnerable and expensive offshore oil
    drilling is. – What’s ‘Unsuitable’ That’s an accurate description.

    I sports injury management thought you were way back there.
    And now, says he goes as far as we are concerned, she has
    declared her intention to take a year off from politics entirely.

  12. Its culpture and customs are a fusion of African, Spanish, Dutch and French to name just a few.
    The Developers of Cappe Wind, Energy Management Inc. Besides also can treat women lack of physical and cold syndrome, etc.

Leave a Reply