HTTPWatch automation with JavaScript

April 10th, 2011. Tagged: JavaScript, performance, speaking, tools

Background

I gave this short presentation at the recent Yahoo FE summit's open mic, here are the slides and some notes.

Slides and screencast vid

Screencast to see the thing in motion:

Notes

Here's the transcript of the slides as produced by slideshare.net. I've added some notes here and there to make it more readable when the presenter is missing.

JavaScript shell scripting - Presentation Transcript

  1. JavaScript is everywhere #42:
          C:> WSH
  2. Stoyan

    I do programming.

  3. Programming

    There are many options to choose from when you decide top practice the
    art and craft of programming.

  4. JavaScript

    ... is a very good one. Simply because JavaScript...

  5. ... is everywhere
  6. On the server

    node.js, asp.net

  7. Mobile

    html5, phonegap, titanium

  8. Desktop

    XULRunner lets you create cross-OS GUI apps

  9. Browser extensions

    FF, Chrome, bookmarklets...

  10. Photoshop

    yep, that too
    Several Adobe products actually let you script common/uncommon/programmable tasks

  11. Form validation too!

    (this was supposed to be funny)

  12. Shell scripting

    let's talk about shell scripting with JavaScript

  13. In Windows
  14. WSH: Windows Scripting Host

    All reasonable Windows machines (at least as old as Win2000) have this Windows Scripting Host in there.
    You can write VBScript or JavaScript (OK, JScript) to ... well, script.
    How does it work?

  15. C:>edit hello.js

    You create a file.

  16. var a = "Hello",
        b = " WSH!",
        c = a + b;
            
    WScript.Echo(c);

    Put any old JavaScript in there and print out some results

  17. C:>cscript hello.js
    Hello WSH!

    And this is how you run it.

    Or this:

    C:>wscript hello.js
  18. Open apps

    In addition to regular sysadmin tasks (copy, write files, move) you can open and script applications too.

  19. var ie = new ActiveXObject("InternetExplorer.Application");
    ie.Visible = true;
    ie.navigate(yahoo.com);

    This is how you open IE and point it to a page.

    Notice something familiar? ActiveXObject - the thing we used in IE before it got native XMLHttpRequest

  20. Firefox?

    Can you also open FF?

    Not really, as it doesn't have COM interface (whatever that is).

    But there's an easy solution

  21. HTTPWatch

    Finally we come to the topic of the talk.

  22. Speed

    Performance is critical for the success of any web app.

    Really, it is.

    When talking about improving speed there are two main steps:

  23. 1. Fix with YSlow

    Take a slow page, run YSlow, do what it says.

    Voila - a fast(er) page.

  24. 2. No regressions

    The second step is to not allow regressions.

    Whatever you fix in step 1 will be slow in the next few months.

    Even less than months the bigger the team or the rate of changes.

    So to prevent regressions, you need to constantly...

  25. Monitor
  26. Set limits

    The simplest way to prevent regressions is to set some limits.

    If you go over the limits, an email is sent, an alarm sounds, panic instills and you've got to fix whatever cause it was.

  27. e.g.
    max 2 scripts
    max 2 styles
    max 9 images
    max 0 redirects
  28. Scripting HTTPWatch

    Watching for violations of the limits manually every day is not a job anyone would want.

    So automating it will help a great way towards employee satisfaction :)

  29. var http = new ActiveXObject("HTTPWatch.Controller"),
        ie = http.IE.New(),
        ff = http.Firefox.New();

    This is how you open IE and FF with HTTPWatch's help.

    FF - yey!

  30. // browser cache
    ie.clearCache();
    
    // show HTTPWatch
    ie.OpenWindow(false);

    Examples of stuff you can do with the HTTPWatch API.

    You can for example hit the page with empty cache and then again with full cache.

    Best of all - these are the real browsers with their sometimes kinky behaviors.
    Actually if you setup several machines for the monitoring (or somehow do multiple IEs)
    you can test with different versions of the browsers. Nothing synthetic!

  31. ie.Record();
    ie.GotoUrl("yahoo.com");
    http.Wait(ie, -1);
    ie.Stop();

    Start monitoring, go to a page, stop monitoring after the page "settles" meaning some time after onload.

    ie.CloseBrowser();
  32. new HTTPWatch()
          http://github.com/stoyan/etc/

    I did this JavaScript thingie to make everything a little easier.

  33. var http = new HTTPWatch(ff);
    http.go(search.yahoo.com);
    http.done();

    Example usage.

  34. 
    
  35. var har = http.toHAR();
    har = eval(( + har + ));
    
    print(har.log.browser.name);
    print(har.log.browser.version);
    print(# requests: );
    print(har.log.entries.length);

    Opening and navigating browsers is cool. But we need some data back.

    HTTPWatch can export a HAR (HTTP Archive) file. I have this toHAR() method.
    It writes the file, than reads and returns it.
    You can than eval() it because it's just a JSON string.
    And you get the data back in convenient JS objects and arrays.

  36. Internet Explorer 6.0.29...
    # requests: 10
    
    Firefox 3.5.6
    # requests: 15

    Result of running the above.

  37. 
    
  38. var comps = http.getComponentsByType();
          
    for (var i in comps) {
      print(i);
      print(comps[i].length);
    }

    Another method I thought would be useful is getComponentsByType()

  39. redirect: 1
    text/html: 3
    image/gif: 4
    image/png: 3
    text/javascript: 1

    Results of the code above.

  40. But wait...

    There's more :)

  41. What about DOM?

    So far we only talked about HTTP traffic inspection - headers and such.

    Good news is that you can also inspect the DOM (in IE only) for any potential red flags.

    For example having the number of DOM elements sharply increase.

  42. 
    
  43. var http = new HTTPWatch();
    http.go(search.yahoo.com);
    
    var d = http.watch.container.document;
    
    print(d.getElementsByTagName(*).length);
    print(d.documentElement.innerHTML);

    That works!

    All your DOM voodoo skillz are suddenly reusable.

  44. require(statz.js);
          
    var doc = http.watch.container.document;
    var html = http.har.log.entries[0].response.content.text;
    
    var out = statz(document, html);
    print(out.join("\n"));

    This is me repurposing two old bookmarklets that gather some interesting stats (one of them was even featured on Ajaxian, remember?).

    It was pretty easy to repurpose the bookmarklets, because it's just JavaScript.

    The stats thingie can inspect both raw HTML that went over the wire, as well as innerHTML that was the result of any additional DOM manipulations.

  45. JS attributes (e.g. onclick): 1207 bytes
    CSS style attributes: 883
    Inline JS: 5243
    Inline CSS: 5015
    All innerHTML: 17283
    # DOM elements: 134
    
    Total size: 14124 bytes
    Content size: 401 bytes
    Content-to-markup ratio: 0.03

    Sample results.

  46. To summarize...
  47. JavaScript
    WSH
    HTTPWatch
    Monitor
    DOM and HTTP
    IE and Firefox
  48. Thanks
          
    phpied.com

Tell your friends about this post: Facebook, Twitter, Google+

4 Responses

  1. Hi – I want to start HTTPWatch automatically, and I follow your instruction, but I have a runtime error when running this short script.

    var http = new ActiveXObject (“HTTPWatch.Controller” ) ,
    ie = http.IE.New() ,
    ff = http.Firefox.New() ;

    ‘http.IE is null or not an object. Thanks

  2. Heya i am for the primary time here. I came across this board and I in finding It truly helpful & it helped me out much. I hope to offer something again and aid others like you helped me.

  3. [...] Fiddler to monitor and export the traffic and loading a bazillion sites in IE with a script. Or using HTTPWatch. Pros: big sample. Cons: reinvent the wheel and use a different sampling criteria every [...]

  4. We’re a bunch of volunteers and opening a new scheme in our community. Your website offered us with useful info to work on. You have performed an impressive task and our entire group will likely be grateful to you.

Leave a Reply