The techniques used to make it faster are simple and effective: better resource packaging, reducing number of requests, inlining CSS, reducing the amount of CSS and JS by untangling dependencies, cleaning up, and sometimes simply rewriting. You can read more about these in the previous posts.

Now the results. The before-after comparison is accurate because it uses snapshots taken at the same time. This is possible because we kept an old endpoint serving the old code path. The official URL is /plugins/recommendations.php but we kept the legacy URL /widgets/recommendations.php pointing to the old code for a little while.

Before

• WPT: http://www.webpagetest.org/result/130320_JH_7BB/
• Render start: 0.854s
• Fully loaded: 1.187s
• # of requests: 15
• Total payload: 333 KB

After

• WPT: http://www.webpagetest.org/result/130320_SJ_7B8/
• Render start: 0.378s
• Fully loaded: 0.569s
• # of requests: 10
• Total payload: 48 KB

Some analysis

The total payload change is drastic. Mainly due to better packaging. And better JS modules and dependencies.

The number of requests is reduced by 1/3. Not as drastic, but not too bad either. Most of the requests are images, which is ok. They don't block anything and whenever they arrive, they are welcome. But they are not on the critical path. The reduced requests are all JavaScript. We already had a previous optimization so CSS was already inlined. But thanks to rewrites, now the HTML payload (including inline CSS) is 6.2K gzipped, down from 9.4K. Which means the initial paint can start sooner.

The initial paint (render start) now happens in half the time. And, even better, the initial paint is a complete plugin, except for the images. While before it was just a partial content. This is because CSS is here early and all JS is out of the way (async loaded, also take a peek here)

The fully loaded time is not all that important since the user has a usable list of recommendations already delivered with the initial paint. But it's still 2x faster which makes me happy.

All in all

2x faster plugin overall, 2x faster (and infinitely better) first impression. 7x payload improvement.

Making the web faster and other personal notes

Just want to take a second to mention how good it feels to be working on such high-impact performance optimizations. These social plugins are everywhere on the web. By making them faster I am fortunate to have the opportunity to make the whole web faster. Meaning make millions of sites faster, affecting the live of billions of people, every day.

What can I say, Facebook is a great place to work. The people, the impact. Every line you write matters. It's also up to you to pick what do you want to work on and where your talents and interests will have the greatest impact. And then there are the hackathons and hackamonths which means even more freedom.

I recently finished a hackamonth project, which explains why I've been silent here and on Twitter and everywhere (yet, thanks to O'Reilly folks, even though I missed a few deadlines, we were able to push this baby out the door). Let me tell you - a hack-a-month is better than vacation. Being left alone for a month to explore a completely new (to you) territory - priceless!

(Oh, if that sounds something you'd like to do, hit me up on ssttoo at ymail with your resume. FB now has engineering offices in NYC, Seattle and London, so if moving was a problem, now there are more options)

• Browser support: Desktop Safari, iOS Safari, Android Chrome.
• You need to use CSS transforms. Animating regular properties doesn't work.

Update: (see comments) confirmed support in IE10. Reported support in Firefox OS too, but I cannot personally confirm

More details and test page below.

Single UI thread

As you probably know the browser is single threaded. Do something heavy in ECMAScript land and everything freezes.

The big idea

CSS animations should be excluded from the "everything" that freezes.

Test page

Here's a test page with some animations. Click the kill button and see what happens.

Animations

The red box that spins is animated like:

.spin {
  animation: 3s rotate linear infinite;
}
 
@keyframes rotate {
  from {transform: rotate(0deg);}
  to {transform: rotate(360deg);}
}

The green one is also animated with a transform:

.walkabout-new-school {
  animation: 3s slide-transform linear infinite;
}
 
@keyframes slide-transform {
  from {transform: translatex(0);}
  50% {transform: translatex(300px);}
  to {transform: translatex(0);}
}

The blue one is animated using the margin-left property, not a transform:

.walkabout-old-school {
  animation: 3s slide-margin linear infinite;
}
 
@keyframes slide-margin {
  from {margin-left: 0;}
  50% {margin-left: 100%;}
  to {margin-left: 0;}
}

Kill switch

The kill button just pegs the CPU in a infinite loop for 2 seconds:

function kill() {
  var start = +new Date;
  while (+new Date - start < 2000){}
}

Results

In non-supporting browsers, which is most of them, the kill switch kills all the animations. Business as usual.

In the supporting browsers (All Safaris and Andriod Chrome) the kill only affects the blue button, the one that animates a CSS property, as opposed to using a CSS transform. But the animations that use a transform keep on going!

Take aways

1. Rejoice! The future is here! Drink and dance uncontrollably around the campfire!
2. After you sober up, make sure your CSS animations use transform: where possible
3. Keep migrating them JS animations to CSS
4. Bug your browser vendor to support this

What JavaScript?

If you've read the previous post, you see that the most features in a third party widget are possible only if you inject JavaScript from the third party provider into your page. Having "a secret agent" on your page, the provider can take care of problems such as appropriately resizing the widget.

Why is this a problem?

Third party scripts can be a SPOF (an outage), unless you load them asynchronously. They can block onload, unless the provider lets you load it in an iframe (and most don't). There can be security implications because you're hosting the script in your page with all permissions associated with that. And in any case, it's just too much JavaScript for the browser to parse and execute (think of mobile devices)

If you include the most common Like, Tweet and +1 buttons and throw in Disqus comments, you're looking at well over 100K (minified, gzipped) worth of JavaScript (wpt for this jsbin)

This is more than the whole of jQuery, which previous experiments show can take the noticeable 200ms just to parse and evaluate (assuming it's cached) on an iPhone or Android.

What does all of this JS do?

The JavaScript used by third parties is not always all about social widgets. The JS also provides API call utilities, other dialogs and so on. But the tasks related to social widgets are:

1. Find html tags that say "there be widget here!" and insert an iframe at that location, pointing to a URL hosted by the third party
2. Listen to requests from the new iframes fulfill these requests. The most common request is "resize me, please"

Now, creating an iframe and resizing it doesn't sound like much, right? But every provider has to do it over and over again. It's just a wasted code duplication that the browser has to deal with.

Can't we just not duplicate this JavaScript? Can we have a common library that can take care of all widgets there are?

C3PO draft

Here's a demo page of what I have in mind. The page is loading third party widgets: like, tweet, +1 and another one I created just for illustration of the messaging part.

It has a possible solution I drafted as the c3po object. View source, the JS is inline.

What does c3po do?

The idea is that the developer should not have to make any changes to existing sites, other than remove FB, G, Tw, etc JS files and replace with the single c3po library. In other words, only the JS loading part should be changed, not the individual widgets code.

c3po is a small utility which can be packaged together with the rest of your application code, so there will be no additional HTTP requests.

Parsing and inserting iframes

The first task for c3po is to insert iframes. It looks for HTML tags such as

<div class="fb-like" data-href="http://phpied.com"></div>

Similar tags are generated by each provider's "wizard" configuration tools.

In place of this tag, there should be an iframe, so the result (generated html) after c3po's parsing should roughly be like:

<div class="fb-like" data-href="http://phpied.com">
  <iframe 
    src="http://facebook.com/plugins/like.php?href=http://phpied.com">
  </iframe>
</div>

The way to do this across providers is to just have every data- attribute passed as a parameter to the 3rd party URL.

Third parties can be setup using a register() method:

// FB
c3po.register({
  'fb-like': 
    'https://www.facebook.com/plugins/like.php?',
  'fb-send':
    'https://www.facebook.com/plugins/send.php?',
});
 
// Tw
c3po.register({
  'twitter-share-button':
    'https://platform.twitter.com/widgets/tweet_button.html#'
});
 
// ...

The only additional parameter passed to the third party URL is cpo-guid=..., a unique ID so that the iframe can identify itself when requesting services.

The parsing and inserting frames works today, as the demo shows. The only problem is you don't know how big the iframes should be. You can guess, but you'll be wrong, given i18n labels and different layouts for the widgets. It's best if the widget tells you (tells c3po) how big it should be by sending a message to it.

X-domain messaging

What we need here is the iframe hosted on the provider's domain to communicate with the page (and c3po script) hosted on your page. X-domain messaging is hard, it requires different methods for browsers and I'm not even going to pretend I know how it works. But, if the browser supports postMessage, it becomes pretty easy. At the time of writing 94.42% of the browsers support it. Should we let the other 5% drag us down? I'd say No!

c3po is meant to only work in the browsers that support postMessage, which means for IE7 and below, the implementers can resort to the old way of including all providers' JS. Or just have less-than-ideally-resized widgets with reasonable defaults.

When the widget wants something, it should send a message, e.g.

var msg = JSON.stringify({
  type: 'resize',
  guid: '2c23263549d648000',
  width: 200, 
  height: 300
});
parent && parent.postMessage(msg, '*');

See the example widget for some working code.

The c3po code that handles the message will check the GUID and the origin of the message and if all checks out it will do something with the iframe, e.g. resize it.

Again, take a look at the demo code to see how it all clicks together

Next?

As you see in the demo, only the example widget is resized properly. This is because it's the only one that sends messages that make sense to c3po.

Next step will be to have all widget providers agree on the messages and we're good to go! The ultimate benefit: one JS for all your widget-y needs. One JS you can package with your own code and have virtually 0 cost during initial load. And when you're ready: c3po.parse() and voila! - widgets appear.

Of course, this is just a draft for c3po, I'm surely missing a lot of things, but the idea is to have soemthing to start the dialogue and have this developed in the open. Here's the github repo for your forking pleasure.

Make sense? Let's talk.

Facebook's like button is much much faster now than it used to be. It also uses much fewer resources. And lazy-evaluates JavaScript on demand. And so on. But it's still not the only option when it comes to putting a "share this article on Facebook" widgety thing on your site.

The list of options is roughly listed in order of faster (and least features) to slowest (and most features).

#1: A share link

Note that this feature has been deprecated but it still does work. And you see it all over the place.

A simple link to sharer.php endpoint is all it takes. The u parameter is your URL. E.g.:

<a 
  href="https://www.facebook.com/sharer/sharer.php?u=phpied.com" 
  target="_blank">
  Share on Facebook
</a>

The above is a hardcoded URL. You can, of course, spit the current URL on the server side. A JS-only client-side solution could be to take the document.location. You can also pop a window. And use a button, or an image. Say something like:

<button id="sharer">Share</button>
<script>
document.getElementById('sharer').onclick = function () {
  var url = 'https://www.facebook.com/sharer/sharer.php?u=';
  url += encodeURIComponent(location.href);
  window.open(url, 'fbshare', 'width=640,height=320');
};
</script>

Method #1's performance price: none

This is just a link you host in your HTML or bit of JavaScript you can inline or package with your own JavaScript (it is, after all, your own JavaScript)

#2: Feed dialog

The feed dialog a next incarnation of the share popup.

It can also be as simple as a link, like so

https://www.facebook.com/dialog/feed
  ?link=jspatterns.com
  &app_id=179150165472010
  &redirect_uri=http://phpied.com

You need a redirect_uri which can be something like a thank you page. But instead of "thank you", you can simply go back to the article by making redirect_uri and link point to the same URL

Again, a client-only solution could be something like:

  var feed = 'https://www.facebook.com/dialog/feed?app_id=179150165472010';
  var url = encodeURIComponent(location.href);
  feed += '&link=' + url + '&redirect_uri=' + url;
  window.open(feed, 'fbshare', 'width=640,height=480');

The result is a dialog that looks like:

But this feed dialog can also be a popup. You do this by adding &display=popup. This hides the FB chrome. And you can also make the "thank you" page just a simple page that closes the window.

The result:

The other required thing is the app id. You need one. But that's actually cool because it has side benefits. For example better error messages for you (the app admin) that the users don't see. It also gives you a little "via phpied.com" attribution linked to the App URI which is a nice traffic boost hopefully as your sharer's friends see the story in their newsfeed or timeline and click the "via".

So, App ID is good, you can get one here.

Additionally there's a bunch of other params you can pass to the feed dialog to control how the story is displayed. You can provide title, description, image, etc. Full list here.

Method #2's performance price: none

Feed dialog has the same (non-existing) performance requirements as the share links. It's all inline. Any content coming from Facebook is only on user interaction.

BTW, this is the method youtube currently uses.

#3: Feed dialog via JS SDK

Now we move on from simple links and popups to using the JavaScript SDK.

First things first, you absolutely must load the SDK asynchronously. Or non-onload-blocking-asynchronously in an iframe. More on these two later.

After you load the SDK like so:

(function(d, s, id) {
  var js, fjs = d.getElementsByTagName(s)[0];
  if (d.getElementById(id)) return;
  js = d.createElement(s); js.id = id;
  js.src = "//connect.facebook.net/en_US/all.js";
  fjs.parentNode.insertBefore(js, fjs);
}(document, 'script', 'facebook-jssdk'));

Then, whenever you're ready, you can make a call to get the feed dialog:

FB.ui({
  method: 'feed',
  redirect_uri: 'http://phpied.com/files/fb/window.close.html',
  link: 'http://phpied.com',
  // picture: 'http...jpg',
  caption: 'Awesomesauce',
  // description: 'Must read daily!'
});

For a working example, check this example in jsbin

The result:

As you can see, this is now a real properly resized popup. No FB chrome, nice and clean. In general the JS SDK makes everything better. But you need to load it first - the performance price you pay for all the magic.

Method #3's performance price: an async JS

Opening the feed dialog this way requires you to load the Facebook JavaScript SDK. It's one JS file with a short expiration time (20 mins). When it loads, it also makes two additional requests required for cross-domain communication. These requests are small though and with long-expiration caching headers. Since the JS SDK is loaded many times during regular user's surfing throughout the web, these two additional requests have a very high probability of being cached. So is the JSSDK itself. If not cached, at least it's a conditional requests with likely a 304 Not Modified response.

Here's the waterfall of loading the jsbin test page where you can see the JS SDK loading (all.js) and the two x-domain thingies (xd_arbiter.php)

Note that by default the JS SDK sends an additional request checking whether the user is logged in. If you don't need that, make sure you set the login status init property to false, as shown in the test page, like:

FB.init({appId: 179150165472010, status: false});

When loading the JS SDK you must absolutely make sure it's loaded asynchronously, and even better - in an iframe, so the onload of your page is never blocked.

#4: Like button in an iframe

We're coming to the Like button. There are two ways to load it: either you create an iframe and point it to /plugins/like.php or you include the JS SDK and let the SDK create the iframe. Let's take a look at the you-create-iframe option first.

The integration is straightforward: You go to the help page, use the "wizard" configurator found there and end up with something like:

<iframe 
  src="//www.facebook.com/plugins/like.php?href=phpied.com&amp;width=450&amp;show_faces=true&amp;height=80" 
  scrolling="no" 
  frameborder="0" 
  style="border:none; overflow:hidden; width:450px; height:80px;" 
  allowTransparency="true"></iframe>

You're done!

The button comes in three layouts: standard (biggest), box_count and button_count

As you can see, you get quite a bit more features here, e.g. number of likes and social context (who else has liked) in the standard layout. Also in the standard layout you get a little comment input. You don't get one in the other layouts because there's no space in the little iframe. You define the iframe and the code inside the iframe cannot break out of it and do something wild (or useful), e.g. open a big commenting dialog. Or make the iframe bigger because the word "Like" may be significantly longer in some languages. When you "trap" the iframe in your dimensions, it stays there.

Method #4's performance price: iframe content

In this method every time someone loads your page, they also visit a page (like.php) hosted by facebook.com. Now, this page is highly optimized: it only has html, sprite and async lazy-executed JS (which doesn't block onload). 3 requests in total. Maybe some faces (profile photos), depending on the layout and whether the user's friends have liked the URL.

As you probably know, every iframe's onload blocks the parent window's onload. So, if you feel so inclined you can always do any old lazy-load trick in the book. E.g. create the iframe after window.onload, or "double-frame" it, or (for the webkits out there) write the iframe src with a setTimeout of 0.

Another thing to consider is to always load the iframe via https, so there's no http-https redirect if the user has opted to always use facebook via https.

#5: Like button via SDK

This is building on what you already know about #3 and #4: You load the SDK. You sprinkle <fb:like> (or <div class="fb-like">) where you want buttons to appear. The SDK finds these and replaces them with iframes.

<!-- all defaults -->
<fb:like></fb:like>
 
<!-- layout, send button -->
<div class="fb-like" data-send="true"></div>

If you don't need to specify the URL to like, it's the current page.

This is the most full-featured button implementation. It will resize the button as required by content and i18n. It will always present a comment dialog. (When people share with their own comment, these stories do better, because it's always nice to see a friend's comment attached to a URL, right?)

The good thing about this method is that you can load any other FB plugin (e.g. follow button by just adding an fb:follow in the HTML) without re-loading the SDK, it's already there and can handle all the plugins, dialogs and API requests.

Method #5's performance price: JSSDK + iframe content

Combining the features of methods #3 and #4 also combines their perfromance impact. Again, the like.php iframe is heavily optimized and tiny. Also the SDK has a chance of being cached from the users visit on another page. And, of course, you always load the SDK asynchronously so it's impact on your initial page loading is minimal. Or load the SDK in an iframe so the impact is virtually 0.

So the total cost in terms of number of requests in empty cache view is 6. 3 from the iframe + 3 from the SDK. Full cache view should be 1 request - just the like.php frame with the current count, faces and so on.

But again, to minimize the impact, you just load the SDK in an iframe (so the whole widget doesn't block onload and doesn't SPOF) or asynchronously (so it doesn't SPOF and doesn't block onload in IEs)

Summary

I mentioned a few times in the article but let me repeat once again for the TL;DR folks. If you're loading the JS SDK, it's absolutely mandatory that you make sure it's either loaded asynchronously to avoid SPOF, or even better - in an iframe to avoid blocking onload.

1. You write a new test to confirm a JavaScript-related performance speculation
2. You click
3. Your test runs in a bunch of browsers

Glossary

JSperf.com is the site where all you JavaScript performance guesswork should go to die or be confirmed. You know how the old wise people say "JSperf URL or it didn't happen! Now off my lawn!". Yup, that jsperf.com

WebPagetest.org (WPT) is the site where you get answers to the ol' question: "Why do people say my oowsome site is slow? And what should I do about it?"

Bookmarklet is a little piece of JavaScript you conveniently access from your browser bookmarks and inject into other non-suspecting sites.

Github is where you host code.

Bookmaker tool makes a bookmarklet from a .js file URL (probably hosted on github)

Trouble in paradise

These days we're so happy and spoiled with all these amazing tools around us. And yet, when you create a JSPerf test, you have to open all these browsers and run the test everywhere. Even IE. And, when on Mac, IE is usually not readily available. Plus it comes in a bunch versions - from almost-but-not-quite-forgotten IE6, all the way to IE10 The Greatest - and they have different, sometimes contradicting, performance characterics.

To the rescue: WPT

WebPagetest has: a/ ability to run in a bunch of browsers and b/ an API

The bookmarklet

The bookmarklet. It's here, on github

It starts by inquiring about your WPT API key. I know, you have to get one. You can read the API docs on how to get one, but let me save you the trip: you just need to ask pmeenan@[the tool's domain].org for a key. Politely. Tell him I sent you. Promise not to abuse.

  var key = localStorage.wpt_key;
  if (!key) {
    var prompt = window.__proto__.prompt;
    key = prompt('Your WebPageTest API key, please?');
    if (!key) {
      return gameOver();
    }
    localStorage.wpt_key = key;
  }

The key is stored in your localStorage so you don't have to paste it all the time.

Oh, you may wonder what's up with that:

var prompt = window.__proto__.prompt;
prompt('Message...');

Looks like something somewhere on jsperf is doing window.prompt = function(){}, same for window.open and probably others. Makes sense, you don't want popup-y stuff (by the thousands) while running a test a gazilion times. So the bookmarklet has to go the window.__proto__ for the original prompt

Moving on.

Setting up the constant params of the API call. The variable param will be the location which will tell what browser to use. We also give the (undocumented) time a value of 60s, so that the test has time to run. We also want only one run and just the first run (no full cache run).

The URL to test will be the current page loaded in jsperf.com which is where you run the bookmarklet. And we'll append #run for autorun.

  // base params
  var wpt = 'http://www.webpagetest.org/runtest.php?';
  var params = {
    k: key,
    time: 60,
    runs: 1,
    fvonly: 1,
    url: 'http://jsperf.com' + location.pathname + '#run'
  };
  Object
    .keys(params)
    .forEach(function(key) {
      wpt += key + '=' + encodeURIComponent(params[key]) + '&';
    });

Finally, setup the locations with browsers IE6,7,8,9,10 and open all these browser windows:

  var locations = localStorage.wpt_locations;
  if (!locations) {
    locations = ['Dulles_IE6', 'Dulles_IE7', 'Dulles_IE8', 'Dulles_IE9', 'Dulles_IE10'];
  }
  
  // pop some windows up
  var open = window.__proto__.open;
  locations.forEach(function(loco){
    open(wpt + 'location=' + encodeURIComponent(loco));
  });

Again, the full source is here on github

Github has a "raw" version, e.g. this so we take this url, paste it in the bookmaker tool and we get a nice installable bookmarklet link.

Install

Drag this link to you bookmarks:

jsperf -> wpt

Run

1. Go to any jsperf test, e.g. http://jsperf.com/array-proto-vs/3
2. Click the bookmarklet
3. Observe 5 new tabs with 5 IE versions running your test!

More browsers

In addition to the browsers (locations) I've defined you can always add more, like Chrome and Firefox. However you probably have these already handy so no need to kill WPT's servers. But the option is there, just edit your localStorage.wpt_locations

Thanks for reading! Comments welcome!

Use case

Think a "single page app" use case. You click a button. Content comes via XHR. But content is complex (and app is as lazy-loading as possible) and content requires extra CSS. In an external file.

Only when the external CSS arrives should the app show the content. Otherwise content will be weirdly styled.

Execution

Two "modules" (or "widgets") of the app require two different CSS files. Both modules are requested at about the same time. We listen to onload of the CSS files. Expected behavior: whenever a module and its CSS dependency arrive - show that module. Asynchronously. No one cares which module shows first, as long as they show up as soon as possible.

Experimentation

Two modules. Two CSS files. 1st CSS happens to take one second. The second CSS takes 5 seconds.

Test pages: one and two

Here's the end result. The first module is pinky, the second is yellow. All good.

Same with network panel ON:

The question is what does the user see during the ?-mark - between the first CSS is done and the second one is still loading.

Oh, and here's the load() function that runs when the user clicks the button "load" initiating the new modules to appear:

function load() {
  var these = ['class1.css.php', 'class2.css.php'];
  var classes = ['class1', 'class2'];
  var head = document.getElementsByTagName('head')[0];
  
  for (var i = 0; i < these.length; i++) {
    var url = these[i];
    var link = document.createElement('link');
    
    link.type = "text/css";
    link.rel = "stylesheet"
    link.href = url;
    link.onload = (function (i) {
      return function () {
        console.log(these[i]);
        var div = document.createElement('div')
        div.appendChild(document.createTextNode(these[i]));
        result.appendChild(div);
        div.className = classes[i];
        //s = getComputedStyle(result).height;
      }
    }(i));
    
    head.appendChild(link);
  }
 
}

Expected behavior

In FF, whenever the the first CSS is loaded, we see a new module.

Test for yourself (in Firefox)

#1 issue: "efficient" webkit

You know that browsers batch layout and paints tasks because these tend to be expensive. For example they wait for all CSS (even useless print and other @media stylesheets) to arrive and block the rendering of the page. (More on these topics: here, here)

So turns out that here webkit also waits for both CSS files to arrive before rendering anything.

You see in the console we know (in JavaScript) that CSS has arrived. But webkit (chrome, safari, mobile safari) doesn't paint anything, waiting for the second CSS. Bummer!

Issue #2: painting unstyled content

While issue #1 is just a bummer that can be done better for the progressive feedback-y user experience, #2 is a bug. This is the issue that Jeff and Andrey found and were floored.

If there's a paint going on between the two stylesheets, the browser dumps the unstyled content on the page. Ugly stuff.

This was only happening sometimes, but after forming and testing an hypothesis, I was able to distill a reproducible test case. The only change is: after the load of the first CSS, flush the rendering queue by requesting a style information. e.g.

link.onload = function () {
  s = getComputedStyle(dom).height;
}

Repro and file a bug

You can reproduce for yourself. Use chrome and try:

1. Issue: no paint till all CSS is here
2. Bug: unstyled paint while waiting for all CSS

I was faced with "registration wall" trying to file a webkit bug, hence this post. Someone please file a bug and feel free to use the provided test cases.

The solution, IMO, is to make webkit behave like FF. No waiting for all CSS. This solves both issues. In the worst case, at least the unstyled bug (issue #2) should be addressed.

Interim solution for web developers: inline CSS required by the module together with the module content.

Thanks for reading!

Task: figure out what % of the JPEGs out there on the web today are progressive vs baseline. Ann Robson has an article for the perfplanet calendar later tonight with all the juicy details.

Problemo: there's no such information in HTTPArchive. However there's table requests with a list of URLs as you can see in the previous post.

Solution: Get a list of 1000 random jpegs (mimeType='image/jpeg'), download them all and run imagemagick's identify to figure out the percentage.

How?

You have a copy of the DB as described in the previous post. Now connect to mysql (assuming you have an alias by now):

$ mysql -u root httparchive

Now just for kicks, let's get one jpeg:

mysql> select requestid, url, mimeType from requests \
    where mimeType = 'image/jpeg' limit 1;
+-----------+--------------------------------------------+------------+
| requestid | url                                        | mimeType   |
+-----------+--------------------------------------------+------------+
| 404421629 | http://www.studymode.com/education-blog....| image/jpeg |
+-----------+--------------------------------------------+------------+
1 row in set (0.01 sec)

Looks promising.

Now let's fetch 1000 random images, while at the same time dump them into a file. For convenience let's make this file a shell script so it's easy to run. And the contents will be one curl command per line. Let's use mysql to do all the string concatenation.

Testing with one image:

mysql> select concat('curl -o ', requestid, '.jpg "', url, '"') from requests\
    where mimeType = 'image/jpeg' limit 1;
+-----------------------------------------------------------+
| concat('curl -o ', requestid, '.jpg "', url, '"')         |
+-----------------------------------------------------------+
| curl -o 404421629.jpg "http://www.studymode.com/educ..."  |
+-----------------------------------------------------------+
1 row in set (0.00 sec)

All looks good. I'm using the requestid as file name, so the experiment is always reproducible.

mysql>
 SELECT concat('curl -o ', requestid, '.jpg "', url, '"') 
  INTO OUTFILE '/tmp/jpegs.sh' 
  LINES TERMINATED BY '\n' FROM requests
  WHERE mimeType = 'image/jpeg'
  ORDER by rand() 
  LIMIT 1000;

Query OK, 1000 rows affected (2 min 25.04 sec)

Lo and behold, three minutes later, we have generated a shell script in /tmp/jpegs.sh that looks like:

curl -o 422877532.jpg "http://www.friendster.dk/file/pic/user/SellDiablo_60.jpg"
curl -o 406113210.jpg "http://profile.ak.fbcdn.net/hprofile-ak-ash4/370543_100004326543130_454577697_q.jpg"
curl -o 423577106.jpg "http://www.moreliainvita.com/Banner_index/Cantinelas.jpg"
curl -o 429625174.jpg "http://newnews.ca/apics/92964906IMG_9424--1.jpg"
....

Now, nothing left to do but run this script and download a bunch of images:

$ mkdir /tmp/jpegs
$ sh ../jpegs.sh

curl output flashes by and some minutes later you have almost 1000 images, mostly NSFW. Not 1000 because of timeouts, unreachable hosts, etc.

$ ls | wc -l
     983

Now back to the original task: how many baseline and how many progressive JPEGs:

$ identify -verbose *.jpg | grep "Interlace: None" | wc -l
     XXX
$ identify -verbose *.jpg | grep "Interlace: JPEG" | wc -l
     YYY

For the actual values of XXX and YYY, check Ann's post later tonight

Also turns out 983 - XXX - YYY = 26 because some of the downloaded images were not really images, but 404 pages and other non-image files.

One way to do web performance research is to dig into what's out there. It's a tradition dating back from Steve Souders and his HPWS where he was looking at the top 10 Alexa sites for proof that best practices are or aren't followed. This involves loading each pages and inspecting the body or the response headers. Pros: real sites. Cons: manual labor, small sample.

I've done some image and CSS optimization research grabbing data in any way that looks easy: using the Yahoo image search API to get URLs or using Fiddler to monitor and export the traffic and loading a bazillion sites in IE with a script. Or using HTTPWatch. Pros: big sample. Cons: reinvent the wheel and use a different sampling criteria every time.

Today we have httparchive.org which makes performance research so much easier. It already has a bunch of data you can export and dive into immediately. It's also a common starting point so two people can examine the same data independently and compare or reproduce each other's results.

Let's see how to get started with the HTTP archive's data.

(Assuming MacOS, but the differences with other OS are negligible)

1. Install MySQL
2. Your mysql binary will be in /usr/local/mysql/bin/mysql. Feel free to create an alias. Your username is root and no password. This is of course terribly insecure but for a local machine with no important data, it's probably tolerable. Connect:

$ /usr/local/mysql/bin/mysql -u root

You'll see some text and a friendly cursor:

...
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

3. Create your new shiny database:

mysql> create database httparchive;
Query OK, 1 row affected (0.00 sec)

Look into the new DB, it's empty, no tables or data, as expected:

mysql> \u httparchive
Database changed
mysql> show tables;
Empty set (0.00 sec)

4. Quit mysql for now:

mysql> quit;
Bye

5. Go the archive and fetch the database schema.

$ curl http://httparchive.org/downloads/httparchive_schema.sql > ~/Downloads/schema.sql

While you're there get the latest DB dump. That would be the link that says IE. Today it says Dec 15 and is 2.5GB. So be prepared. Save it and unzip it as, say, ~/Downloads/dump.sql

6. Recreate the DB tables:

$ /usr/local/mysql/bin/mysql -u root httparchive < ~/Downloads/schema.sql

7. Import the data (takes a while):

$ /usr/local/mysql/bin/mysql -u root httparchive < ~/Downloads/dump.sql

8. Log back into mysql and look around:

$ /usr/local/mysql/bin/mysql -u root httparchive;

[yadda, yadda...]

mysql> show tables;
+-----------------------+
| Tables_in_httparchive |
+-----------------------+
| pages                 |
| pagesmobile           |
| requests              |
| requestsmobile        |
| stats                 |
+-----------------------+
5 rows in set (0.00 sec)

Dataaaa!

What's in the requests table I couldn't help but wonder (damn you, SATC)

mysql> describe requests;
+------------------------+------------------+------+-----+---------+----------------+
| Field                  | Type             | Null | Key | Default | Extra          |
+------------------------+------------------+------+-----+---------+----------------+
| requestid              | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| pageid                 | int(10) unsigned | NO   | MUL | NULL    |                |
| startedDateTime        | int(10) unsigned | YES  | MUL | NULL    |                |
| time                   | int(10) unsigned | YES  |     | NULL    |                |
| method                 | varchar(32)      | YES  |     | NULL    |                |
| url                    | text             | YES  |     | NULL    |                |
| urlShort               | varchar(255)     | YES  |     | NULL    |                |
| redirectUrl            | text             | YES  |     | NULL    |                |
| firstReq               | tinyint(1)       | NO   |     | NULL    |                |
| firstHtml              | tinyint(1)       | NO   |     | NULL    |                |
| reqHttpVersion         | varchar(32)      | YES  |     | NULL    |                |
| reqHeadersSize         | int(10) unsigned | YES  |     | NULL    |                |
| reqBodySize            | int(10) unsigned | YES  |     | NULL    |                |
| reqCookieLen           | int(10) unsigned | NO   |     | NULL    |                |
| reqOtherHeaders        | text             | YES  |     | NULL    |                |
| status                 | int(10) unsigned | YES  |     | NULL    |                |
| respHttpVersion        | varchar(32)      | YES  |     | NULL    |                |
| respHeadersSize        | int(10) unsigned | YES  |     | NULL    |                |
| respBodySize           | int(10) unsigned | YES  |     | NULL    |                |
| respSize               | int(10) unsigned | YES  |     | NULL    |                |
| respCookieLen          | int(10) unsigned | NO   |     | NULL    |                |
| mimeType               | varchar(255)     | YES  |     | NULL    |                |
.....

Hm, I wonder what are common mime types these days. Limiting to 10000 or more occurrences of the same mime type, because there's a lot of garbage out there. If you've never looked into real web data, you'd surprised how much misconfiguration is going on. It's a small miracle the web even works.

9. Most common mime types:

select count(requestid) as ct, mimeType 
  from requests 
  group by mimeType 
  having ct > 10000 
  order by ct desc;

+---------+-------------------------------+
| ct      | mimeType                      |
+---------+-------------------------------+
| 7448471 | image/jpeg                    |
| 4640536 | image/gif                     |
| 4293966 | image/png                     |
| 2843749 | text/html                     |
| 1837887 | application/x-javascript      |
| 1713899 | text/javascript               |
| 1455097 | text/css                      |
| 1093004 | application/javascript        |
|  619605 |                               |
|  343018 | application/x-shockwave-flash |
|  188799 | image/x-icon                  |
|  169928 | text/plain                    |
|   70226 | text/xml                      |
|   50439 | font/eot                      |
|   45416 | application/xml               |
|   41052 | application/octet-stream      |
|   38618 | application/json              |
|   30201 | text/x-cross-domain-policy    |
|   25248 | image/vnd.microsoft.icon      |
|   20513 | image/jpg                     |
|   12854 | application/vnd.ms-fontobject |
|   11788 | image/pjpeg                   |
+---------+-------------------------------+
22 rows in set (2 min 25.18 sec)

So the web is mostly made of JPEGs. GIFs are still more than PNGs despite all best efforts. Although OTOH (assuming these are comparable datasets), PNG is definitely gaining compared to this picture from two and a half years ago. Anyway.

It's you time!

So this is how easy it is to get started with the HTTPArchive. What experiment would you run with this data?

Traditions are traditions. Marry Christmas with a new cover, this time it's Deck the Halls:

Deck the halls (part one)

(It's only part one because I didn't find the time to properly record what I had in mind - another 7/8th cover like last year. But you can hear the idea at the end)

Here are the previous years for your entertainment.

• We Wish You a Merry Christmas (7/8th and 9/8th)
• Feliz Navidad 2009
• Jingle Bells 2008

Here's to a happy new 2013!

Gear: Ovation wine-colored guitar (depicted above) plugged in one channel of M-Audio box. AKG C452 microphone ponited at 12th fret and plugged in the other channel of the M-Audio box. Out of the box and into Garageband.

Let's check it out.

Here's the demo, but first a prerequisite: go chrome://flags, search for "Web Audio Input" and enable it. Restart Chrome Canary.

With a guitar

I wanted to have a little trickier setup and capture guitar sound not just voice with a microphone.

As always, it was bigger hurdle to get guitar sound to the computer, than anything else JavaScript-wise.

I have a guitar amp that has a mini-USB out. This goes to the USB of the computer. Wrestle, system settings, garage band to the rescue.... eventually the computer makes sound.

Capturing

I was assuming the stream you get from getuserMedia can go directly to an HTML <audio> src. No such luck. Works for video but not yet for audio.

So... WebAudio API saves the day.

Setting up audio context (like in the previous post), shimming getUserMedia and setting up callbacks for it:

  // for logging
  function fire(e, data) {    
    log.innerHTML += "\n" + e + " " + (data || '');
  }
 
  // globals
  var audio_context;
  var volume;
 
  // one-off initialization
  (function init(g){
    try {
      audio_context = new (g.AudioContext || g.webkitAudioContext);
      fire('Audio context OK');
      // shim
      navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia;
      fire('navigator.getUserMedia ' + (navigator.getUserMedia ? 'OK' : 'fail'));
      // use
      navigator.getUserMedia(
        {audio:true},
        iCanHazUserMedia, 
        function(e){fire('No live audio input ' + e);}
      );
    } catch (e) {
      alert('No web audio support in this browser');
    }
  }(window));

When the user loads the page, here's what they see:

In my case I select the guitar amp and click "Allow" button.

This little window informs me the page is using the audio input:

Playing back

Now that the user has allowed audio access, let's play back the audio we receive, but pass it through a volume control.

All this work happens in the iCanhazUserMedia(), the success callback to getUserMedia.

  function iCanHazUserMedia(stream) {
    
    fire('I haz live stream');
    
    var input = audio_context.createMediaStreamSource(stream);
    volume = audio_context.createGainNode();
    volume.gain.value = 0.8;
    input.connect(volume);
    volume.connect(audio_context.destination);
    
    fire('input connected to destination');
  }

What we have here (ignoring fire()):

1. setup an input stream from the user stream, this is the first node in the audio chain
2. setup a volume (Gain) node with initial volume 0.8 out of 1
3. connect input to volume to output/speakers

And this is it!

Additionally an input type=range max=1 step=0.1 can change the volume via volume.gain.value = value;

Go play! Isn't it amazing that you can now grab microphone or any other audio input and play around with it? All in JavaScript, all in the browser without any plugins.

Moar!

This was a very basic exploratory/primer example. For more:

• Pedalboard.js with more effects besides volume
• One of Chris Wilson's demos has even more effects
• and how 'bout a guitar tuna