phpied.com

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 13 This post is part of the 2009 performance advent calendar experiment. Stay tuned for the articles to come.

People are often afraid to use PNG because they think that:
a/ it doesn't work in all browsers, or
b/ filesizes are bigger than GIF

While these have some grain of truth to them, they are mostly misconceptions. Before addressing them, one quick background point - what's PNG8 and why it's cool.

PNG8

There are several types of PNG files, which can be grouped into those main kinds:

• Truecolor PNGs with or without alpha transparency channel, also known as PNG24 and/or PNG32 (the one with alpha)
• Grayscale PNGs with or without alpha
• Indexed PNG, aka palette PNG, aka PNG8

PNG8 is like a GIF - it has a palette of 256 colors and supports transparency. While GIF supports true/false transparency (a pixel is either transparent or it isn't), PNG8 supports variable alpha transparency. Right there you see - PNG8 can do anything that GIF can, plus more.

There's a little glitch in IE6 where a semi-transparent pixels in PNG8 are seen as fully transparent, just like a GIF. So here's an option for progressive enhancement - you use the same image and IE6 gets a degraded GIF-like experience, while modern browsers get the full experience.

Here's an example, taken from this excellent article - modern browsers get the light bulb with the glow:

IE6 and under get the gracefully degraded experience and no glow:

Another pain point is that Photoshop doesn't produce semi-transparent PNG8 (although they came up with the name "png8" instead of saying palette or indexed PNG). Only Fireworks does export alpha transparent PNG8, which makes it a bit of a challenge. You also need a good designer to undertake this tricky task of making sure the image looks OK in both experiences. One way is to assume you're working with a GIF and then upgrade the experience with the carefully selected semi-transparent pixels. It could also help you keep the gif-like version in a layer and use other layers for the semi-transparent stuff, so you can quickly preview what the image will look like in IE6.

In any event - the important thing to remember is that in the worst case (IE6) PNG8 is as good as a GIF.

PNG doesn't work in browsers?

PNG works in browsers since forever with the exception of two edge cases:

• the glitch where PNG8's semi-transparency is gone in IE6 (see above), but here GIF can't help you either
• transparency in truecolor PNGs is shown as a solid (usually grey) color in IE6. But again - GIF can't help here either, because it doesn't support alpha (variable) transparency to begin with. People often use GIF to "solve" this problem (moving to GIF will mean potentially losing colors), but if you can solve with a GIF, you can solve it even better (and with smaller filesize) with PNG8

Another solution to the second problem is to use IE's AlphaImageLoader CSS filter (and there's a number of scripts to do so automatically), but this filter has serious performance drawbacks and should only be used as a last resort. Three things to try before resorting to AlphaImageLoader:

1. Try PNG8 for progressively enhanced experience
2. Try without transparency - if the background is a solid color, convert the image to use the solid color. In imagemagick you can use -flatten for this purpose:

   $ convert source.png -flatten -background yellow result.png

3. Forget about IE6

If you end up using AlphaImageLoader, make sure you use the underscore hack so that only IE6 users experience the performance degradation.

#some-element {
    background: url(image.png);
    _background: none;
    _filter:progid:DXImageTransform.Microsoft.AlphaImageLoader(src='image.png', sizingMethod='crop');
}

PNGs are bigger than GIFs?

This misconception comes from the fact that people compare truecolor PNG with GIF which is not a fair comparison because you often compare image with thousands of colors (the PNG24) with an image with 256 colors (GIF). Often people work on an image in Photoshop or another program and when they decide to export for the web, they try PNG24, see that it's bigger and switch to GIF. But in this step GIF may strip a lot of colors. And if you're going to strip colors, well, PNG8 will give you the same colors and smaller filesize. (Another thing is that sometimes Photoshop does a poor job exporting the PNG8. If the PNG8 looks crappy, but the GIF is OK, then export as a GIF but then convert to PNG with another utility, such as optipng)

Again - PNG8 is the file format comparable to GIF and it's almost always smaller in filesize than GIF.

Comparing GIF vs. PNG filesizes

(This and the next experiment is something I did over an year ago, bored to death in the middle of the ocean on the board of a Carnival cruise ship, but since then I never really looked at the data. So here's my chance to flush some old data and clean up 20 Gigs of lets-keep-just-in-case test images )

Using Yahoo! image search web service I downloaded some GIFs (matching the queries "logo", "mail" and "graph"), ended up a little over 1700 images. Then I used optipng to convert them all to PNG and see the results.

I used OptiPNG simply with no special options:

$ optipng *.gif

As the next experiment will show, optipng can do better, so can pngout for example. So consider these results the least you can do to make GIFs smaller (by turning them to PNG)

So some stats from the experiment:

• The average, median actually, GIF image on the web (last year, judging from this small sample) is 525x388 and has 139 colors (I just love semi-useless stats )
• The median GIF is 24K
• After conversion to PNG, the median becomes 18K
• The median savings from converting all GIFs to PNG is about 23%

Interestingly enough 4% of the images were smaller as GIFs - utter disappointment (and don't tell anyone!). So I had to try just a little harder. I didn't run OptiPNG with its best -o7 option, but ran PNGOut instead. The results is that now only 4 of the 1706 images were smaller as GIF. I'm pretty sure that trying a little harder (with PNGSlim, see yesterday's post) would've probably fixed it, but 4 out of 1700 is something I could live with. BTW, the images where OptiPNG failed to produce smaller PNG, then PNGOut converted with the ratio of 21% median savings. Not bad for taming the few shrew GIFs.

BTW, some GIFs lost over 100K of filesize, the max was over almost 600K savings! So, you never know.

If you like to look at numbers, here's a csv dump - the optipng results and the selected few that ran through PNGout.

So, take-home message: turn your GIFs into PNGs and win at minimum 20% fewer bytes over the network.

Comparing PNG optimizers

For this experiment I downloaded over 12000 images (again, Yahoo! search API) and ran them through a bunch of optimizers, sometimes with different options. In retrospect, it's probably not that useful of an experiment, because (see previous post) different optimizers specialize in different areas - compression, pre-compression filtering, chunks removal, etc, and your best bet is to run several tools. But still it's at least some data points (the cvs dump is here)

The images were 1000 matches for each of the searches for "baby", "background", "bkg", "flower", "graph", "graphic", "icon", "illustration", "kittens" (of course), "logo", "monkeys", "png", "transparency". After removing 4xxs, 5xxs and other mishaps and cleaning up a bit, I ended with over 10000 images. I ran them thorough:

• pngcrush - pngcrush -rem alla -reduce before.png after.png
• pngcrush-none - to keep all chunks pngcrush -rem none -reduce before.png after.png
• pngcrush-brute - more filter attempts - pngcrush -rem alla -brute -reduce before.png after.png
• pngout - pngout /q /y /force before.png after.png. default compression level in PNGOut is "extreme", so I tried two less extreme below
• pngout-match - pngout /s2 /q /y /force before.png after.png
• pngout-intense - pngout /s1 /q /y /force before.png after.png
• pngrewrite - pngrewrite before.png after.png PNGRewrite only works with PNG8, it also converts truecolor to PNG8 whenever the truecolor happens to be under 256 colors,
• optipng - optipng before.png -force -out after.png. OptiPNG's default level is 2 (of 7) so I had to try below and above the default:
• optipng1 - optipng before.png -o1 -force -out after.png
• optipng3 - optipng before.png -o3 -force -out after.png
• optipng7 - optipng before.png -o7 -force -out after.png
• advpng - cp before.png after.png; advpng -z -f -q after.png
• advpng-insane - with the "insane" 4th level of compression cp before.png after.png; advpng -z4 -f -q after.png
• deflopt - cp before.png after.png; deflopt -s -f after.png
• pngoptimizercl -cp before.png after.png; pngoptimizercl -file:"after.png"

And the results:

"Success rate" is how often the tool managed to produce a smaller result than the original. For example PNGRewrite's success rate is pretty low, because it only works with up to 256 colors. Median time to run is the median value that the tool takes to optimize one image.

And now, madames et monsieurs, introducing...

Give PNG a chance (.com)

I hope you'll find this as funny as I do, I thought it was pretty funny, at least in my head

My secret goal was that everybody who hears the song or watches the video, will think twice the next time when doing "Save for the Web..." in Photoshop.

Enjoy!

Music: Drums from GarageBand, I play two guitars, also bass (a guitar with effect actually) and vocals. If you think you hear a woman's voice, it's still me, with "Helium" effect. The MP3 is here. If you want to experiment with the song yourself, here's a zip with each channel as an MP3.

Video: It may be lousy, but it's all web dev It's all JavaScript and CSS. The video is a screen capture of the Safari window. Also there are no images, only HTML entities. Heavy use of -webkit-* animations and transitions. The source and a live version you can play in Safari is here. The StarWars-like effect is borrowed from here.

The http://givepngachance.com URL is currently pretty blank, but I intend to add more PNG-related stuff there. Oh, and the lyrics.

Thanks!

Thanks for reading. And watching. And listening. Peace. And give PNG a chance

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 12 This post is part of the 2009 performance advent calendar experiment (12 articles down, 12 more to go). Stay tuned for the articles to come.

Let's continue the topic of reducing file sizes started with the previous post and talk about making images smaller.

Engineer's guide to smaller images

Just to set the frame of the discussion - this is not about using Photoshop or setting the quality of the JPEGs and so on. I realize that we, web developers, wear many hats - we're designers, client/server coders, Apache/Linux admins, database heros. But this post is not about using image programs and assumes that you or your designer has already created the images to be used on the site with the appropriate colors, quality and so on.

Now, repeat after me: you should never take an image from the designer and put it up on the web.

Most often this image is bigger than it should be. It's not the designer's fault, it's usually the software used to produce the image.

You shouldn't put an image up on your server before running it through a few tools. These tools are free, open source, cross-platform and can be run on the command line, hence scripted and run in batches over a large number of files - by you, or even better, automatically by a build deployment process. It's OK to batch-run those files without human intervention, because these tools simply optimize the files, they don't change the pixel information, so the "after" images look exactly like "before", only smaller.

Selecting the right file format

The first step towards leaner images is to select the correct file format. There are three options:

1. JPEG for photos. Photos contain millions of colors and smooth transitions of colors. Blue skies, clouds, sunset, your dog, lolcats - all photos.
2. GIF is for the occasional "loading..." progress animation. This is it, no other uses for GIFs.
3. PNG is for everything that's not a photo or an animation. That includes all icons, graphs, buttons, gradients, and what not. Any image with sharp transitions of colors. Think (but don't use for) text. Sharp transitions become "dirty" in JPEG.

PNG is an interesting topic for a follow up post, for now let's stop here. If it's not a photo, it should be PNG. An edge case is a screenshot for example. Depends on what's on the screen of course, but most often JPEG will give a smaller size if you can live with the artifacts around the sharp edges (like text).

Optimizing GIFs

Unfortunately many people still use GIFs even for non-animated images. That's a mistake. PNG is a superior format and yields smaller file sizes.

People still use GIFs because they think either that a/ GIF is smaller than PNG or b/ there's lack of support for PNG in browsers. These are misconceptions and I'll talk more about them tomorrow.

So, the way to optimize a GIF is to convert it to PNG.

You can use many tools to turn your GIFs to PNGs, including ImageMagick and OptiPNG.

# option 1: ImageMagick (if you know the filename)
$ convert logo.gif logo.png

# option 2: ImageMagick again (if you just convert all files in a directory)
$ mogrify -format png *.gif

# option 3: OptiPNG
$ optipng *.gif

These are just some of the options, I'm sure there are others.

After the conversion you can optimize the new PNGs like all other PNGs

Optimizing PNGs

There are various ways to write a PNG file. Unfortunately not all image editing programs do a good job at writing PNGs for the minimal file size.

Luckily, to fill the void, there's a great number of tools that excel in writing small PNGs. There are different ways to optimize a PNG:

1. Stripping out "chunks" - PNG is an extensible format. Extensions come in the form of chunks and most chunks are not needed for the web.
2. Reducing the number of colors and switching between PNG types - truecolor PNG, grayscale, palette...
3. Chosing the best "filter". Filters are a pre-compression step. You can compress any type of file, but when you know the file is an image, you can do better. Filters are for this purpose.
4. Optimizing the actual DEFLATE compression algorithm

Different tools specialize in one or more of these areas. So the more tools you run, the better the results will be. But you have to run at least one tool, always. You'll be surprised how unoptimized are most PNGs coming from common commercial image programs.

So, to optimize a PNG you shoul run as many of the following programs as possible:

# optipng (skip -o7 to run faster)
$ optipng -o7 my.png

# pngcrush (skip -brute to run faster)
$ pngcrush -rem alla -brute -reduce my.png my.png.temp
$ mv my.png.temp my.png

# pngout - closed source, non-windows binaries here
# (add parameter -s2 to run faster)
$ pngout my.png

# advpng (use -z2 to run faster)
$ advpng -z4 my.png

# deflopt - windows only
$ deflopt my.png

Other tools to note include PNGrewrite, PNGNQ and PNGquant, but they are limited because they deal only with PNG8 (256 colors) files. PNGNQ and PNGQuant are actually converters from truecolor to PNG8, so they are not guaranteed to be lossless. PNGreqwrute is safe to use, it will just silently fail if the file has more than 256 colors, so there's nothing to lose.

Oh, and another, excellent tool - PNGOptimizer, windows-only has both command line interface and a GUI.

PNGSlim for the hardcore PNG optimization

If you're really serious about optimizing your PNGs, the tool is called PNGSlim. It's a Windows-only batch file that runs pretty much all tools above and runs them (especially PNGOut) with all kinds of parameters, hundreds of times. So it can take a while to run.

Optimizing JPEGs

JPEG is a lossy format (you lose information every time you save it, even if you choose 100% quality), but there are some operations that can be done losslessly - such as tweaking comments and meta information, cropping, rotating to 90, 180, 270 degrees. The tool that does this magic is called JPEGTran and is likely already on your unix/linux box. If not - here's how to install it (for Windows - get the .exe here)

So to optimize a JPEG losslessly you remove the meta information and optimize the so called Huffman tables. For bigger JPEGs (bigger than 10K) you can also convert the image to progressive coding.

# strip meta and optimize
$ jpegtran -copy none source.jpg > destination.jpg

# strip meta and convert to progressive coding
$ jpegtran -copy none -progressive source.jpg > destination.jpg

# keep all meta but still optimize
$ jpegtran -copy all source.jpg > destination.jpg

Important note on stripping meta

Only strip meta information from images you own the rights for and have permission. Otherwise you're committing a crime. Photographers put important copyright information in meta markers.

Optimizing GIF animations

Remember - no GIFs other than animations. For animations, run GIFsicle (pronounced "yo' mama" ) berfore you put them up:

# GIFSicle
$ gifsicle -O2 source.gif > destination.gif

More tools?

These are the core tools for image optimization. There's a number of wrapper tools that are more or less UIs on top of these, because there are people who don't like consoles (really?!). I'll list the few that I can think of, please comment if you know of others, especially for windows. It's nice to give nice UIs to give to designers so they can drag-drop optimize images too.

• smush.it, created by yours truly and Nicole Sullivan, now part of YSlow - runs pngcrush, jpegtran, gifsicle
• PageSpeed runs optipng, jpegtran
• PunyPNG - originally inspired by smush.it, but more advanced
• ImageOptim - Nice easy UI for Mac, runs most of the tools above (hope your company firewall doesn't block the site because of the domain name )
• PNGSquash another UI for Mac, runs advpng, pngcrush, optipng
• PNG Monster is for Windows, runs many PNG tools, you can drag/drop on it
• IrfanView - my favorite image viewer for Windows has a plugin to use PNGOut
• WP-Smushit is a WordPress plugin by Alex Dunae which sends all your image uploads to smush.it for optimization. Talk about easy!

More reading

Series of articles on YUIBlog:

• Choosing the right format
• 4 steps to size reduction
• Progressive JPEG

presentations:

• Image Optimization - 7 mistakes
• Image Weight Loss Clinic - a 5-minute "ignite" case study of optimizing a web site. Spoiler: 30.15% image size reduction. Transcript here.

and more:

• Derek Tonn's graphics optimization blog
• Sergey Chikuyonok's excellent articles on optimizing PNGs and optimizing JPEGs
• Installing PNG tools on the Mac
• The book "Even Faster Web Sites" has a chapter written by yours truly and Nicole Sullivan

Thanks!

Thank you for reading. Now you have a whole lot of tools/toys to install and play with. Image optimization is an easy way to improve performance, it's just running a bunch of tools. You don't need to worry that the quality will suffer (so the designer won't be disappointed in you ). So you can only win. You may win quite a bit, you may win just a little (anywhere between 5 to 30% savings is what I've seen on random live sites when I was working on and testing smush.it). The thing is you'll almost always win something.

And because it's human to forget to optimize the images before you push them live, do take the time to setup the optimization step as part of the automated deployment process.

And to summarize once again the steps of what this automated process would be:

1. Convert GIFs to PNG. Then relax and take a deep breath (instead of flaming the unfortunate soul who created the GIFs) while you string replace "gif" with "png" in all your styleshets
2. Run PNGs through optipng, pngcrush, pngout, any or all of the tools listed above
3. Run JPEGtran
4. Run GIFsicle

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 11 This post is part of the 2009 performance advent calendar experiment. Stay tuned for the next articles.

After removing all the extra HTTP requests you possibly can from your waterfall, it's time to make sure that those that are left are as small as they can be. Not only this makes your pages load faster, but it also helps you save on the bandwidth bill. Your weapons for fighting overweight component include: compression and minification of text-based files such as scripts and styles, recompression of some downloadable files, and zero-body components. (A follow-up post will talk about optimizing images.)

Gzipping plain text components

Hands down the easiest and at the same time quite effective optimization - turning on gzipping for all plain text components. It's almost a crime if you don't do it. Doesn't "cost" any development time, just a simple flip of a switch in Apache configuration. And the results could be surprisingly pleasant.

When Bill Scott joined Netflix, he noticed that gzip is not on. So they turned it on. And here's the result - the day they enabled it, the outbound traffic pretty much dropped in half (slides)

Gzip FAQ

How much improvement can you expect from gzip?

On average - 70% reduction of the file size!

Any drawbacks?

Well, there's a certain cost associated with the server compressing the response and the browser uncompressing it, but it's negligible compared to the benefits you get

Any browser quirks?

Sure, IE6, of course. But only in IE6 service pack 1 and fixed for after that. You can boldly ignore this edge case, but if you're extra paranoid you can disable gzip for this user agent

How to tell if it's on?

Run YSlow/PageSpeed and they'll will warn you if it's not on. If you don't have any of those tools just look at the HTTP headers with any other tool, e.g. Firebug, webpagetest.org. You should see the header:

Content-Encoding: gzip

provided, of course, that your browser claimed it supports compression by sending the header:

Accept-Encoding: gzip, deflate

What types of components should you gzip?

All text components:

• javascripts
• css
• plain text
• html, xml, including any other XML-based format such as SVG, also IE's .htc
• JSON responses from web service calls
• anything that's not a binary file...

You should also gzip @font-files like EOT, TTF, OTF, with the exception of WOFF. Average about 40% to be won there with font files.

How-to turn on gzipping

Ideally you need control over the Apache configuration. If not full control, at least most hosting providers will offer you ability to tweak configuration via .htaccess. If your host doesn't, well, change the host.

So just add this to .htaccess:

AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml application/javascript application/json

If you're on Apache before version 2 or your unfriendly host don't allow any access to configuration, not all is lost. You can make PHP do the gzipping for you. It's not ideal but the gzip benefits are so pronounced that it's worth the try. This article describes a number of different options for gzipping when dealing with uncooperative hosts.

Rezipping

As Billy Hoffman discovered, there's potential for file size reduction with common downloadable files, which are actually zip files in disguise. Such files include:

• Newer MS Office documents - DOCX, XLSX, PPTX
• Open Office documents - ODT, ODP, ODS
• JARs (Java Applets, anyone?)
• XPI Firefox extensions
• XAP - Silverlight applications

These ZIP files in disguise are usually not compressed with the maximum compression. If you allow such downloads from your website, consider recompressing them beforehand with maximum compression.

There could be anywhere from 1 to 30% size reduction to be won, definitely worth the try, especially since you can do it all on the command line, as part of the build process, etc. (re)Compress once, save bandwidth and offer faster downloads every time

15% uncompressed traffic

Tony Gentilcore of Google reported his findings that a significant chunk of their traffic is still sent uncompressed. Digging into it he realized there's a number of anti-virus software and firewalls that will mingle with the browser's Accept-Encoding header changing into the likes of:

Accept-Encoding: xxxx, deflxxx
Accept-Enxoding: gzip, deflate

Since this is an invalid header, the server will decide that the browser doesn't support gzip and send uncompressed response. And why would the retarded anti-virus program do it? Because it doesn't want to deal with decompression in order to examine the content. Probably not to slow down the experience? In doing so it actually hurts the user to a greater extend.

So compression is important, but unfortunately it's not always present. That's why minification helps - not only because compressing minified responses is even smaller, but because sometimes there is no compression despite your best efforts.

Minification

Minification means striping extra code from your programs that is not essential for execution. The code in question is comments, whitespace, etc from styles and scripts, but also renaming variables with shorter names, and various other optimizations.

This is best done with a tool, of course, and luckily there a number of tools to help.

Minifying JavaScript

Some of the tools to minify JavaScript include:

• YUICompressor
• Dojo ShrinkSafe
• Packer
• JSMin
• ... and the new kid on the block - Google's Closure compiler

How much size reduction can you expect from minification? To answer that I ran jQuery 1.3.2. through all the tools mentioned above (using hosted versions) and compared the sizes before/after and with/without gzipping the result of minification.

The table below lists the results. All the % figures are % of the original, so smaller is better. 29% means the file was reduced to 29% of its original version, or a saving of 71%

As you can see gzipping alone gives you about 70% savings, minification alone cuts script sizes with more than half and both combined (minifying then gzipping) can make your scripts 85% leaner. Verdict: do it. The concrete tool you use probably doesn't really matter all that much, pick anything you're comfortable with to run before deployment (or best, automatically during a build process)

Minifying CSS

In addition to the usual stripping of comments and whitespaces, more advanced CSS minification could include for example:

// before
#mhm {padding: 0px 0px 0px 0px;}
// after
#mhm{padding:0}

// before
#ha{background: #ff00ff;}
// after
#ha{background:#f0f}
//...

A CSS minifier is much less powerful than a JS minifier, it cannot rename properties or reorganize them, because the order matters and for example text-decoration:underline cannot get any shorter than that.

There's not a lot of CSS minifiers, but here's a few I tested:

• YUI compressor - yes, the same YUI compressor that does JavaScript minification. I've actually ported the CSS minification part of it to JavaScript (it's in Java otherwise) some time ago. There's even an online form you can paste into to test. The CSS minifier is regular expression based
• Minify is a PHP based JS/CSS minification utility started by Ryan Grove. The CSS minifier part is also with regular expressions, I have the feeling it's also based on YUICompressor, at least initially
• CSSTidy - a parser and an optimizer written in PHP, but also with C version for desktop executable. There's also a hosted version. It's probably the most advanced optimizer in the list, being a parser it has a deeper understanding of the structure of the styleshets
• HTML_CSS from PEAR - not exactly an optimizer but more of a general purpose library for creating and updating stylesheets server-side in PHP. It can be used as a minifier, by simply reading, then printing the parsed structure, which strips spaces and comments as a side effect.

Trying to get an average figure of the potential benefits, I ran these tools on all stylesheets from csszengarden.com, collected simply like:

<?php
$urlt = "http://csszengarden.com/%s/%s.css";
for ($i = 1; $i < 214; $i++) {
  $id = str_pad($i, 3, "0", STR_PAD_LEFT);
  $url = sprintf($urlt, $id, $id);
  file_put_contents("$id.css", file_get_contents($url));
}
?>

3 files gave a 404, so I ran the tools above on the rest 210 files. CSSTidy ran twice - once with its safest settings (which even keep comments in) and then with the most aggressive. The "safe" way to use CSSTidy is like so:

<?php
// dependencies, instance
include 'class.csstidy.php';
$css = new csstidy();

// options
$css->set_cfg('preserve_css',true);
$css->load_template('high_compression');

// parse
$css->parse($source_css_code);

// result
$min = $css->print->plain();
?>

The aggressive minification is the same only without setting the preserve_css option.

Running Minify is simple:

<?php
// dependencies, instance
require 'CSS.php';
$minifier = new Minify_CSS();

// minify in one shot
$min = $minifier->minify($source_css_string_or_url);

As for PEAR::HTML_CSS, since it's not a minifier, you only need to parse the input and print the output.

<?php
require 'HTML/CSS.php';

$options = array(
    'xhtml' => false,
    'tab' => 0,
    'oneline' => true,
    'groupsfirst' => false,
    'allowduplicates' => true,
);

$css = new HTML_CSS($options);
$css->parseFile($input_filename);
$css->toFile($output_filename);
// ... or alternatively if you want the result as a string
// $minified = $css->toString();

So I ran those tools on the CSSZenGarden 200+ files and the full table of results is here, below are just the averages:

Again, the numbers are percentage of the original, so smaller is better. As you can see, on average gzip alone gives you 70% size reduction. The minification is not so successful as with JavaScript. Here even the best tool cannot reach 40% reduction (for JS it was usually over 50%). But nevertheless, gzip+minification on average gives you a reduction of 80% or more. Verdict: do it!

An important note here is that in CSS we deal with a lot of hacks. Since the browsers have parsing issues (which is what hacks often exploit), what about a poor minifier? How safe are the minifiers? Well, that's a subject for a separate study, but I know I can at least trust the YUICompressor, after all it's used by hundreds of Yahoo! developers daily and probably thousands non-Yahoos around the world. PEAR's HTML_CSS library also looks pretty safe because it has a simple parser that seems to tolerate all kinds of hacks. CSSTidy also claims to tolerate a lot of hacks, but given that the last version is two years old (maybe new hacks have surfaced meanwhile) and the fact that it's the most intelligent optimizer (knows about values, colors and so on) it should be approached with care.

204

Let's wrap up this lengthy posting with an honorable mention of the 204 No Content response (blogged before). It's the world's smallest componet, the one that has no body and a Content-Length of 0.

Often people use 1x1 GIFs for logging and tracking purposes and other types of requests that don't need a response. If you do this, you can return a 204 status code and no response body, only headers. Look no further that Google search results with your HTTP sniffer ON to see examples of 204 responses.

The way to send a 204 response from PHP is simply:

header("HTTP/1.0 204 No Content");

A 204 response saves just a little bit but, hey, every little bit helps.

And remember the mantra: every extra bit is a disservice to the user

Thank you for reading!

Stay tuned for the next article continuing the topic of reducing the component sizes as much as possible.

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 10 This post is part of the 2009 performance advent calendar experiment. Stay tuned for the next articles.

Looking back at the life of Page 2.0. and all the opportunities for optimization, the posts I've put up so far as part of this advent calendar have been mostly around optimizing the waterfall - making it shorter by having fewer components in it. This post will be probably the last in this area of optimization. After that let's talk about having smaller components and then move away from the waterfall and talk about what happens next - rendering, user interaction, JavaScript...

Caching

Caching comes into the picture for repeat page visits. When the user comes to your page for the second time, wouldn't it be beautiful if all components are in the cache and the waterfall is really short?

Here are the two views - first and repeat - from visiting the same page.

first view	repeat view

Ideally the repeat views should be with full cache and everything should be loaded from the local disk. That's what many people still assume: "Who cares about CSS? It's in the cache. Images? How do you mean sprites? But it's all in the cache, right?" Wrong. There are surprisingly few visits with full cache that we're used to believe.

This research demonstrates that on an extremely popular site - yahoo.com - about 50% of the visits are empty cache visits and 20% of all page *views*.

You cannot control what the users do what their cache, but you can suggest that their browsers keep a copy of your files for as long as they can.

"Never expire" policy

The never expire policy is to set a far future Expires (or Cache-Control) header effectively saying to the browser - hey this copy is good for a long time, keep it and reuse it.

The way to set the expires header for your static components is to tweak the Apache configuration (in .htaccess if your host gives you no other options) for example like so:

ExpiresActive On
ExpiresByType application/x-javascript "access plus 10 years"
ExpiresByType text/css "access plus 10 years"
ExpiresByType image/png "access plus 10 years"

Now if you access a PNG on Dec 10th 2009, Apache will add this header to the HTTP response:

Expires: Sun, 10 Dec 2019 05:47:47 GMT

The browser should never request this file again till 2019, effectively "forever".

The drawback is, of course, that you cannot modify this file anymore, since some users have already cached it forever and ever till the end of 2019. If you need to change the file, you have to save it under a different name and update all references to it. For the new name some use consecutive numbers, some use timestamps, some even hashes of the content, so that the name reflects the actual body of the component.

Caching vs. inlining

External components have a shot at being cached. But on the other hand having inline components (script, styles) means having fewer HTTP requests. Which one is better?

It depends from one case to the next but generally inlining is better for first visits (for homepages) and external components are better for repeat visits. But homepages are so 1998, with the modern search engines people find deep content pages and don't browse from the homepage. There are exceptions, of course. Some pages are common user homepages (the first page when the browser loads).

Anyway - inline vs. external? How about both?

Inlining, then caching

During first visits you can inline some components, then once the page is loaded, lazy-load the same components but this time as external files (and proper far-future Expires date).

For repeat visits, you only link to the external components as they are already in the cache!

So how do you determine first visits? How can you tell that the user's cache is empty?

You can use cookies and assume missing cookie means empty cache. Drawbacks: 1. someone can delete cache without deleting cookies and 2. what happens when the external component updates

Both of these scenarios are not so bad, because the page will still work, only less optimally. And you can take care of 2. by introducing (complexity and) a component version information in the cookie.

Oh, and drawback 3. - writing longer term cookies increases the size of your headers.

Another option is to track sessions only and assume empty cache for all new sessions. This way you don't need longer-term (and unreliable, see drawback 1.) cookies. In this case the problem might be that you may be missing out on optimizing the first visit in the session. The user may have visited the site yesterday, closed the browser and there she comes today with the full bag of cached components. But you assume first visit = empty handed visit, so you don't benefit from the components already in the cache during the first page view.

Which case is the right for you? Only your logs and experimentation can tell.

For an example in the wild, check Bing's search results. The first visit in the session you get a number of inline stylesheets, which then get lazy-loaded as external files. Second search in the same session and you get the stylesheets as external files.

Thanks for reading!

And that concludes the "fewer HTTP requests" part of the calendar. Tomorrow's topic will be about making smaller those components that slip through the cracks of the HTTP reduction effort.

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 9 This post is part of the 2009 performance advent calendar experiment. Stay tuned for the next articles.

One of Yahoo!'s first batch of performance best practices has always been "Avoid duplicate scripts" (check Steve Souders' post). Later we added "... and styles". This is a pretty obvious, kind of a "Duh!" type of recommendation, it's like saying "Avoid sleep() in your server-side scripts". But it didn't come up out of thin air, duplicates were noticed on some quite high-profile sites.

Duplicates are easy to spot (and YSlow will warn you), but let's talk a bit about another concept - let's call it near-duplicates - when two components are similar, almost the same, but not quite.

Duplicate scripts and styles

As a refresher and a quick illustration of the effects of duplicate scripts and styles, fire off your HTTP sniffer and hit this test page.

(btw, this is a simple page I put up to test different YSlow scenarios, you can actually use it as a web service of sorts to create any type of components with different options)

Firefox 2 downloading both duplicate styles and scripts:

IE6 and duplicate scripts:

Exact details of when/which browsers chose to download duplicates are not that interesting, it's obviously bad to waste time downloading the same resource. Even if no repeating download happens, the browser still has to parse through and execute the script/style for a second time.

Even if you have iframes you don't need to repeat the same JS/CSS in each frame, you can "borrow" them from the parent page, here's an example.

Near-duplicates

Near duplicates can be:

• components with the exact same response bodies but different URLs causing the browser to do double work
• components (images) that are too close to each other - in terms of looks or purpose. Only one component should be selected in this case.

Same component, different URLs

This could happen especially when you have user-generated content such as image uploads for profile photos and avatars in social sites, forums, images people put in comments on MySpace and so on.

Also images of stuff for sale (Craigslist, eBay). Often different sellers offering the same item would take the same photo from the manufacturer's site and upload it over and over again.

Luckily, PageSpeed warns about components with identical content, so those can be identified:

In the screenshot above, you see one image (2.3K) repeated 3 times, another (the iPhone, 1.7K) is repeated 4 times, and yet another one (2.8K) repeated 2 times.

It's not exactly trivial to avoid this type of duplications with user-generated content (for example, the first poster may delete the photo, in which case the second poster's photo will need to "shine through"). But it's not impossible, using for example a hash of the component's content as an identifier.

Loading...

Ajax loading indicators are a great idea to give feedback to the user that something is happening. They come in all shapes and sizes... sometimes on the same page, unfortunately. And again, sometimes it's the same stock image but used at different stages of gradual "ajaxification" of the page and with different URLs.

As we're moving more and more towards modular pages and client-side logic, often different modules on the same page are coded by different teams at different times, independently, without being aware of each other's assets. This way of building pages has it's challenges and one is that common components, such as Ajax loading indicators, should be shared.

Too similar modules

Along the same lines - different modules are sometimes created by different designers at different times. The result - one rounded corner box with 1px shadow and one with 2px shadow, both on the same page. Or two different shades of the same gray color, which no one can tell apart. That's just a waste. (See Nicole Sullivan's presentation for illustration, e.g. slides 44, 45)

Below is an example, can you tell that these 5 rounded corner boxes are all different - slightly different shadow, color or radius? How many different boxes does this page need?

Different sizes of the same image

It's highly recommended to not scale images in HTML (or CSS). If you need a 100x100 image you don't use a 400x400 one with <img width="100" height="100" ... />. That's a good rule of thumb... to break sometimes

In cases where the same image is used with different sizes and likely even on the same page, it may be beneficial to reuse the same bigger image and scale it down, because this could be saving extra HTTP requests of downloading the same (but slightly smaller) image.

Facebook is an example, the same hairy guy on the screenshot has two images with different sizes. It's actually the same image but resized in CSS.

The relevant CSS which shows the profile image in LARGE and SMALL (and looks like there's even a TINY view, although I couldn't find an example on this page)

.UIProfileImage_LARGE{width:50px;height:50px}
.UIProfileImage_SMALL{width:32px;height:32px}
.UIProfileImage_TINY{width:25px;height:25px}

Thank you!

Thanks for reading! Reducing HTTP requests is critical for page performance. You've merged your scripts and styles as much as reasonable, you've crafted CSS sprites and inlined images with data URIs. Now it's time to look at what's left - are there components that are way too similar, are there any near-duplicates or even exact duplicates? Same image on different backgrounds? Ever-so-subtle gradients and shadows? Time to pick up the old axe and cut.

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 8 This article is part of the 2009 performance advent calendar experiment. This is also the first ever guest post to this blog.
Please welcome the world-famous Christian Heilmann! And stay tuned for the next articles.

Chris Heilmann is a self confessed data junkie and worked for over 12 years as a professional web developer. Having published several books on JavaScript, Accessibility and web development using web services he right now works as a developer evangelist for the Yahoo Developer Network. He blogs at http://wait-till-i.com/, has all his talks and videos at http://icant.co.uk/ and can be found on Twitter as @codepo8.

 

RSS is a wonderful format to get information from all kind of different sources. It is dead easy to provide, has a predictable (albeit limited) format and is very easy to use. The problem of course is that with the amount of different feeds used in one page its performance goes down.

The reason is the classic HTTP request issue - the more you negotiate, find and pull the slower your page renders. Therefore you need to try to shorten the time the calls happen.

Say you want to pull the following five RSS feeds and display them:

• http://code.flickr.com/blog/feed/rss/
• http://feeds.delicious.com/v2/rss/codepo8?count=15
• http://www.stevesouders.com/blog/feed/rss
• http://www.yqlblog.net/blog/feed/
• http://www.quirksmode.org/blog/index.xml

The least effective way of doing that is pulling and displaying them one after the other:

$oldtime = microtime(true);
$url = 'http://code.flickr.com/blog/feed/rss/';
$content[] = get($url);
$url = 'http://feeds.delicious.com/v2/rss/codepo8?count=15';
$content[] = get($url);
$url = 'http://www.stevesouders.com/blog/feed/rss';
$content[] = get($url);
$url = 'http://www.yqlblog.net/blog/feed/';
$content[] = get($url);
$url = 'http://www.quirksmode.org/blog/index.xml';
$content[] = get($url);
display($content);
echo '<p>Time spent: <strong>' . (microtime(true)-$oldtime) .'</strong></p>';
function get($url){
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $output = curl_exec($ch);
  curl_close($ch);
  return $output;
}
function display($data){
  foreach($data as $d){
    $obj = simplexml_load_string($d);
    echo '<div><h2><a href="'.$obj->channel->link.'">'.
          $obj->channel->title.'</a></h2>';
    echo '<ul>';
    foreach($obj->channel->item as $i){
      echo '<li><a href="'.$i->link.'">'.$i->title.'</a></li>';
    }
    echo '</ul></div>';
  }
}

Using Stoyan's Multi Curl function you already realise quite an increase in speed.

$data = array(
  'http://code.flickr.com/blog/feed/rss/',
  'http://feeds.delicious.com/v2/rss/codepo8?count=15',
  'http://www.stevesouders.com/blog/feed/rss',
  'http://www.yqlblog.net/blog/feed/',
  'http://www.quirksmode.org/blog/index.xml'

);
$r = multiRequest($data);
display($r);

function multiRequest($data, $options = array()) {
  // array of curl handles
  $curly = array();
  // data to be returned
  $result = array();
  // multi handle
  $mh = curl_multi_init();
  // loop through $data and create curl handles
  // then add them to the multi-handle
  foreach ($data as $id => $d) {
    $curly[$id] = curl_init();
    $url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
    curl_setopt($curly[$id], CURLOPT_URL,            $url);
    curl_setopt($curly[$id], CURLOPT_HEADER,         0);
    curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
    // post?
    if (is_array($d)) {
      if (!empty($d['post'])) {
        curl_setopt($curly[$id], CURLOPT_POST,       1);
        curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']);
      }
    }
    // extra options?
    if (!empty($options)) {
      curl_setopt_array($curly[$id], $options);
    }
    curl_multi_add_handle($mh, $curly[$id]);
  }
  // execute the handles
  $running = null;
  do {
    curl_multi_exec($mh, $running);
  } while($running > 0);
  // get content and remove handles
  foreach($curly as $id => $c) {
    $result[$id] = curl_multi_getcontent($c);
    curl_multi_remove_handle($mh, $c);
  }
  // all done
  curl_multi_close($mh);
  return $result;
}
function display($data){
  foreach($data as $d){
    $obj = simplexml_load_string($d);
    echo '<div><h2><a href="'.$obj->channel->link.'">'.
          $obj->channel->title.'</a></h2>';
    echo '<ul>';
    foreach($obj->channel->item as $i){
      echo '<li><a href="'.$i->link.'">'.$i->title.'</a></li>';
    }
    echo '</ul></div>';
  }
}

However, there are still two things that are annoying:

• You pull far more data than you really need
• You do all the request from your server

Yahoo Pipes has been used for that kind of task for quite a while, but the issue was that it is a visual interface and therefore hard to maintain. The server was also not the best performing out there.

The good news is that there is a new(er) kid on the block in Yahoo Land called YQL running on a massively fast server farm and with one purpose: making it easier to use web services, mix them and get only the data back that you want.

YQL in itself is a web service and you post queries to it that access other web services in a SQL style syntax. Normally you'd get RSS feeds using the RSS table, like so:

select * from rss where url="http://feeds.delicious.com/v2/rss/codepo8?count=15"

See the single RSS in the YQL console (you need a Yahoo account to log in). You can also see the single RSS retrieval output.

The issue with this is that it only retrieves the items of the RSS feed and not the title, which we need for the headings. Therefore we need to use the XML table:

select * from xml where url="http://feeds.delicious.com/v2/rss/codepo8?count=15"

See the single XML in the YQL console (you need a Yahoo account to log in). You can also see the single XML retrieval output.

This gives us the same data the normal cURL calls give us. The cool thing about YQL is though that you can filter the data you get back to the bare minimum. In our case, this means replacing the * with the title and the link of the feed and of the items:

select channel.title,channel.link,channel.item.title,channel.item.link 
  from xml 
  where url="http://feeds.delicious.com/v2/rss/codepo8?count=15"

See the filtered RSS in the YQL console (you need a Yahoo account to log in). You can also see the filtered RSS retrieval output.

In order to use this with all of our RSS feeds, we can use the in() command:

select channel.title,channel.link,channel.item.title,channel.item.link
    from xml where url in(
      'http://code.flickr.com/blog/feed/rss/',
      'http://feeds.delicious.com/v2/rss/codepo8?count=15',
      'http://www.stevesouders.com/blog/feed/rss',
      'http://www.yqlblog.net/blog/feed/',
      'http://www.quirksmode.org/blog/index.xml'
    )

Check the aggregation in the console or the aggregation output.

This leaves all the hard work to the Yahoo Server farm. YQL pulls all the RSS feeds, adds one after the other and then gives it back to us as XML. We could simply use the generated URL from the console, but it is much more versatile to assemble the query in PHP:

$data = array(
  'http://code.flickr.com/blog/feed/rss/',
  'http://feeds.delicious.com/v2/rss/codepo8?count=15',
  'http://www.stevesouders.com/blog/feed/rss',
  'http://www.yqlblog.net/blog/feed/',
  'http://www.quirksmode.org/blog/index.xml'
);
$url ='http://query.yahooapis.com/v1/public/yql?q=';
$query = "select channel.title,channel.link,channel.item.title,channel.item.link from xml where url in('".implode("','",$data)."')";
$url.=urlencode($query).'&format=xml';
$content = get($url);
display($content);
function get($url){
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $output = curl_exec($ch);
  curl_close($ch);
  return $output;
}
function display($data){
  $data = simplexml_load_string($data);
  $sets = $data->results->rss;
  $all = sizeof($sets);
  for($i=0;$i<$all;$i++){
    $r = $sets[$i];
    $title = $r->channel->title.'';
    if($title != $oldtitle){
      echo '<div><h2><a href="'.($r->channel->link.'').'">'.
           ($r->channel->title.'').'</a></h2><ul>';
    }
      echo '<li><a href="'.($r->channel->item->link.'').'">'.
           ($r->channel->item->title.'').'</a></li>';
    if($title != $sets[$i+1]->channel->title.''){
      echo '</ul></div>';
    }
    $oldtitle = $r->channel->title.'';
  };
}

As you can see the loop to display the different RSS feeds a bit clunky and we could use an open YQL table to move the whole conversion to a server-side JavaScript. However, as it is the performance of this way of retrieving the RSS feeds beats all the others hands-down already:

You can try it yourself, get the demo code from GitHub and run it on your own server to see the magic of YQL.

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 7 This is the seventh in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

UPDATE: While this post is an interesting study, the problem it solves turns out to be much simpler. The details are here. In resume: you need a closing separator and it all works fine in IE7/Vista/Win7

Let's start this post as a dialog:

- Data URIs are a way to embed the base64-encoded content of images inside HTML and CSS. Inlining images in markup or stylesheets helps you save precious HTTP requests. It's an alternative to CSS sprites.
- BUT! IE6 and IE7 don't support data URIs
- Yes, for them you can inline the images using MHTML
- BUT! Looks like IE7 on Vista and IE7 on Win7 have problems with MHTML
- [Sigh...] For those browser/OS combos there's a solution too.

I'll quickly go over what are data URIs and how MHTML addresses IE6 and IE7. If you're familiar with these, feel free to skip down to the Vista blues part.

Data URIs

Here's how to use data URIs in your pages:

1. take an image:
   
2. Use PHP to read this image's binary content and encode it using base64 encoding:

   $ php -r "echo base64_encode(file_get_contents('horoscopes.png'));"
   iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAMAAADXqc3KAAADAFBMVEX///8mPnru9...
   ...more scary stuff goes here...
   dajNGGzlDAAAAABJRU5ErkJggg==

3. To use this image in CSS, paste the encoded image content in the stylesheet following this syntax:

   .horoscopes {
       background-image: url("data:image/png;base64,iVBOR...rkJggg==");
   }

   Note: the trailing == is part of the image content it's not part of the syntax

4. Alternatively if you want to use this image in the HTML, you can go like:

   <img src="data:image/png;base64,iVBOR...rkJggg==" />

And that's about it for data URIs, it's quite simple actually.

Data URIs in the wild

To see real-life, high-traffic sites using data URIs look no further than your favorite search engines.

Yahoo! Search using data URI in CSS for a button background:

Google Search using data URI in an IMG tag for a video thumb:

Base64-encoded filesizes

The base64 encoding adds about 33% to the filesize. But, then you gzip the result, the compression brings the size back to the original. Plus or minus. Interestingly enough, sometimes after base64 and gzip, the result is smaller than the original. But don't count on that.

Theoretically also when you base64 encode a bunch of images and inline them in the same CSS or HTML, you'll have a larger string to compress, so increasing the chance of repetitions and compressing better. (Todo: test this assumption)

In any event, the benefit of reducing HTTP requests will likely greatly outweigh any fluctuation in the file size.

MHTML

IE8 supports data URIs, but earlier IEs do not. For them you can use MHTML (Multipart HTML, blogged previously here)

Continuing with the previous example, in order to display the horoscopes image in IE < 8 you can use a stylesheet like this:

/*
Content-Type: multipart/related; boundary="_MY_BOUNDARY_SEPARATOR"

--_MY_BOUNDARY_SEPARATOR
Content-Location:horoscopes
Content-Transfer-Encoding:base64

iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAMAAADXqc3KAAADAFBMVEX///8mPnru9....U5ErkJggg==
*/

.horoscopes{*background-image:url(mhtml:http://example.org/styles.css!horoscopes);}

A few notes on this example:

• the image content is base64-encoded just like before
• if you need more images, use your chosen separator prepended with --, in this case --_MY_BOUNDARY_SEPARATOR
• Content-Location:horoscopes is how you give a name to the image in order to use it later
• then later in your stylesheet you can reference the name in the stylesheet using http://url/styles.css!horoscopes
• you need absolute URLs in the mhtml:http://.... part (that's retarded, hope I'm mistaken. Didn't work for me with relative URLs)
• you can use a single CSS file to support both old IE as well as new browsers, but you'll have to repeat the image stream twice, probably a better approach is to use browser specific stylesheets

For more examples of MHTML (which is also used for multipart email messages) check this and for a brilliant example of using one MHTML to embed images in HTML (not CSS), check Hedger's test page here (appears 404 at the moment of writing). For a tool to automate the dataURI-zation/MHTML-ization, be sure to check Nicholas Zakas' CSSEmbed

The problem with IE7 on Vista (and Win7)

So far we have data URIs and MHTML fallback. Turns out we're not done yet, because IE7 on Vista and Windows 7 fails with the MHTML example. Two points:

• IE8 is the default in Windows 7, but it comes with several IE7 modes (either by default or turned on by users when their favorite pages break). Compatibility view, ie7 mode, ie8 in ie7 mode, blah-blah, it's all pretty confusing, but all modes with the exception of IE8 in proper IE8 standards don't work with the MHTML
• I tested Windows 7 evaluation copy, but I have the bad, bad feeling that Windows 7 normal copy will be as broken when it comes to MHTML. I believe it has something to do with security, if anyone finds an article on MSDN, or anywhere, please share.

The solution to the Vista problem is to split the CSS shown above (the one that uses MHTML) in two: one which contains the encoded images (the commented part) and a second one which only has the normal CSS selectors part.

In other words have something like:

1. mhtml.txt:

   Content-Type: multipart/related; boundary="_MY_BOUNDARY_SEPARATOR"
   
   --_MY_BOUNDARY_SEPARATOR
   Content-Location:horoscopes
   Content-Transfer-Encoding:base64
   
   iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAMAAADXqc3KAAADAFBMVEX///8mPnru9....U5ErkJggg==
   

   No need to use comments. Could be .txt, .css or anything you like. In my tests the content-type didn't matter - text/html, text/css, text/plain all worked

2. styles.css:

   .horoscopes{*background-image:url(mhtml:http://example.org/mhtml.txt!horoscopes);}

   The normal CSS simply references the new MHTML file appending !identifier

3. in your HTML you don't need to reference the MHTML document, you just point your link tag to styles.css as usual and styles.css contains the reference to the MHTML doc

Time to celebrate? Almost.

Another ugly bug surfaces - this procedure outlined above works only once. Once the MHTML.txt is cached, it stops working. That means repeating visits to the page or other pages using the same mhtml. The effect also means hovering on the same page. Say you have two images - one normal and one mouse over, both in the same MHTML. The normal works and the hover breaks, mouseout breaks too. Insane, isn't it?

There's a solution though - you should make the browser request the MHTML document every time. This is kind of backwards when it comes to performance optimization, where we want to cache everything. It's pretty unfortunate. The best you can do is avoid sending the MHTML every time, but only send a "not-modified" header. In order for this to work, the browser has to send If-Modified-Since or If-None-Match.

So you can send your MHTML file with an ETag (yes, ETags can help sometimes ) Then the browser will send an If-None-Match and you can happily reply "304 Not Modified".

Sounds complicated? It sure is. And, remember, all this is only for IE7 (or any IE7 mode) on Vista and Windows 7. All other IEs on all other platforms work with the easy MHTML and IE8 works with data URIs.

Everything is a trade-off (see last night's post) and I can understand how you may think "That is one big tradeoff". There's always the option of serving normal images to the affected browser/os and just don't implement this optimization for them.

But it's good to know there is a solution.

DataSprites class

Let me offer this DataSprites PHP class I coded that takes care of all the scenarios. It takes a bunch of image filenames as input and produces:

• a CSS containing data URIs for normal browsers (example output)
• an MTHML CSS fallback for IE6,7 (example output)
• a CSS that refers to an additional MHTML for IE7 on Vista, Win7 (example output - css and mhtml). The MHTML in this case is sent out with an ETag (derived from the input image filenames) and consecutive requests with the same ETag return 304 Not Modified

You can see a test page here and view the source code here. The directory listing is also available if you want to grab the images for testing.

The perfect use case for such an approach is when you want to combine background images on the fly. When you would normally use CSS sprites, but you want to produce them dynamically at run time (maybe because you have too many combinations?).

I've tested this in Safari, Firefox, Opera, IE6, IE7 and IE8 (in all compat modes) on Vista and Windows 7 evaluation copy. If you find the test page is not working for you, please let me know.

In this DataSprites class, I'm checking the problem OS looking for the strings "Windows NT 6" and "Windows NT 7" in the user agent string. My Win7 evaluation copy had "Windows NT 6.1" in the UA, so I may be a little forwards-aggressive with the check, but somehow I thought Win7 proper should have "Windows NT 7" in the UA string. And also that Win7 will suffer from the same issue as Vista and Windows 7 evaluation.

UPDATE: Added a hover example.

UPDATE 2: Recorded a screencast to demo the IE7/Vista experience

Thanks!

Thanks for reading! Now you know all about data URIs, MHTML, and maybe a little too much about IE7/Vista's challenges Ready to use data URIs and save some HTTP requests?

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 6 This is the sixth in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

Last night I talked about the benefits of reducing the number of page components and the resulting elimination of HTTP overhead. A little carried away into making the case (because, surprisingly, I still meet people that are not convinced about the benefits) I didn't talk about some of the drawbacks and pain points associated with reducing the number of components. Kyle Simpson posted a comment reminding that it's not all roses (also see his blog post).

On drawbacks

A lot of the activities related to performance optimization have drawbacks associated with them. That's the nature of the game. Rarely an improvement is a clear win. Usually you need to sacrifice something - be it development time, maintainability, browser support, standards compliance (reluctantly), accessibility (never!). It's up to the team to decide where to draw the line. My personal favorite type of optimizations are those that (for example image optimization) are taken care of automatically, by a build process, offline (as opposed to runtime), at the stage right before deployment. But this also has drawbacks - you need to setup the build process to begin with, a build process that doesn't introduce errors, doesn't require more testing, is easy and friendly to use, doesn't take forever and so on.

So, drawbacks.

Here's how Jerome K. Jerome jokes about trade-offs in his Three Men in a Boat:

"...everything has its drawbacks, as the man said when his mother-in-law died, and they came upon him for the funeral expenses"

It's never easy, is it?

Combining creates atomic components

Say you merge three JavaScript files - 1.js, 2.js and 3.js to create all.js. A few days later you change 2.js. This invalidates all the cached versions of all.js that existing users have in their browser cache. You need to create a new bundle, say all20091206.js. Then a few more days pass and you fix a bug in 3.js. Again, you need a new bundle. Than the library (1.js) comes up with a new verison and you upgrade - yet another bundle.

While first-time visitors benefit from the fewer requests, repeating visitors don't take much advantage of the cache with such frequent updates. In optimization we also say - set the expiration date in far future, but what's the point if you're going to change the component every other day.

One way to deal with this is organizational - set up a release schedule. You don't push that new feature today when you know that the library update is coming up tomorrow. Queue the changes and roll them out in batches (could be pre-defined intervals). This is important in larger organizations where more than one team may touch parts of the bundled components and push out the whole uber-component.

Another way is to identify which files change relatively often and put the movable parts in a separate bundle. This way you end up with one core, slow moving component and one that changes quickly. When you change it, you at least don't invalidate the core.

But in order to make such a call you need to measure and analyze. And answer the two questions:

1. How many repeat visits do I have? Forums and online communities are more sticky for example. Other sites may be surprised how few visitors come with full cache.
2. Which files to put in the changing bundle? Sure not all change at the same rate. Also the rate of changes also changes. e.g. a new feature matures and has less bugs, therefore requires less updates. When do I move a piece of new code into the slow-to-change core?

Facebook have gone a pretty scientific in this regard. David Wei And Changhao Jiang presented their system for monitoring the usage of the site and adaptively bundling features together. They show how they have (slide 26) total of 6 megs JS and 2 megs CSS. Merging all of that into one atomic component is unthinkable, so more intelligent methods are a must.

So the third way is to follow common user paths across the pages visited in a session and bundle together the files that the user is likely to need in his/her pattern of site navigation.

Yet another way to combine stuff is to have pre- and postload bundles. One has the bare minimum to get the user interacting with the page, the other has more of the bells and whistles.

In summary, the atomic components create challenges of their own. That shouldn't be used as an excuse though. As you see, there are ways.

Atomic bundles and personal projects

I doubt that the options above are really ever going to work for a personal project or a blog though. We're lazy people and since the site ain't broken... why fix it?

A personal project usually gets us excited initially and then the interest fades away for an extended period of time. If you don't change something now while you're into the project, then changes happen less often as the time passes. (You also forget how the code works and are reluctant to touch/break it)

A good strategy to optimize a personal project is to come back to it a week of two after a major update. Things are still fresh in your head, but if you haven't changed anything in two weeks, you probably won't touch it for another year. So 'tis the time for a one-off little project - turn on gzipping, minify CSS and JS (which otherwise you changed much too often a week ago), combine components.

Post-launch one-off optimization sprints don't work in bigger organizations and projects, but they can be a "cost"-effective way to optimize a small, personal site.

Merging components is an extra step

Scenario 1: why merge now when in the p.m. I'll work that thing again?
Scenario 2: I have to move to another task in the p.m. and I barely have the time to fix this bug before lunch, let alone to combine components and stuff.

Merging components is boring, there's nothing particularly challenging to it. How hard could it be?

$ cat 1.js 2.js 3.js > all.js

And it's an extra step.

Any extra work you introduce, any extra step has a great chance of being skipped. That gradually forgotten.

That's why there are two options for the optimizations:

1. Make them easy to do. Otherwise no one's going to do it.
2. Make them harder to not do it

What I mean is for example take the few minutes to create a script for this new project or feature. The script minifies, combines and pushes to the server. All that with one command line. And especially if you do it several times a day, you can keep one console open and just hit ARROW UP + ENTER to repeat the command after you're done with a change. Two keyboard strokes sure beats opening up an ftp program, logging in, navigating your HDD, navigating the server... tens of clicks easily.

To summarize: it's best not to underestimate the laziness, we shouldn't ignore the fact that we're lazy, but embrace it. Make the right thing easy and the wrong thing hard.

Run-time merging

It's best to keep the static components static. Static is simple, static is fast. But sometimes in the interest of merging, minification and so on in a very dynamic (or technically lacking) environment, you may resort to dynamic file bundling. Sometimes also the little pieces can be merged in so many combinations that dynamic components make sense.

Take for example the combo handler script used by YUI. You have a combo script at:

http://yui.yahooapis.com/combo

which simply takes a list of files to merge, finds them on the disk and spits them out with the correct Expires header.

For example here's a URL produced by the dependency configurator:

http://yui.yahooapis.com/combo?3.0.0/build/yui/yui-min.js&3.0.0/build/oop/oop-min.js&3.0.0/build/event-custom/event-custom-min.js&3.0.0/build/attribute/attribute-base-min.js&3.0.0/build/base/base-base-min.js

The little pieces that make up the YUI library can be combined in so many ways that pre-merging all possible bundles is not an option.

As part of YUI3, there was a PHP loader recently released which you can use to host your own combo solution - doesn't have to do anything with YUI. Or you can roll your own combo script, how hard could be to have a script that merges a number of files. Check Ed Eliot's script for inspiration/borrowing.

SmartOptimizer is another open source project that you can use for comboing, minification and so on.

If you do create your own combo script, don't forget to store the merged results in disk cache directory, so you don't have to re-merge the same bundles over and over.

CSS sprites are a pain

While combining text source files like CSS and JS is fairly trivial, creating sprites is a pain. Not only you have to create the image, but them get the coordinates and write the CSS rules. Then when an element in the sprite changes, it may affect the others and you may have to move them around, then update the CSS. Luckily there are tools.

Here are some tools and resources that should help with the sprites. CSS sprites are an excellent way to reduce HTTP components, especially when it comes to tiny icons, those that are sometimes smaller than the HTTP request/response headers needed to download them. It's a technique that is still underused.

• Intro to CSS sprites by Chris Coyier, includes a how-to for the SpriteMe tool
• SpriteMe - bookmarklet tool by Steve Souders
• Cool runnings - the sprite generation service behind spriteme.org
• SpriteGen - sprite generation tool
• SmartSprites is an interesting project - a downloadable tool that looks for specific instructions you leave in your normal CSS file
• Spritr - another online tool

And now, re-introducing CSSSprites.com

CSSSprites.com

CSSSprites.com is a fun weekend project I did two and a half years ago. Back then I was even more convinced that this technique is wildly underused, still considered unstable by many people, although it was fearlessly in use by prominent sites such as yahoo.com. There was also a bit of lack of understanding how to build sprites and also, just like today, it's just a pain to create them.

So I wanted to raise awareness of the technique by creating an online tool for sprites generation, probably the first tool (which explains why I was able to get such a domain name). The idea was just to let people know that this technique exists, it's not rocket science and you can start quickly by using the tool. The tool was actually quite ugly and I posted a comment on css-tricks.com asking if anyone wants to contribute a skin.

CSSSprites.com: BEFORE

CSSSprites.com: AFTER

Luckily, Chris Coyier who runs css-tricks.com responded and sent me a skin. It was exactly 2 years minus 2 days ago.

(I remember January 2nd 2008 right after new year's, I was writing a list of things I want to accomplish in 2008. It was a list of unfinished projects actually. I decided I was sick of not finishing projects so 2008 would be the year of completion. I wasn't going to start anything new until I finish all abandoned projects. Needless to say, as all new year resolutions, it didn't work. I actually started something pretty cool and at the same time never finished csssprites.com with Chris' skin)

So today, after two years of procrastination, I sat down and updated the site. Additional kick in the behind was that few days ago I had to take the site down, because the exceptionally reliable host site5.com in their care for the well being of the sites on the shared hosting, shut down all my sites, including this blog, because csssprites.com was consuming too much resources. I had to take it down. Today it's proudly back up hosted by dreamhost, the all-hero host that survived the smush.it explosion.

What's new in the tool?

• nicer skin!
• the old "skin" is sill available
• some options - padding between sprite elements, background color, border size, choice of top vs. left alignment of the elements
• no more gifs are being generated
• the generated PNG is optimized with pngout and pngcrush
• upgrade to YUI3

So give it a spin and comment with any problems/wishes and so on. I haven't posted the code yet, it's not in a presentable statge, plus the "core" of it, the generation of the sprite image, is already out there.

That's all folks

When I talked about optimizing personal project above, I had in mind sites like csssprites.com. Currently it's not optimized at all - it's a sprites tool that doesn't use sprites. But that's because I only worked one evening on it and it's probably broken here and there. I intend to revise it in a few weeks when it stabilizes.

So, parting words - combining components into atomic dowanloads is not without drawbacks, just like most other performance optimizations. But this doesn't mean we shouldn't do it. Care and analysing users' behavior is a way to optimize the process of optimizing but meanwhile just go with your feeling and look at the metrics to see if you were right. Rinse, repeat. When in doubt - err on the side of less HTTP requests.

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 5 This is the fifth in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

Let's talk a but about waterfall optimization - the first thing that happens in Mr.Page's life. The best way to optimize and speed up the waterfall is to have less stuff in it. The fewer page components, the faster the page - simple as that.

Fewer components vs. component weight

The size of the page components, meaning their size in kB, is important. It makes sense - smaller pages will load faster, 100K JavaScript will load faster than 150K. It's important to keep sizes low, but it should be clear that the number of components is even more important than their filesize.

Why? Because every HTTP request has overhead.

OK, but how bad can it be, someone might ask. If you look at an HTTP request - it has a header and a body. A 100K body will greatly outweigh the size of the headers, no matter how bloated they are.

Here's the headers for a request to Yahoo! Search:

Host: search.yahoo.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5;) Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

To that request the server responds with the body (content) of the response prepended with some headers like:

HTTP/1.1 200 OK
Date: Sat, 05 Dec 2009 07:36:25 GMT
P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR... blah, blah"
Set-Cookie: sSN=nTMt3Lo2...crazy stuff...nLvwVxUU; path=/;domain=.search.yahoo.com
Cache-Control: private
Connection: close
Content-Type: text/html; charset=ISO-8859-1

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">

<html lang="en"><head><meta...

This is 352 bytes request header and 495 bytes response header. Not so bad, eh? No matter how hard you try to make your cookies monster-size, the response body (9k gzipped in this case) will always be significantly bigger. So what's the problem with the overhead of the HTTP requests then?

The size of the headers is a problem when you make requests for small components - say requests for small icons - for example 1K or under. In this case you exchange 1K headers to get 1K of useful data to present to the user. Clearly a waste. Additionally this 1K of headers can grow once you start writing more cookies. It may very well happen that the HTTP headers size is bigger than the actual icon you need. And even if the headers are not bigger that the component, they are still big when you think percent-wise. 1K of 10K is 10%.

But the HTTP headers size is only one (and the smaller) of the problems.

The bigger problem is the HTTP connection overhead.

HTTP connection overhead

What happens (at a high level) when you type a URL and hit Enter? The browser sends a request to the server. Which server? The browser needs to know the IP address of the server, so if it doesn't have it in the cache, it makes a DNS lookup. Than the browser establishes a connection to the server. Then it waits for the first byte of the response from the server. Then it receives the full response (payload).

Here's how this looks like graphically represented by webpagetest.org

And the color legend:

1. DNS lookup
2. Initial connection
3. TTFB (Time to first byte)
4. Payload

So what do we have here - looks like in this particular case the browser is downloading content about 40% of the time. The rest of the time it's... well, not downloading content. How's that for an overhead. And the part of not downloading can be even greater, this above was just one example.

Now how about this - a bird-eye view of the 4 last pages in webpagetest.org's test history - just some random pages people have tested.

Do you see a lot of blue (time spent downloading content). Not as much as you would've hoped. There's some DNS lookups, some orange... and OMG, talk about going green!

In fact you may notice that the smaller the component, the smaller is the blue part.

What does all this tell us?

1. A significant chunk of time is spent in activities other than downloading.
2. Smaller components still incur HTTP overhead and for them the relative penalty (relative to their size) is atrocious.

So what's a performance optimizer to do? Reduce the number of components and so pay fewer penalties.

Just remove stuff

Truth is - a lot of stuff on the pages today is not needed. Features no one likes or uses clutter the page and make it heavier. Well, what can you do, the boss/client/marketing guy wants that feature there. What you can do is at least try. You can introduce some science into the marketing activities - measure how much a specific feature is used. Or if you already have the data - look at it. Decide what can a page go without.

It's going to be tough convincing people to remove stuff. After all you spend time developing it. Someone dreamt up that feature initially. Someone (not the users) loves it. People hate to let go. But still, it's worth to try.

Combine components

Now that the phase of convincing people to remove stuff is done, what's left needs to be combined. How do you combine components? Simple - all JavaScripts go into a single file, all CSS into a single file. All decoration images go into a sprite.

JavaScript example (from a page to remain anonymous)

Before:

<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js"></script>
<script src="/javascripts/application.js?1258423604"></script>
<script src="/javascripts/ui/minified/jquery.ui.all.min.js?1258423604"></script>
<script src="/javascripts/ui-ext/ui.bgiframe.min.js?1258423604"></script>
<script src="/javascripts/ui-ext/ui.stars.pack.js?1258423604"></script>
<script src="/javascripts/ui-ext/ui.dimensions.js?1258423604"></script>
<script src="/javascripts/ext/jquery.form.min.js?1258423604"></script>

After:

<script src="/javascripts/all.js"></script>

Sizes, gzipped: 70029 bytes before, 65194 bytes after. Simply merging files and there's even 6.9% saving!
And the more important saving: 6 less HTTP requests

Repeat for CSS. Before:

/stylesheets/general.css?1258423604
/stylesheets/global.css
/stylesheets/ui.stars.css
/stylesheets/themes/enation/enation.all.css
/public/template/css/132/1245869225

After:

<link type="text/css" rel="stylesheet" href="/stylesheets/all.css" />

Sizes, gzipped: before 14781 bytes, after 13352 bytes, saving 9.6%.
But the bigger saving: 4 less HTTP requests.

If you wonder how come the sizes before and after are different since we merely concatenate the contents of the files, well, the savings come from gzip compression. When you have more characters in the file, there's more chance that some will repeat which means they will compress better. That's one. And then, the compression itself has an overhead which you incur once for the whole bundle of files, as opposed to for every file.

Now - let's make decoration images into sprites. Before:

... 15 image requests, 6.8K

After: (1 sprited image)

Result size: 1.4K, 7 times smaller!

Here the savings are so dramatic partially because the source files are GIFs and the result is a PNG8 but that's a whole other post.

So in conclusion: file concatenation is just awesome. You save both: bytes to download and, much more importantly, HTTP requests. Less of the green stuff in the waterfall!

x-type component concatenation

So far we combined .js with .js, css with css and images with images. How about a cross-component type concatenation?

You can inline images inside HTML and CSS (and why not JS if you want) using data URIs (another post coming).

And you can inline CSS and JS inside HTML too.

This means you can have your whole application inside of just one HTML file if you wish. Inside of the HTML you have inline styles, scripts and images.

Combining CSS with JS

Now how about mixing CSS and JS into one component. You can do that, and it's especially suitable for lazy-loaded, widget-type functionality.

Say you've loaded the page, then the user clicks some rarely used button. You haven't downloaded that content that is supposed to wow the user upon click of the button. So you send a request to grab it. The new content may come in the form of a JSON string. And what if the new content requires some stylesheet that was not part of the base page? You'll have to make another request to download that stylesheet too.

Or, you can download both content and styles in the same JSON response. You simply inline the style information as a string in the JSON. So:

1. uses clicks, you request feature.js which goes like:

{"content":"<p class=\"wow\">I'm a feature</p>", "style": "wow{font-size: 60px}"}

2. You process the JSON and shove the content into the page

var o = JSON.parse(xhr.responseText);
$('result').innerHTML = o.content;

3. You add the styles to the head:

var wow = document.createElement('style');
wow.type = "text/css";
if (wow.textContent) { // FF, Safari
    wow.textContent = o.style;
} else {
    wow.styleSheet.cssText = o.style; // FF, IE
}
document.documentElement.firstChild.appendChild(wow);

Nice and simple. Makes the features (that progressively enhance the page) atomic and self-contained.

More to reducing components?

For more and creative ways to reduce HTTP components you can take a look at MXHR and Comet

Another thing to check is the Keep-Alive setting on your server. Remember how there were 4 steps in the component download. When you request a second component you can have the connection open so that you don't need to re-establish it (skipping step 2). And since the DNS lookup was already made you get rid of step 1. Skipping 2 out of 4 is not bad at all.

Summary

Reducing the number of page of components is the top priority of any web performance optimization effort. HTTP requests are costly. In part because of the headers size overhead, but mostly because of the connection overhead. The browser spends a disturbing amount of time not downloading stuff, and we can't allow this!

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 4 This is the fourth in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

Measuring time is an important activity in your performance efforts. After all, how else would you know if gzipping your javascripts helped or it didn't matter, or it slowed you down (because your server is now busier)? You have to measure and monitor what you measure in order to quantify whether or not you're making progress. With the exception of your own personal sites you have to measure time in order to show your boss or client that you are, in fact, working on something that matters. Time is important but the milliseconds don't tell the whole story.

Remember folks, we're dealing with humans here. The users of our sites are mainly humans (with the exception of a few (ro)bots, spiders and worms) and humans are irrational, some would say predictably so. Not only are we irrational, but we also have a very skewed idea of reality, we're easy to manipulate and our senses are not nearly as accurate as we believe them to be.

Mind hacks

"Mind hacks" and "Your brain: the missing manual" are excellent reads that show how unreliable our brains and our senses are. There's no other way actually, the reality is extremely complex and there's so much information to absorb and process and make sense of at any given time. Doing everything properly is just not possible. We need to take shortcuts in information processing, make guesses and sometimes just make up stuff so that out mental idea or the outside world is complete and consistent.

Magicians for example use our attention and processing flaws to do "impossible" things. Optical illusions work the same way. In the image above both A and B squares are the same color. Have fun figuring it out (short of using a color picker). It is the same rgb (100, 100, 100) or #646464 but you literally have to hide everything else on that image in order to see it. And even then, when you look at the image again your mind will still refuse to believe that it's the same color. Not only we're easily swayed, but we're also reluctant to accept a fact even after it has been proven to us.

The problem with that image is that we perceive things relatively, compare new information with previous knowledge and make a decision based on that. We're also suckers for patterns.

That's just the way we work - imagine you wake up tomorrow and before getting out of bed you decide to consider all the options ahead of you, their pros and cons. The number of choices is so daunting, you'll never do anything, you'll just lay there and calculate the risks and uncertainties associated with your every move. Because, you know, getting out of bed has its drawbacks (and is, generally, wildly overrated )

Time

Back to the time - the unit of measure that helps us judge how fast a page is. Needless to say, our idea of time is also inaccurate. And relative. Think about it - time flies when you're doing something you love and crawls when you're on the line at the cache register (although smart folks have decided to put celebrity magazines there to keep you distracted from the thought how much you hate waiting).

Time is relative to our enjoyment of what we do.

Turns out time also flows differently depending on the age - perceived 3 minutes for a 20 year old are in reality 3:03 and for a 60 year old 3 minutes are in reality 3:40.

Time is also relative to our expectations.

Maister's First Law of Service (service as in fast food service) states that:

Service = Perception – Expectation

When people perceive that you exceed expectations, they are happy and everything is fast and pleasurable. So you have to care about how users perceive your page load time and also what their expectations were. Naturally, both of these are subjective to begin with.

Competition

When it comes to expectations you should consider what type of site you have and how does the competition's sites compare. This will give you an idea of what experience the users expect. If you're in the social networking business, no one expects you to be search engine-fast. People come to comment to their friends' photos, to write messages and so on. They can wait a second more when they'll be spending minutes crafting their comment, they are probably thinking about the comment already and don't necessarily pay attention to the load time. The search engine experience is different - users are hunting for information, they want to move away from your page ASAP and get to their answer.

Painful = slow

Back to relativity of time. You've probably noticed that when you're in pain time goes slower, when you're frustrated with a site that is in your way of finishing a task, you find it pretty slow. When your bank site prompts you with half-baked extra security features and prevents you from paying that late bill, you don't care that time-to-onload of that page was under a second. You still hate the experience and think the site was slow (paying the bill actually took you longer than you imagined it should take).

On the opposite, when you're happy and having fun, you can tolerate a slower site and it will actually feel faster than the miserable experience with the objectively faster bank site.

People also expect certain types of pages to behave a certain way. If it's a light page with one-two graphics, it better be fast. People will perceive it as slow if it doesn't meet their expectations. On the other hand, a YouTube page is ok to be slower, you're there to watch funny cats (and enjoy yourself) and everybody has learned through experience that web video is normally slower than text-only pages.

Another illustration of pain and frustration being equal to slow. In WWII people who were likely to be captured and tortured for information were trained to deal with pain by counting. Because pain makes time run slow, the prisoner feels the torture go on forever. When you count, you have a more realistic idea of time and that helps.

Speed doesn't matter

Of course it does, just wanted to make sure you're paying attention

There's a fascinating study (and podcast) that claim "the truth about download time". The findings were that when people complete their task they perceive the site as fast, although it may be slower than another site that frustrated them.

This study was done in 2001 and things have changed since then. I mean they talk about 36 seconds page load times for Amazon. I found it particularly amusing how they measured the real page load time - interns with stop watches looking at the videos of the usability study and checking when the Netscape logo stops spinning (actually was it IE's logo that was spinning and NS had some shooting stars?)

These days we have data showing that sub-second delays make you lose visitors (sometimes forever) and it's clear that when given a faster and slower version of the same site, people will prefer the faster, naturally. This study was comparing sites against each other and looks like with different tasks.

But still, the findings that task completion determines you speed perception is fascinating and something to keep in mind when designing user interactions. If you bog down the user with pages and pages of lengthy forms with insanely strict and annoying validation, then no amount of super fast page loading will cause the people judge your site fast.

Flow

Flow is a concept that aims to answer the question what makes us tick, motivates us and makes us happy. In the words of the flow researcher Mihály Csíkszentmihályi flow is:

Being completely involved in an activity for its own sake. The ego falls away. Time flies. Every action, movement, and thought follows inevitably from the previous one, like playing jazz. Your whole being is involved, and you're using your skills to the utmost.

Web page performance can play critical role in stimulating the sense of flow. If a page takes too long, you leave the flow, you start thinking about things other than what you do, the brain wanders and the experience is ruined. On the other hand - clear goals, immediate feedback, balance between skills and challenge help create and maintain the state of flow.

It's an interesting topic, you can read up a bit on how flow relates to web sites here (see the pdf) and here.

Blink of an eye

"In a blink of an eye" is fast, right? As fast as it gets. Turns out a blink of an eye lasts on average 300 to 400 miliseconds. Can you load your page in a blink of an eye? Probably pretty difficult, but what you can do is provide some feedback in this time window. This way help the user maintain flow and get an idea something is going on. That's why progressive rendering is really important (subject for a separate post) - to let the user know that there was no error and the page is coming. Feedback, progressive rendering and progress indicators of any kind communicate and reassure.

Talking about milliseconds here's another number to keep in mind - 200ms. Our eyes move in a succession about 5 times a second or every 200ms, taking snapshots of the outside world. But these 200ms are relative too. When something appears or moves we're likely to focus on it and stay focused (who knows maybe there's a danger in there ) for more than 200ms before the eye moves to examine the rest of the world. If that thing dissapears, we're likely to move our eyes faster. That's why if you want to highlight something on the screen, say you showed object A and then B, B is more likely to be noticed if you first hide A.

Transitions

Things in real world don't just pop and disappear. It's actually pretty scary when someone we didn't notice all of a sudden seem to appear from nowhere. In real world objects move from one state to another. You can simulate that on the screen with transitions and animations. Otherwise people are taken by surprise a little bit and are feeling not in control which may hurt the state of flow. Often if you animate something on the screen, it will feel more natural even if it takes longer. But not too long, of course, not to make it look unnatural (As a friend said for my first YUI animation - it felt painful and made him tense waiting for it to finish, because I was so proud I can animate stuff on the page that I made the animation take its time). Using easing animations is also preferable.

White is fast

Google Reader used to have this blue background on the left hand side menu (where the list of feeds is), but it's now white. Turns out they made a user study to ask people what they think given the two options and nothing else changed in the app. People consistently said that the version with the white background was faster, although it's the same page. How crazy is this?

Clutter

One last thought - clutter. In these times of TV and limited attention span (who doesn't suffer from ADD, raise hand!) an often mentioned advise is to minimize clutter. It makes sense - the less information you have to process, the less anxious you feel about the new environment (the new web page just loaded). We don't have control over how we percieve things - a flashing animation will attract our attention away from the reason why we're on this page. And we won't like that. So it's good to avoid clutter, too many options, decoration images of beautiful people shaking hands and so on. Less stuff will also mean less markup, images, less page weight, which is great for performance. But clutter is also... relative.

Clutter is not simply a lot of things. Clutter is a lot of useless things and distractions. If you have a lot of exactly what you need, that's not so bad. (If I had a room full of guitars and gear and virtually no option to walk in this room, I won't complain, I'd say it's awesome.)

Also - you can consider culture differences. In Asia people often find it better if they have more content and options to parse through and make sense of on the same page and they don't find it cluttered. They would prefer to wait a bit in order to have their options in front of them. For illustration - just checked that yahoo.com has 146 links on the page and Yahoo! Taiwan has 334.

Parting words

And this concludes day 4 of the performance advent calendar. Takeaways?

• Time is relative
• Keep flow in mind
• Consider the competition, the purpose of the page, the user expectations
• Try to answer "what's the user perceived load time?"
• Provide feedback
• Transitions
• Progressive rendering
• Strive to render something (part of the page) in a blink of eye

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 3 This is the third in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

There's a number of performance rules and best practices that every web developer should know. Some of them are really simple and straightforward - like turning on compression for example. It's so simple it's a sin not to do it. Actually someone from Google search (Kyle Scholz or Yaron Friedman) said that every extra byte you send over the wire is a disservice to your user. I loved this way of putting it.

BTW, I myself am guilty of not turning gzip on this blog, I also don't have a favicon which causes a bunch of 404s every hour, and there's other mistakes. But that's intentional - I want to show that everyone has room for improvement, no matter how big or small a site, no matter if there's a budget for CDN or not. Fast and pleasurable experiences are not reserved for big sites with millions to invest. Many of the performance optimizations are straightforward and within every developer or sysadmin's reach.

Online resources

The place to start learning about performance best practices is the performance section of the Yahoo! Developer Network site. The list of best practices may look overwhelmingly long, but you don't have to do everything at once, pick one low hanging fruit, fix, improve, rinse and repeat.

The description of Yahoo!'s best practices may be a little high level, so I've written some time ago an article on SitePoint that aims to present some of those practices with more concrete steps. For example how to configure gzip when you deal with shared hosting providers that run different versions of PHP and Apache and with different set of restrictions. (Some ISPs are really overly restrictive. If yours is - time to change them. Here's a list questions that your ISP better answer YES to)

Another great place to read up on performance best practices are these two sections in the Google code site - PageSpeed best practices and there performance articles.

Books

If you prefer to cuddle with a good book, O'Reilly has a pretty decent collection of performance titles.

• High Performance Websites (a.k.a. The Bible ) by Steve Souders is the place to start. It covers the first 13 of Yahoo's best practices but with much more detail than the Yahoo! Developer site. Concise and to the point, highly recommended.
• Even Faster Web Sites is the second book by Steve Souders and it's probably a little more advanced than the first. I am also a contributor to this book, I wrote the chapter on image optimization together with Nicole Sullivan. Being a co-author prevents me from gushing this book too much without sounding like an infomercial dude. But hey, just look at some of the other contributors - Douglas Crockford, Nicholas Zakas, Dion Almaer and Ben Galbraith from Ajaxian...
• High-Performance JavaScript is a new title, still not published, where I also contributed a chapter. The book's author is Nicholas Zakas with contributing authors including Julien Lecomte (of YUICompressor fame), Matt Sweeney (YUI3 architect), Ross Hermes (also author of "Pro JavaScript Design Patterns" and Flickr front-end). With an all-star team of contributors and reviewers, you can expect this to be a valuable read.
• Website Optimization by Andrew King is half about performance and half about SEO and conversions. But SEO is also good. Andrew has a very scientific and diligent approach and that's refreshing to see when it comes to SEO.

Community

There's two places to hang out online and ask performance questions and learn from the community:

• The Yahoo! exceptional performance group
• The Google "let's make the web fast" group

Blogs

And then there's blogs. There's quite a few of them.

I tried to assemble today a nice list of performance blogs (and I'm sure I've missed some). Shoved them into a Yahoo! Pipe and did a quick UI with JavaScript to show the posts tail from all the blogs.

So, ladies and gentlemen... announcing...

Planet "Performance"

It's available here - http://www.perfplanet.com/. This site shows the last 20 posts from all the performance blogs I managed to collect.

If you're not much of a browse type of person, the RSS feed for the planet is right here.

If you see something broken on the site, please let me know.

Also if you know of (or run) a blog that talks about performance from time to time, please let me know too so I can add it to the planet.

Thank you

And happy reading!

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 2 This is the second in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

While theoretically you can speed up your site by just blindly following advice from this blog and other sources, it is much better to understand what's going on on the page and what you're dealing with. That's where the tools come in. Some tools give you insight about the network activities going on between the server and the browser (packet sniffers), some help you benchmark code execution on the client (profilers), some even give recommendations specific to performance improvements. You should aim at mastering as many of the tools as possible, because there's no single one that is The Tool. And that's not a bad thing, it's normal, because performance optimization is a multi-discipline activity touching a lot of different aspects of the the development process.

YSlow

(Full disclaimer: I helped with YSlow development and I was the architect of YSlow 2.0 so this fact may or may not have something to do with why YSlow is the first in the list. Plus, this is an unordered list.)

YSlow is an extension to Firebug, that:

• inspects the DOM
• hooks into the Net panel to listen to network activity and discover components that are not part of the DOM

Then the tool looks at the page and the components and tries to figure out how closely they match with Yahoo!'s performance best practices. Then you're given a list of findings with some advice and links for more information on how to improve.

PageSpeed

PageSpeed inspects the page and the components and checks it against conformance with Google's performance best practices.

In addition to that, PageSpeed has some quite advanced features like the Activity Panel which shows more detailed information on the page's, well, activity - such as the browser paint events, javascript parsing, execution, DNS lookups and so on. PageSpeed also tells you how many (and which) JavaScript functions were never called before onload so you can take some hints to lazy-load some of the JavaScript payload (after analysing, of course that the code is not needed in other browsers or other page use cases). Same with CSS - PageSpeed gives you a list of unused selectors so you can check whether you have leftovers from previous versions of the page.

MSFast

MSFast (from MySpace) inspects the page and helps answer many questions left open by YSlow and PageSpeed, such as:

• What's going on with IE?
• What's the memory and CPU footprint of your code?
• How does the page looks like (as in screenshots) while it's being loaded so you can see what the people with slower connections have to experience

PageTest

AOL's PageTest is an IE plugin but also a hosted service which is a great way to show your boss/client performance details without inconveniencing them with challenging download and installation activities. PageTest gives you a waterfall view of the page load and a checklist of things to improve, plus some screenshots of interesting moments during load and even a video - an excellent view of how the page looks like in slow speeds. The hosted service can show you the dial-up experience in 4 different places in the world.

DynaTrace's Ajax Edition

Dynatrace Ajax is a very detailed lower level tool that not only shows the waterfall of components downloads but also the rendering time, CPU consumption, JavaScript parsing and execution. The screenshot above is just the tip of the iceberg of the tool's plethora of views and insights. It's highly recommended. (free, registrationware)

Packet sniffers

A good packet sniffer is indispensable for inspecting the HTTP traffic and figuring out how the page loads and what the request/responses and their headers look like. Here's a list of recommended sniffers, each with something good on top of the others:

• IBM PageDetailer - a mature tool, somewhat simple which makes it a good start, requires registration to download
• Fiddler - very powerful, extensible
• HTTPWatch - (paid, but with a free version) integrates into the browser (both IE and FF) as a panel - very convenient to use. Extensible.
• Microsoft Visual Round Trip analyser (and an excellent writeup)- goes even lower into the packet level of the requests and draws a different view of the waterfall, one that visualizes the TCP packets and the TCP slow start. It also gives recommendations for performance improvements. Built on top of NetMon (Microsoft Network Monitor) to present the data in a more useful and friendly way.
• Charles proxy - the only non-windows tool in the list is an excellent packet sniffer for Mac

Time for a little rant - a more detailed view into the HTTP chunks is something that I think is important (will blog about it as part of this series) and missing from the current tools. HTTPWatch is the only tool that at least tells you the number of chunks and Fiddler prompts you to de-chunk HTTP responses when inspecting the body, which gives you a hint that the response was chunked. I hope to see more in that area, hopefully soon.

Thanks for reading

That concludes day 2 of the performance advent calendar. Hope you'll have fun installing and playing with new toys!

Did I miss a tool that should've been in the list? Let me know.

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 1 This is the first in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

As you've probably heard (and maybe all too often), we live in Web 2.0. This may mean different things to differently inclined folks but for us developers it means more rich Ajaxy pages, communicating more frequently with the server, one-page type of apps (think Gmail). Where "the web as she was meant to be" used to be a document-serving system, things are different now and web pages are increasingly more like applications, and much less document-y.

Let's take a look at what I once called (obviously still under the effect of that dental anesthetic) "The Life of Page 2.0" - a parallel between the human life and the modern day web app.

The life of Page 2.0

On this timeline above you can see the key moments of the life of a page and how they correspond to the human life.

1. It all starts with the page request. Someone types in a URL or clicks a link. This is the moment of page's conception
2. What follows next is the server is "pregnant" with that page. Your server side code will fetch data from somewhere - a database, web services - and crunch that data stitching a string of HTML code output
3. Luckily if there's no complications, abortions, 404s and such, the pregnancy is over, HTML is sent to the browser and the page is officially born. Look, it even has a <title>, which is hopefully something different than "500 Internal Server Error"
4. Then comes the waterfall - downloading all the extra page components required by the page - images, scripts, styles... This phase ends with the onload event. This phase roughly corresponds to the childhood and teen years of our little human, which eventually graduates and becomes a full blown Mr.Page.
5. Right after onload comes a settling process (the young fella finding his/her identity) - attaching even handlers to DOM elements, some initialization work, maybe fetching a few more components, or getting some data via an Ajax call. The page then settles. Status bar stops showing stuff that's being downloaded, indicators stop spinning, cursor is not busy. All is good, the young adult is back from backpacking across Western Europe and Tibet and ready to get married, take over that sales position in dad's company and own a barbeque grill
6. Then life goes on, user interacts with the page. Some pages are quite uneventful, others are full of ups and downs (uploads/downloads), always getting more data, updating and self-improving, always on the move
7. Sooner or later along comes the Grim Reaper to end it all. The user clicks away from the page, making a new request and our Page is laid to rest after a brief onunload moment.

Some comments on Mr. Page's life

First - if that looks like a lot happening, it is. The good news for the performance folk is that since there's a lot going on, it means there's a lot to improve. Performance optimization is a fun and challenging activity which is all but boring.

Next - the onload. While technically onload is a concrete event which should signify when the page is ready, it's not always that simple. The "user onload" is an undefined point in time that could happen before the onload and way after. Depends on the page and on the user. An article type of page could be considered ready when the article title and content are ready. The user happily reads and hence interacts with the page, while images, ads and what not is still being downloaded. Other times the onload may happen relatively quickly, but the actual page content is still being retrieved (Google Reader on the iPhone comes to mind) and the page is far from usable. It's up to you to figure out the "user onload" for your type of page.

Where to improve

Probably the first and most important place for optimization for most pages is the waterfall stage. But you can optimize the page in any of the stages above, prioritizing on where the most time is spent.

Below is a summary of the main optimization activities in each stage, many of these will be discussed in follow-up articles.

1. at request time - for example send less cookies, no cookies for static requests
2. optimize the waterfall - this is pretty simple to understand. "Simple laws of physics", as my fellow Yahoo! and perf extraordinaire Venkat says. I would divide the waterfall optimizations into these categories:
   1. less stuff - your waterfall will be shorter when there's less things to fall. Have less HTTP requests, merge components, use sprites, data URIs/MHTML, remove what you don't need, lazy-load the rest. Use caching and "never expires" policy to improve repeat visits. Remove duplicates, near duplicates, and plain old 404 errors
   2. smaller stuff - once you've removed or merged components, the ones that are left should be as small as they can be - that means use compression, minification, image optimization, zero body 204 components
   3. move out of the way - some things in the waterfall harm more that others, not everything happens in parallel. Parallel is good, means that more stuff is downloaded at the same time and the waterfall will finish earlier. JavaScripts block downloads. Sometimes CSS does too. CSS blocks rendering. Redirects should be burned. DNS lookups cost too, so use less domains.
   4. start early so you can finish early, kind of obvious, eh? This means use flushing and HTTP chinked encoding to start the waterfall even before the page is born. (You know it's going to be a baby girl, err on the pink color scheme)
3. the settling after-onload phase - you can do stuff here too. For example, you can start some of the work at DOMContentLoaded, you don't always need to wait for onload
4. the interactive/life phase - this includes CSS and JavaScript optimizations to make your user interaction smoother and more pleasant and the UI more responsive. Touching the DOM lightly. Towards the end-of-life you can help too. What is a poor old page to do, but leave a good inheritance by preloading some of the components that the children and grandchildren pages may need

End of day 1

Thanks for reading so far! This first article was more high level, an overview of some of the more technical parts to follow. It's good to have a view. It helps you prioritize and adopt a more holistic approach to the optimization effort. At the end of it, it's a marathon, not a sprint.

2010 update:
Lo, the Web Performance Advent Calendar hath moved

I like 24ways.org's idea of a webdev advent calendar - publishing one article per day from Dec 1st to Dec 24. I thought it would be cool to have the same thing on web performance topics. Our young performance community would benefit from a year-end heartwarming bunch of articles, maybe some motivational, some wrapping up interesting ideas and techniques that popped up in the past year, this kind of stuff. Ideally it would be a co-effort by performance folks from around the world, but lacking the PM skills and time to organize it (and coming up with the idea kinda late) I'll try to start alone, hopefully next year it would be a community effort.

So this is now just a placeholder post, I'll update it with links to the articles as I write them, one per day is the plan, will see how it will work. So watch this space, you can follow me via the RSS feed or Twitter.

I have a preliminary idea of the articles I intend to write, but it's not at all final or complete, so if you want to see an article on something, or you want to write a guest post, please comment or get in touch in email.

1. The performance roadmap
2. Performance tools
3. Required reading
4. Psychology of performance
5. Reducing the number of page components
6. The pain points of having fewer components
7. Data URIs, MHTML and IE7/Win7/Vista blues
8. Collecting web data with a faster, free server by Christian Heilmann
9. Duplicates and near-duplicates
10. Caching vs. inlining
11. Reducing the payload: compression, minification, 204s
12. Big list of image optimization tools
13. Give PNG a chance (video)
14. Free-falling waterfalls
15. JavaScript loading strategies by Ara Pehlivanian
16. How To Measure Web Site Performance by Eric Goldsmith
17. Rendering: repaint, reflow/relayout, restyle
18. DOM access optimization
19. The new game show: "Will it reflow?"
20. Extreme JavaScript optimization by Ara Pehlivanian
21. Progressive rendering via multiple flushes
22. iPhone caching
23. CSS performance: UI with fewer images
24. The performance business pitch

In the spirit of the content-to-markup ratio bookmarklet, here's another one that gives you some more data points to help you judge the quality of a page's markup and help answer the old question - where does all this page weight go.

Install the statsy bookmarklet

Drag this link to your bookmarks:

statsy

the results

Once you run the bookmarklet it alerts these stats points:

• JS attributes (e.g. onclick) - this is the sum of all onclick, onmouseover and so on including the attribute names. So for example <a onclick="#"> is 11 characters (bytes) of JavaScript attributes code
• CSS style attributes - the sum of all style="..."
• Inline JS - the sum of all the contents of all script tags (excluding the tag itself)
• Inline CSS - sum of all <style> tag contents
• All innerHTML - this is document.documentElement.innerHTML.length, it should be close to the ungzipped size of a page, provided the page is not doing a lot of DOM manipulation
• # DOM elements - the total number of elements on the page is counted simply using document.getElementsByTagName('*').length

Here's example output:

The code

The code is here for your tweaking pleasure

Thanks!

Hope you'll find this bookmarklet useful when looking at a page as a companion to YSlow/PageSpeed.

What else should I add to this bookmarklet? # of font tags, # of table tags...?

Since my previous post on @font-face and gzipping, Paul Irish has asked (and so has @KLTF) what about WOFF?

WOFF is a newer format with built in compression and ability to store meta data.

So I took this stnf2woff utility and converted all the TTF and OTF files from my previous tests to WOFF. Below are the results.

Basically, the tests confirm that WOFF is already compressed, it doesn't need to be gzipped by the server and woff file sizes is comparable to the compressed TTF or OTF.

The tables show the original font-file size and the percentage shaved off after gzipping, converting to WOFF and gzipping the WOFF.

TTF to WOFF

OTF to WOFF

Adding custom fancy fonts to a web page seems to be all the rage these days. Looking at some examples with Net panel on, I saw some of those font files are 100K which is a pretty big price to pay for an ornament like this. I mean you can build whole pages, with fancy scripts, images, gradients and all, that are less than that size.

Made me wonder about the sizes of those font files and in particular is there an easy gzipping win. Turns out yes - you can and should gzip the custom font files you use with @font-face.

UPDATE Oct 19, 09: Gzip all font files except those in WOFF format, more info...

Quick background

In order to include a custom font on a page you can use Paul Irish's bulletproof way, something like:

@font-face {
  font-family: 'gzipper';
  src: url(yanone.eot);
  src: local('gzipper'),
         url(yanone.ttf) format('truetype');
}

body {
    font-family: gzipper;
}

What's going on here?

1. You give your custom font a name, like "gzipper"
2. This line is for IE - src: url(yanone.eot);. It asks for an .eot font file
3. This line is for pretty much every other browser url(yanone.ttf) format('truetype'). It needs a .ttf font file
4. That last line has variations, you can use an .otf file instead of .ttf, interchangeably, like so url(yanone.otf) format('opentype')
5. SVG font file (for Chrome) is also an option instead of, or in addition to, ttf, like: url(yanone.svg) format('svg')

So that leaves us with 4 different types of files:

• svg
• ttf
• otf
• eot

How about gzipping these?

SVG

SVG is easy, it's an XML file. XML files are plain text files so they must always be gzipped. Potential savings? Anywhere around (and I'm pulling this off my hat) 30-70%.

TTF

Turns out TTF is also mostly a text file, and it can safely be served gzipped. I tested FF 3.5, Opera 10, Safari 4, works everywhere. The test page is here.

Grabbing some files to test from here and one from here (and having my kid's tae-kwon-do lesson to sit through on this beautiful Santa Monica Saturday morning) I gzipped them and compared the results.

On average - 41.48% savings.

I couldn't help but notice huge differences between different font files, even in this small sample. Sansation_Regular.ttf is 10K while FFF_Tusj.ttf is 1 Meg! So keep in mind to check the size of the font you're about to use, there might be smaller and similar fonts.

OTF

Same thing, testing with some files found here, the results are 45.40% savings on average when gzipping the .otf font files.

EOT

The case with .eot is a little more interesting (like anything to do with IE, right?). Apparently MS provides software, called WEFT to create those files, but looks like people have hard time making it work.

The good thing about this tool is that it can create compressed .eot files which is excellent. You can see some examples linked to from this MSDN article. Some of those EOTs in the examples are as small as 4K which is totally acceptable. So it's worth the time to figure it out because it can save quite a bit.

The other option to create EOTs is to use the free open source tool ttf2eot and convert TTFs to EOTs. The tool doesn't create compressed files, so generally the size of the EOT is about the size of the input TTF. In this case you must gzip the .eot before you send it to the browser.

So I took my test TTFs, converted to EOT and checked the gzip savings. On average - 41.49% savings when gzipping the uncompressed .eot

I tried my test page in IE6 and it worked with the gzipping, so I'm assuming IE7,8 are fine too.

Summary

• Always send your font files gzipped, average 40% to be won, but could be 70% you never know
• Dig into WEFT to make sure you create small, compressed EOTs
• Keep an eye on the font file sizes you're about to use, they can be huge. When you have a choice, opt in for the 10K font file and not the 1.5 megs one

links

• My gzipped fonts test page
• Configure gzip in hosting environments with variable degree of hostility
• Paul Irish on "FOUT" and more on loading font files
• Paul Irish - bulletproof font face declaration
• Snook's font mastery post
• TTF-to-EOT converter
• WEFT tool for EOTs
• MSDN examples
• Free fonts - awesome
• 10 free fonts
• Free fonts

Not sure if I've ever put that in writing, but CSS irks me with its verboseness. I mean things like: background-position, padding-bottom, text-decoration... those are long property names, repeatedly used in stylesheets. And there's pretty much nothing you can do about them in terms of minification, it's not like they are variables in JavaScript which your YUIcompressor can rename to, like, a, b and c, respectively.

Or can you do something about it? Here's the big idea:

the big idea

I though of using JavaScript arrays to store each CSS property only once, and also each value only once. Then have another array with selectors and a way to match selector to indices in the property/value arrays. A simple loop in JavaScript can reconstruct the stylesheet. Simple, right? And avoids all these verbose repetitions.

Drawback - breaking the separation of concerns. Relying on behavior (JS) in order to get presentation (CSS). If someone has disabled JS they get no styles. That's a big no-no, but there's one case where you can safely break the separation - in lazy-loaded functionality. Here's what I mean...

Combining lazy-loaded assets

Page loads ok in its basic form - styles and all. Then it gets progressively enhanced with JavaScript. JS adds new features. If you have JS OFF, you don't get them. But if you do, it makes sense to have the feature atomic - one file containing both JS and CSS. This way saving HTTP requests (rule #1 for faster pages). That's a pretty cool idea in itself and you can find implementations in the wild, including high-traffic sites such like Google search and Yahoo homepage.

Ok, with the SoC out of the way, what about that failed experiment?

A failed experiment

(psst, demo here)

You start with something like:

#test {
    border: 1px solid blue;
    font-family: Verdana;
}
a {
    padding: 0;
    font-family: Arial;
}

Then (all that during build time, not run time!) you have a parser that will walk over the CSS and "understand it" into an array, like:

[
    {
        selector: '#test',
        rules: [
            {
                property: 'border',
                value   : '1px solid blue'
            }
        ]
    },

]

Then from that structure you produce 4 arrays that contain:

1. all selectors,
2. unique properties (sorted),
3. unique values (sorted) and
4. a map that connects selectors to their properties and values.

s:['#test','a'], // s-selectors
p:['border','font-family','padding'], // p-properties
v:['1px solid blue','Verdana',0,'Arial'], // v-values
r:[[0,0,1,1],[2,2,1,3]] // r-map or rules

Finally, your build process produces a JavaScript function that reconstruct the CSS string, which you can then shove into a style tag and be done with it.

(function(o,s,i,j){
s='';
for(i=0;i<o.s.length;i++){
  s+=o.s[i]+'{';
  for(j=0;j<o.r[i].length;j=j+2){
    s+=o.p[o.r[i][j]]+':';
    s+=o.v[o.r[i][j+1]]+';';
  }
  s+='}';
}
return s;
})({
s:['#test','a'],
p:['border','font-family','padding'],
v:['1px solid blue','Verdana',0,'Arial'],
r:[[0,0,1,1],[2,2,1,3]]
})

Results

The demo is here, you can paste your CSS and see what the results are for your styles. Here's what I found:

• Really tiny stylesheets don't make sense, because the overhead exceeds any savings
• Otherwise it helps! You get smaller CSS, yey! Champagne! Caviar! Everybody dance!
• But... after gzipping the result, the AFTER is actually bigger than BEFORE. Cold shower. The hole thing goes down the drain. C'est la vie, gzip does a better job than I can do with JS arrays.

Here's the results of munging one 16K gzipped CSS found on the Yahoo! homepage;

Raw:
  source: 95735
  result: 87300
    percent: 8.81077975662%
Gzipped:
  source: 16211
  result: 16730
    percent: -3.20152982543%

cssmin.js is a JavaScript port of YUICompressor's CSS minifier.

The motivation

Minifying CSS helps reduce file sizes and makes your pages faster and your users happier. YUICompressor is cool but is written in Java, which can be a blocker for some folks - you know JVM, command line, classpaths... No more excuses, now you have a simple light JavaScript version. And as you know, JavaScript is everywhere, so you can run it however you want, integrate with your editor and so on.

The links

• online demo
• the code on github

The integration

If you want to integrate the library into your environment, it's really easy. It's just one file with one function in it. So, just a simple function call:

var result = YAHOO.compressor.cssmin(input_css_code);

The credits

Julien Lecomte - creator of YUICompressor
Isaac Schlueter - he maintains the YUICompressor and is the author of the original cssmin utility which was ported to Java by Julien.

Ha, what about a quiz? Guess the language of Isaac's original cssmin and I'll send you a free copy of Even Faster WebSites and I'll sign my chapter. Seriously.

UPDATE: For Ruby folks, there's a Ruby port from the Ryan Grove.

A few months ago I gave a 5 minute Ignite talk at the Velocity conference. (My previous post talks about the Ignite experience.) I thought they recorded the video and wanted to share but seems like it's not happening. So below are the slides from slideshare.

The ignite talk rules are: 20 slides that change automatically after 15 seconds. But I took the liberty of hacking the format a little bit. For one, the 18th (and 19th) slide which was my main point is duplicated. And then I had some transitions, like in the first two slides for example, where different text or pictures change after a timeout. I expanded those transitions for slideshare, so the slide count is 40-something, but at the presentation it was still total of 20 slides in 5 minutes.

If you follow the slides on slideshare, be sure to click the Notes tab which has what I said, more or less, at each slide. The full script is below.

And yes, if you like it, please vote for my South by Southwest panel proposal. It has the same name "Image Weight Loss Clinic" but will be a much longer version of this 5 minute Iginite and what is more - Derek Tonn of graphicsoptimization.com will share the stage with me. Pretty exciting, so cast you votes, leave a comment and help me get to SxSW.

The Script

Here's what I said, more or less:

1. Let’s talk a little bit about cutting back on that bandwidth bill. Saving some money. And without touching the code. How do we do that? One way is…
2. Images. Ever since the dawn of times people have been using images. Cavemen scraping on their cave walls… and today… Today we have websites. And LOLcats.
3. Half of the average web page weight today is images. And most of these images are too fat. They could use a diet and lose some weight but without losing quality, not even a single pixel.
4. So, welcome to the Image Weight Loss Clinic. Our first patient today is... some random page found on the Internets. Hmm, what have we got here… Header graphic, speaker photos, logos, rotating banners – the usual stuff.
5. First problem - there are some GIFs. We shouldn’t be using GIFs. PNG is the format for graphics - buttons, icons, and so on. The palette type of PNG, also known as PNG8 is good as a GIF and actually better. And PNG filesizes are smaller. Well, except for really tiny GIFs, but let’s ignore these, they don’t matter much.
6. You should convert all your GIFs to PNG8. There’s a number of tools to do this in a batch. You can use ImageMagick for example. “Instant results or your money back”. Especially since ImageMagick is free.
7. For your JPEG photos, use JPEGTran. It’s free, cross-platform, command-line tool, easy to script. And it’s lossless, you keep the photo quality. If you run it without setting any options, that’s good enough. Run with the –optimize and you’ll get even better results.
8. Option –progressive makes JPEGs that render in the browser from low resolution to high, as opposed to top-to-bottom. If the JPEG is bigger than 10K (which is most JPEGs), the progressive encoding has a good shot at giving you a smaller image. You can always brute force – run with both options and take the smaller result.
9. Option –copy none strips all comments and meta data - camera model, geo location, everything. If you own the image, strip the meta data. You’ll be surprised how much bloat is in there. If you don’t own the image, check with the owner. Option –copy all keeps all meta data.
10. Next – PNG. There’s a lot of options to optimize PNGs - all lossless and scriptable on the command line. Some even have nice graphical UIs built on top of them, you know, for our command-line-challenged friends.
11. There’s even browser-based UIs, built on top of some of the tools. For example, smush.it, which is now part of YSlow runs PNGcrush. Google’s Page Speed runs OptiPNG.
12. With so many PNG tools, how do you decide which one to pick? They are all good tools and they do their best, so if you’re in a hurry, just pick one and go with it, script it. You’ll be pleasantly surprised by the outcome.
13. Now if you’re serious about PNG weight loss, you can run all the tools you can lay your hands on. Run the next tool on the result of the previous, try with their different options switches. They all do different things, and results may vary from one image to the next. Run them all in sequence and get an even smaller file at the end.
14. Or, if you’re hardcore – run PNGSlim. PNGSlim is actually a Windows batch file that runs most of the other tools. It’s very slow, it can take hours to go through a handful of files. But at the end, it will give you the leanest, most optimized PiNG.
15. Let’s see some results. That page has a total page weight of over 800K, including scripts, styles, everything. Over 80% of the weight is images. And what happens after optimization?
16. We can strip quite a bit, over 200K of unnecessary image information. 200K, that’s over 30% if the image weight. The page looks exactly the same, pixel by pixel. There was no manual work involved, just running some tools. I think 30% is pretty impressive.
17. To summarize, don’t use GIFs, run JPEGtran on your photos, and for the graphics - any PNG or all PNG tools.
18. So the tools are available out there, now all you need is a process that runs the tools before your images go live. Make image optimization part of the push process.
19. These are all lossless operations that don’t require you to look at the images to verify they’re OK. So you can only win! And often you’ll see some amazing and surprising results. Even if you normally do a good job with image sizes, there’s always a few that slip as you race against the deadline. The thing is – setup a process, so you don’t even need to think about it.
20. Thanks very much, everybody! I’m Stoyan from Yahoo! Search. Enjoy Velocity, it's a great conference!

Yesterday I came across this post (via Ajaxian), it talks about how you might be able to end up with smaller file sizes (after gzipping) if you help gzip a little bit by sorting the CSS properties and values in stylesheets, so they are as similar and close to each other as possible.

Now obviously it's not easy to do a safe sorting, there's many edge cases to be taken care of, but the question before diving into safeness is - is it really worth it?

In attempt to answer that I did a very simple CSS parser in JavaScript that will sort the properties and values, then I tested with a few files.

Long story short - it's probably not worth it. But if you want to help, please use the online tool and report if you see a considerable benefit for your own stylesheets.

How it works

Again, bear in mind this is not safe to do, your pages might look different before and after optimization. This is just a test if alphabetizing is really worth it in terms of gzipped filesize.

Before:

b {
  b: b c a;
  a: a b c;
}

a {
  a: a;
}

After

b {
  a: a b c;
  b: a b c;
}
a {
  a: a;
}

As you can see, order of selectors is preserved, properties within each CSS block are sorted and their values are sorted too.

Help by testing your CSS

Since the original question was - is it worth modifying YUI Compressor to sort properties and values, you can help by following these two steps:

1. minify your real-life CSS with YUI compressor (online version here) and then
2. paste the result in my parser/sorter tool to see if you can gain much after compression.

Please share your results in the comments section of this post.

Results so far?

I tested with two files well under 10K and the results were ever-so-slightly negative - the results were 0.05% bigger.
Tested also with a few files over 10K and the results were 1.5% savings. Hardly worth it. But I hope someone can prove me wrong and show cases where alphabetizing can be well worth it. After all, the sweetest optimization is the one you don't have to do, but your build process can manage by itself

What can your tired old page, once loaded and used and read, can do for your user? It can preload components needed by the next page, so when the users visit the next page, they have the new scripts, styles and images already in the cache. Next page loads faster and the user's happy. On its death bed you tired old page has left a good inheritance for future generations. Good old page.

How can you go about preloading for the next page? By waiting for onload of the current page and requesting the new components. Here are 4 ways to do so, all using a timeout of 1 second after page load so that prefetching doesn't interfere with the user experience on the page.

One way... (DOM)

Using DOM you can create a new LINK element and a new SCRIPT element and append them to the HEAD. For images - it's a one-liner new Image.src="..."

The drawback of this method is that your CSS is executed against the current page and might affect display. Same for JavaScript - it's executed. The image is simply requested and never displayed.

window.onload = function() {
    setTimeout(function(){

        // reference to <head>
        var head = document.getElementsByTagName('head')[0];

        // a new CSS
        var css = document.createElement('link');
        css.type = "text/css";
        css.rel = "stylesheet";
        css.href = "new.css";

        // a new JS
        var js = document.createElement("script");
        js.type = "text/javascript";
        js.src = "new.js";

        // preload JS and CSS
        head.appendChild(css);
        head.appendChild(js);

        // preload image
        new Image().src = "new.png";

    }, 1000);
};

DOM test page

For this way of doing it you can use any JavaScript library's helper methods to load stuff on demand. Good examples - YUI Get and LazyLoad

... or another ... (using iframe)

Another option is to create an iframe and append your components to its head. Using an iframe you can avoid the CSS potentially affecting the current page. JavaScript will still be executed.

window.onload = function() {
    setTimeout(function(){

        // create new iframe
        var iframe = document.createElement('iframe');
        iframe.setAttribute("width", "0");
        iframe.setAttribute("height", "0");
        iframe.setAttribute("frameborder", "0");
        iframe.setAttribute("name", "preload");
        iframe.id = "preload";
        iframe.src = "about:blank";
        document.body.appendChild(iframe);

        // gymnastics to get reference to the iframe document
        iframe = document.all ? document.all.preload.contentWindow : window.frames.preload;
        var doc = iframe.document;
        doc.open(); doc.writeln("<html><body></body></html>"); doc.close(); 

        // create CSS
        var css = doc.createElement('link');
        css.type = "text/css";
        css.rel = "stylesheet";
        css.href = "new.css";

        // create JS
        var js = doc.createElement("script");
        js.type = "text/javascript";
        js.src = "new.js";

        // preload CSS and JS
        doc.body.appendChild(css);
        doc.body.appendChild(js);

        // preload IMG
        new Image().src = "new.png";

    }, 1000);
};

IFRAME test page

... I'm gonna find ya ... (static page in iframe)

If your components are static you can create a page that has them all and load that page into the dynamic iframe. Static means knowing them in advance, not relying on page's JavaScript to figure them out on the fly. As you can see, much simpler than the previous code.

window.onload = function() {
    setTimeout(function(){

        // create a new frame and point to the URL of the static
        // page that has all components to preload
        var iframe = document.createElement('iframe');
        iframe.setAttribute("width", "0");
        iframe.setAttribute("height", "0");
        iframe.setAttribute("frameborder", "0");
        iframe.src = "preloader.html";
        document.body.appendChild(iframe);

    }, 1000);
};

IFRAME static page test

... I'm gonna GETcha, GETcha, GETcha! (with Ajax)

Finally - Ajax. Scripts are not executed, CSS not used for rendering. Nice and clean. For simplicty the code has no support for browsers that don't know what XMLHttpRequest is.

window.onload = function() {
    setTimeout(function(){

        // XHR to request a JS and a CSS
        var xhr = new XMLHttpRequest();
        xhr.open('GET', 'new.js');
        xhr.send('');
        xhr = new XMLHttpRequest();
        xhr.open('GET', 'new.css');
        xhr.send('');

        // preload image
        new Image().src = "new.png";

    }, 1000);
};

Ajax test page

Thanks!

Any other ways you can think of?

Here are the slides from a tech talk I gave at eBay last week, I called it "Don't make me wait! or how to build high-performance web apps", inspired by, you guessed it, the excellent book "Don't make me think" by Steve Krug.

There's some intimate details on YSlow's scoring algo towards the end and some stuff on progressive rendering and chunked encoding.

Thanks very much to the folks who attended the talk and thanks to eBay for organizing this brownbag. And thanks for the gifts!

This is one of those note-to-self type of posts. Just went through the exercise of installing a number of PNG tools on the Mac and here are some notes. The instructions below should probably work on any unix box.

AdvDef, AdvPng, ...

There is a number of Adv* tools (advdef, advpng, advmng, advzip) packed together as AdvComp. Installation difficulty: fairly straighforward.

Download:

$ curl http://softlayer.dl.sourceforge.net/sourceforge/advancemame/advancecomp-1.15.tar.gz \
         > advcomp.tar.gz

Uncompress:

$ tar -xzvf advcomp.tar.gz

Compile and install:

$ cd advancecomp-1.15/
$ sudo ./configure
$ sudo make install

Test:

$ advdef
advancecomp v1.15 by Andrea Mazzoleni
Usage: advpng [options] [FILES...]

Modes:
  -z, --recompress      Recompress the specified files
Options:
  -0, --shrink-store    Don't compress
  -1, --shrink-fast     Compress fast
  -2, --shrink-normal   Compress normal
  -3, --shrink-extra    Compress extra
  -4, --shrink-insane   Compress extreme
  -f, --force           Force the new file also if it's bigger
  -q, --quiet           Don't print on the console
  -h, --help            Help of the program
  -V, --version         Version of the program

Excellent! Next.

OptiPng

OptiPng is another easy install.

Download and decompress:

$ curl http://superb-west.dl.sourceforge.net/sourceforge/optipng/optipng-0.6.3.tar.gz \
            > optipng.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1035k  100 1035k    0     0   171k      0  0:00:06  0:00:06 --:--:--  196k
$ tar -xzvf optipng.tar.gz

Compile and install:

$ cd optipng-0.6.3
$ sudo ./configure
$ sudo make install

Test:

$ optipng
OptiPNG 0.6.3: Advanced PNG optimizer.
Copyright (C) 2001-2009 Cosmin Truta.

Synopsis:
    optipng [options] files ...
Files:
    Image files of type: PNG, BMP, GIF, PNM or TIFF
Basic options:
    -?, -h, -help	show the extended help
    -o <level>		optimization level (0-7)		default 2
    -v			verbose mode / show copyright and version info
Examples:
    optipng file.png			(default speed)
    optipng -o5 file.png		(moderately slow)
    optipng -o7 file.png		(very slow)
Type "optipng -h" for extended help.

Beauty! Next - pngout.

PNGout

PNGOut's source is not distributed openly. But there are binaries for a number of platforms here.

Download:

$ curl http://static.jonof.id.au/dl/kenutils/pngout-20070430-darwin.tar.gz \
         > pngout.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 97597  100 97597    0     0  46704      0  0:00:02  0:00:02 --:--:-- 56739
$ tar -xzvf pngout.tar.gz

This way you end up with a binary named pngout-darwin. Rename and move somewhere where executables live:

$ sudo mv pngout-darwin /usr/bin/pngout 

Test:

$ pngout
PNGOUT [In:{PNG,JPG,GIF,TGA,PCX,BMP}] (Out:PNG) (options...)        Apr 30 2007
by Ken Silverman (http://advsys.net/ken)
Mac port assistance by Jonathon Fowler (http://jonof.edgenetwork.org/pngout)
PNGOUT optimizes PNG size losslessly using my own deflate algorithm (not Zlib)
With the right options, it can often beat other programs by 5-10%. Options:
   -c# PNG output color type: 0=Gray, 2=RGB, 3=Pal, 4=Gray+Alpha, 6=RGB+Alpha
   -f# PNG output filter...

PngRewrite

PNGRewrite was a little trickier, until I realized I need to install libpng first.

Download and install libpng:

$ curl ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-1.2.37.tar.gz \
         > libpng.tar.gz
$ tar -xzvf libpng.tar.gz
$ cd libpng-1.2.37/
$ sudo ./configure
$ sudo make install

Download, unzip pngrewrite:

$ curl http://entropymine.com/jason/pngrewrite/pngrewrite-1.3.0.zip \
         > pngrewrite.zip
$ unzip pngrewrite.zip

Compile pngrewrite (the make file didn't work for me) and copy the binary where executables are comfortable.

$ gcc -lpng pngrewrite.c -o pngrewrite
$ sudo cp pngrewrite /usr/bin/

Test:

$ pngrewrite
pngrewrite v1.3.0: PNG image palette optimizer
Usage: pngrewrite infile.png outfile.png

That's all, folks

Installing PNGCrush? Blogged before.

And if anyone has an idea how to get deflopt installed, please comment.

I'm back from the most excellent JSConf (JavaScript Conference) in Washington D.C. I'm tired and need sleep but the conference was, hands down, the best conference I've ever attended. It was all about the community, it was inexpensive, with parties all around, both speakers and attendees were treated exceptionally well, in fact there wasn't a big difference since attendees presented just as good content on the B-track as did the A-track.

Here are my slides, "High Performance Kick Ass Web Apps (the JavaScript edition)". I changed the slides on the night before the presentation (and after the first night's party). The thing is that I was halfway through the slides and approaching 90 slides and I realized there's no way I will deliver so much content. One day... I'll prepare my slides way in advance and will actually practice delivering the talk...