phpied.com

I'm part of a band that has an album out now. I know, right? (links: excuse-for-a-site, amazon, itunes).

I wanted to put up all the songs on the site, but seems like there's a little dissonance in the band whether this is a good idea. Plan B: 30s samples. Like the bigwigs do on Amazon and iTunes.

But while their samples are random, a band can do a better job in picking parts that are representative of the overall sound. I though - let me pick my solo stuff only as an exercise. So there: Anaconda Limousine: the guitar parts.

I wanted to use command-line ffmpeg, of course, because all music software is like Photoshop to me, just can't figure out what's going on with so much UI. Turned out I needed sox too.

And then I want to use HTML5 Audio to play the samples.

I thought: an audio sprite would be a good idea, put all samples in one file, then JS can update the UI depending on which sample is playing. And I thought might be neat to have the JS turn the volume up and down to fade in/out the samples, like iTunes does. Turns out sox is doing this so nicely, that I let it do it.

Samples

I started by listening to the songs and taking notes with song #, start and end.

var slices = [
  {song: 1,  start:   8, end:  21},
  {song: 1,  start: 301, end: 323}, // from 3:01 to 3:23
  {song: 1,  start: 405, end:   0}, // 0 means till the end
  {song: 2,  start:   0, end:  30},
  {song: 2,  start: 305, end: 318},
  {song: 2,  start: 330, end:   0},
  {song: 3,  start:   0, end:  20},
  {song: 3,  start: 333, end:   0},
  {song: 4,  start: 303, end:   0},
  {song: 5,  start:   0, end:  20},
  {song: 5,  start: 300, end: 333},
  {song: 7,  start:   0, end:  20},
  {song: 7,  start: 340, end:   0},
  {song: 8,  start:   0, end:  25},
  {song: 8,  start: 313, end:   0},
  {song: 9,  start: 155, end: 239},
  {song: 9,  start: 350, end:   0}
];

Start 0 means start from the beginning of the song, end 0 means go to the end.

The time format is optimized for easy typing (I was walking, typing in Notes app on the iPhone). Turned out I need to convert the times to seconds:

function secs(num) {
  if (num <= 60) {
    return 1 * num
  }
  num += '';
  return num[0] * 60 + num[1] * 10 + num[2] * 1;
}

And I need album meta data too, with name of the song, filename and duration:

 
var songs = [
  {name: "Virus",     fname: "01-virus",     duration: 436},
  {name: "Yesterday", fname: "02-yesterday", duration: 346},
  {name: "All for you", fname: "03-all4u",   duration: 404},
  {name: "Damage",    fname: "04-damage",    duration: 333},
  {name: "Everyday",  fname: "05-everyday",  duration: 444},
  {name: "Girl of mine", fname: "06-girlomine", duration: 338},
  {name: "Fool on the hill", fname: "07-fool",  duration: 413},
  {name: "Faultline", fname: "08-faultline", duration: 347},
  {name: "Parting is such sweet sorrow", 
                      fname: "09-parting",   duration: 420}
];

Ripping the album

In the interest of quality I wanted to work with WAV and then encode in m4a, ogg and mp3.

On TuneCore.com there's a nice step-by-step instruction how to rip a CD to WAV in iTunes, because it uses mp3 by default.

So then I had the files:

01-virus.wav
02-yesterday.wav
03-all4u.wav
04-damage.wav
05-everyday.wav
06-girl.wav
07-fool.wav
08-faultline.wav
09-parting.wav

Slicing and fading

I used ffmpeg to do the slicing, like for example the first sample:

$ ffmpeg -i 01-virus.wav -ss 5 -t 20 ff-0.wav

-ss is start time and -t is duration. As you see instead of starting at 8 seconds (as described in the samples array) I start earlier. This is to give some more music for fade in/out.

For fade in/out I used sox. I didn't know this awesome command line tool existed, but found out while searching how to combine wav files (I know for mp3 you can just `cat`)

I don't want to fade in when the sample starts at the beginning. Or fade out then the sample happens to end at the end of the song. sox takes fade-in duration (or 0 for no fade in), stop time (0 for till the end of the file/sample) and fade out duration (again 0 is no fade out)

I used 3 secods fade in, 4 fade out. The sox commands are one of:

$ sox in.wav out.wav fade 3 0 0
$ sox in.wav out.wav fade 3 0 4
$ sox in.wav out.wav fade 0 0 4

And since I have all the configs in JS, JS makes perfect sense to generate the commands. Hope you can make sense of the comments:

var fadein = 3;
var fadeout = 4;

var merge = ['sox'];
slices.forEach(function(s, index){
  var ff = ['ffmpeg -i'];
  ff.push(songs[s.song - 1].fname  + '.wav'); // in file
  ff.push('-ss');
  ff.push(s.start ? secs(s.start) - fadein : 0); // start of the slice
  ff.push('-t');
  ff.push(!s.end ? 
      1000 : 
      secs(s.end) - secs(s.start) + fadein + fadeout); // end slice
  ff.push('ff-' + index  + '.wav'); // out file
  
  var sox = ['sox'];
  sox.push('ff-' + index  + '.wav'); // in file
  sox.push('s-' + index  + '.wav'); // out file
  sox.push('fade');
  sox.push(s.start ? fadein : 0); // fade in, unless it;s the beginning of the song
  sox.push(0); // till the end of the slice
  sox.push(s.end ? fadeout : 0); // fade out unless it's The End
    
  console.log(ff.join(' '));
  console.log(sox.join(' '));
  
  merge.push('s-' + index  + '.wav');
});
 
merge.push('_.wav');
console.log(merge.join(' '));

Running this gives me a bunch of commands:

ffmpeg -i 01-virus.wav -ss 5 -t 20 ff-0.wav
sox ff-0.wav s-0.wav fade 3 0 4
ffmpeg -i 01-virus.wav -ss 178 -t 29 ff-1.wav
sox ff-1.wav s-1.wav fade 3 0 4
[....]

sox s-0.wav s-1.wav s-2.wav s-3.wav [...] s-16.wav _.wav

2 for each sample (slice and fade) and one last one to merge all faded results. Nothing left to do but run the generated commands.

Save for Web

Final step is to convert the result _.wav to a web-ready format: m4a, ogg or mp3:

$ ffmpeg -i _.wav _.m4a
$ ffmpeg -i _.wav _.mp3
$ ffmpeg -i _.wav -acodec vorbis -aq 60 -strict experimental _.ogg

Turn it up!

Enjoy Anaconda Limousine: The Guitar Parts (ogg, m4a or mp3) with all samples in one file.

And come back later for the JS player part.

See you!

Background

• HTTPWatch (automation)
• ...with PHP (and again and again, and response)
• JavaScript shell scripting

I gave this short presentation at the recent Yahoo FE summit's open mic, here are the slides and some notes.

Slides and screencast vid

Screencast to see the thing in motion:

Notes

Here's the transcript of the slides as produced by slideshare.net. I've added some notes here and there to make it more readable when the presenter is missing.

JavaScript shell scripting - Presentation Transcript

1. JavaScript is everywhere #42:
         C:> WSH

2. Stoyan

   I do programming.

3. Programming

   There are many options to choose from when you decide top practice the
   art and craft of programming.

4. JavaScript

   ... is a very good one. Simply because JavaScript...

5. ... is everywhere

6. On the server

   node.js, asp.net

7. Mobile

   html5, phonegap, titanium

8. Desktop

   XULRunner lets you create cross-OS GUI apps

9. Browser extensions

   FF, Chrome, bookmarklets...

10. Photoshop

    yep, that too
    Several Adobe products actually let you script common/uncommon/programmable tasks

11. Form validation too!

    (this was supposed to be funny)

12. Shell scripting

    let's talk about shell scripting with JavaScript

13. In Windows

14. WSH: Windows Scripting Host

    All reasonable Windows machines (at least as old as Win2000) have this Windows Scripting Host in there.
    You can write VBScript or JavaScript (OK, JScript) to ... well, script.
    How does it work?

15. C:>edit hello.js

    You create a file.

16. var a = "Hello",
        b = " WSH!",
        c = a + b;
            
    WScript.Echo(c);

    Put any old JavaScript in there and print out some results

17. C:>cscript hello.js
    Hello WSH!

    And this is how you run it.

    Or this:

    C:>wscript hello.js

18. Open apps

    In addition to regular sysadmin tasks (copy, write files, move) you can open and script applications too.

19. var ie = new ActiveXObject("InternetExplorer.Application");
    ie.Visible = true;
    ie.navigate(yahoo.com);

    This is how you open IE and point it to a page.

    Notice something familiar? ActiveXObject - the thing we used in IE before it got native XMLHttpRequest

20. Firefox?

    Can you also open FF?

    Not really, as it doesn't have COM interface (whatever that is).

    But there's an easy solution

21. HTTPWatch

    Finally we come to the topic of the talk.

22. Speed

    Performance is critical for the success of any web app.

    Really, it is.

    When talking about improving speed there are two main steps:

23. 1. Fix with YSlow

    Take a slow page, run YSlow, do what it says.

    Voila - a fast(er) page.

24. 2. No regressions

    The second step is to not allow regressions.

    Whatever you fix in step 1 will be slow in the next few months.

    Even less than months the bigger the team or the rate of changes.

    So to prevent regressions, you need to constantly...

25. Monitor

26. Set limits

    The simplest way to prevent regressions is to set some limits.

    If you go over the limits, an email is sent, an alarm sounds, panic instills and you've got to fix whatever cause it was.

27. e.g.
    max 2 scripts
    max 2 styles
    max 9 images
    max 0 redirects

28. Scripting HTTPWatch

    Watching for violations of the limits manually every day is not a job anyone would want.

    So automating it will help a great way towards employee satisfaction

29. var http = new ActiveXObject("HTTPWatch.Controller"),
        ie = http.IE.New(),
        ff = http.Firefox.New();

    This is how you open IE and FF with HTTPWatch's help.

    FF - yey!

30. // browser cache
    ie.clearCache();
    
    // show HTTPWatch
    ie.OpenWindow(false);

    Examples of stuff you can do with the HTTPWatch API.

    You can for example hit the page with empty cache and then again with full cache.

    Best of all - these are the real browsers with their sometimes kinky behaviors.
    Actually if you setup several machines for the monitoring (or somehow do multiple IEs)
    you can test with different versions of the browsers. Nothing synthetic!

31. ie.Record();
    ie.GotoUrl("yahoo.com");
    http.Wait(ie, -1);
    ie.Stop();

    Start monitoring, go to a page, stop monitoring after the page "settles" meaning some time after onload.

    ie.CloseBrowser();

32. new HTTPWatch()
          http://github.com/stoyan/etc/

    I did this JavaScript thingie to make everything a little easier.

33. var http = new HTTPWatch(ff);
    http.go(search.yahoo.com);
    http.done();

    Example usage.

34. [video]

35. var har = http.toHAR();
    har = eval(( + har + ));
    
    print(har.log.browser.name);
    print(har.log.browser.version);
    print(# requests: );
    print(har.log.entries.length);

    Opening and navigating browsers is cool. But we need some data back.

    HTTPWatch can export a HAR (HTTP Archive) file. I have this toHAR() method.
    It writes the file, than reads and returns it.
    You can than eval() it because it's just a JSON string.
    And you get the data back in convenient JS objects and arrays.

36. Internet Explorer 6.0.29...
    # requests: 10
    
    Firefox 3.5.6
    # requests: 15

    Result of running the above.

37. [video]

38. var comps = http.getComponentsByType();
          
    for (var i in comps) {
      print(i);
      print(comps[i].length);
    }

    Another method I thought would be useful is getComponentsByType()

39. redirect: 1
    text/html: 3
    image/gif: 4
    image/png: 3
    text/javascript: 1

    Results of the code above.

40. But wait...

    There's more

41. What about DOM?

    So far we only talked about HTTP traffic inspection - headers and such.

    Good news is that you can also inspect the DOM (in IE only) for any potential red flags.

    For example having the number of DOM elements sharply increase.

42. [video]

43. var http = new HTTPWatch();
    http.go(search.yahoo.com);
    
    var d = http.watch.container.document;
    
    print(d.getElementsByTagName(*).length);
    print(d.documentElement.innerHTML);

    That works!

    All your DOM voodoo skillz are suddenly reusable.

44. require(statz.js);
          
    var doc = http.watch.container.document;
    var html = http.har.log.entries[0].response.content.text;
    
    var out = statz(document, html);
    print(out.join("\n"));

    This is me repurposing two old bookmarklets that gather some interesting stats (one of them was even featured on Ajaxian, remember?).

    It was pretty easy to repurpose the bookmarklets, because it's just JavaScript.

    The stats thingie can inspect both raw HTML that went over the wire, as well as innerHTML that was the result of any additional DOM manipulations.

45. JS attributes (e.g. onclick): 1207 bytes
    CSS style attributes: 883
    Inline JS: 5243
    Inline CSS: 5015
    All innerHTML: 17283
    # DOM elements: 134
    
    Total size: 14124 bytes
    Content size: 401 bytes
    Content-to-markup ratio: 0.03

    Sample results.

46. To summarize...

47. JavaScript
    WSH
    HTTPWatch
    Monitor
    DOM and HTTP
    IE and Firefox

48. Thanks
          
    phpied.com

If you haven't seen it yet, YSlow for Chrome hit the streets couple of weeks ago. (And Google's own PageSpeed did too yesterday. (And there's now DynaTrace for Firefox. (And WebPageTest for Chrome. (What a month for x-browsering (word?) the performance tools! (And the month's not even over yet)))))

BTW, I closed all the parentheses (or else...).

So anyway, I was eavesdropping on a twitter conversation where Sergey (of ShowSlow) was asking for beacons from YSlow for Chrome, more specifically - when will they start working. I thought I should check how my old baby YSlow 2.0 (this presentation is still pretty relevant) is doing in its new environment.

YSlow 2.0

In YSlow 2.0 things are pretty decoupled. Makes it easier to bring to any possible environment or browser. So rules are rules (you can add, remove, tweak them, combine them into rulesets), results are results, presentation is separate, and so are the additional tools, HAR import/export (forthcoming), etc. Only (ideally) small additions are needed to glue the core of YSlow (the linting part) with a new environment.

In Chrome

It's my first time touching anything Chrome-y, but turned out its pretty easy. Just a bit of file system hunting revealed where code for the extension goes.

/Users/[USERNAME]/Library/Application Support/Google/Chrome/Default/Extensions/[WEIRD-EXTENSION-ID]/

e.g.

/Users/stoyanstefanov/Library/Application Support/Google/Chrome/Default/Extensions/ninejjcohidippngpapiilnmkgllmakh/

On Windows:

C:\Documents and Settings\[USERNAME]\Local Settings\Application Data\Google\Chrome\User Data\Default\Extensions\ninejjcohidippngpapiilnmkgllmakh\

In there you can find three JavaScript files. I could be wrong but here's what I think goes in there:

1. yslow-chrome.js looks like it contains the reusable parts - rules, etc - packaged for Chrome minus the Firefox stuff and bundled into a single file
2. content.js is small and not too exciting
3. controller.js is the Chrome-related parts

In controller.js is where we hack.

YSlow events

In YSlow 2.0 we decided to make use of a simple observer pattern implementation and fire events whenever interesting stuff happens. Especially useful since figuring all the data for all resources better be asynchronous.

Once yslow "peels" the page figuring out the components, gets each component data, headers, etc, then runs the lint rules, finally it fires a lintResultsReady event.

("Peeling a page" I heard for the first time from Steve Souders and for what I know he should be credited with this term describing the activity of figuring all components that go into a page)

All we need to do is listen to this event and send the beacon.

YSLOW.util.event.addListener(
  'lintResultReady', 
  function (o) {
    //...b-b-beacon! ...
  }
);

There's a YSLOW.util.sendBeacon() which does precisely that, so we need to call it and we're done.

Preferences

Firefox has a built-in (native) system to manage preferences. You know, the stuff you tweak in about:config. This is where we put the preferences - beacon yes/no, beacon URL, beacon data verbosity.

In Chrome such native preference system probably exists, but YSlow is currently not taking advantage. (Just guessing here.)

Luckily all calls to get preferences are abstracted in YSLOW.util.Preferences.getPref(prefname, defaultvalue). The default value is returned if a better one cannot be found.

So we can just overwrite the getPref() method to return the default value, unless it's a preference we care about, such as the beacon URL:

YSLOW.util.Preference.getPref = function(what, defaultval) {
  switch (what) {
    case 'beaconUrl':
      return 'http://www.phpied.com/beacon.png';
    case 'beaconInfo':
      return 'all'; // or "basic"
    default:
      return defaultval;
  }
};

Integration

As mentioned we'll hack into the controller.js, we don't want to touch the yslow core stuff. The controller.js is just one immediate function and our hack goes right before the last line. (Or even after it, probably doesn't matter)

(function () {
    // ... yslowy stuff ...
 
    // hack start
    // ...
    // hack end
 
    doc.ysview.setSplashView(...
}());

The complete thing is something like:

(function () {
 
    // ... slo, slo ...
 
    // hack start
    YSLOW.util.Preference.getPref = function(what, defaultval) {
      switch (what) {
        case 'beaconUrl':
          return 'http://www.phpied.com/beacon.png';
        case 'beaconInfo':
          return 'all'; // or "basic"
        default:
          return defaultval;
      }
    };
 
    YSLOW.util.event.addListener('lintResultReady', function (o) {
      var con = o.yslowContext,
          result = con.result_set;
      YSLOW.util.sendBeacon(result.url, result.overall_score, con);
    });
    // hack end
 
    doc.ysview.setSplashView(...
}());

Conclusion

So there - you can run YSlow in Chrome and send yourself (or showslow.com) beacons.

You probably don't need that so bad that you can't wait till next YSlow for Chrome ships with this thing working. But here it is.

And hopefully you learned a bit about YSlow internals so that you can start hacking yourself and/or wait till YSlow shows up on github (soon!) and start sending diffs. I personally can't wait.

Shoutout goes out to Marcel and Betty who are doing awesome stuff with YSlow (slides). And looks like even more is to come!

Very rough notes from the event that just ended a few minutes ago

======== 1 ========

Tony Gentilcore (google) and Anderson Quach (MSFT) talked about the W3C working group and the specs for measuring performance:

• Navigation timing spec - HTML document: DNS, handshakes, load events
• Resource timing - how long JS, CSS, audio, video... take
• User timing - interactions, such as clicking a link

Challenges: user security, privacy. Small overhead for browsers. Useful actionable data out.

Navigation timing:

• window.performance - IE9 has no vendor prefix
• otherwise window.webkitPerformance
• FF4 probably shipping it too

Properties:

window.performance.timing
  .domainLookup
  .connect
  .domContentLoaded
  .loadEvent
  ....etc

Specs: http://test.w3.org/webperf/specs/NavigationTiming/

Resource Timing spec - upcoming

foreach resource 
  redirect
  fetchStart
  fetchEnd
  ...

User Timing - upcoming

  mark('pagelet')
  mark('something-else')
  ...
  
  >>> getMarks();
  {
    'pagelet': 123456767
  }

Stretch goals:

• painting times
• CPUs
• visibility (is the page in a background tab)
• allow real yealding setTimeout(0)
• don't animate faster than the browser paints

mailto: public-web-perf@w3c.org

======== 2 ========

Above the Fold Time: Measuring Web Page Performance Visually
Speaker: Jake Brutlag
Help from: Zoe Abrams, Pat Meenan (webpagetest) Google

What the user perception of page load timing is.

AFT (Above the Fold Timing) algo
AFT: When content above the fold stops changing and reaches its final state

Integrated in WebPageTest.org

Motivation

• AFT more representative than DOMContentLoaded or onload
• onload - different in IE and FF, e.g. async scripts block onload or not

Method:

• video capture
• AFT cutoff time - upper bound, e.g. 45 sec for DSL
• split pixels into static and dymanic (ads)
• AFT = latest change of a static pixel

Magic number - 5 determined empirically
static pixels are those that change less than 5 times

Application - validate other metrics, cross-browser, visual
Limitation - lab only

{I ran webpagetest.org on facebook.com with AFT on.
The results are very close to onload, as expected.
It's a simple page. Results:

http://www.webpagetest.org/result/110315_P9_5MDJ/

}

== break ==

Note from the sponsor StrangeLoop: a downloadable "State of the Union"
A study of top 2000 Alexa retail sites. Findings:

1. 11.2 sec onload avg
2. top 100 sites slower than the lower ranked
3. those using CDNs are not faster
4. failed to implement the basics - YSlow stuff

======== 3 ========

Next-gen YSlow
Betty Tso and Marcel Duran, Yahoo!

YSlow for Chrome last week
Not as accurate as Firefox cause headers, timing come from:
1. Crawl DOM
2. x-domain requests

Announcing YSlow bookmarklet version
for mobile and all other browsers to be released early april
YSlow now on Git?

Lots coming up, e.g:

• Multi-language support
• HAR-to-YSlow
• YSlow export to JDrop

Bookmarket works like:
1. crawl DOM
2. Use YQL to get headers

     SELECT * FROM data.headers WHERE
      url IN ('http://...', 'http://...')

======== 4 ========

Visibility into Mobile Performance
Steve Souders, Google

Demo #1: http://blaze.io/mobile/
WebPageTest.org framework connected to iPhone and Android apps
Mobile waterfalls
Demo #2: pcapperf
1. setup wifi hotspot on a pc/mac
2. tcpdump to capture .pcap file
3. upload .pcap to http://pcapperf.appspot.com app
4. observe waterfall via pcap2har and harviewer
5. run pagespeed on the har
Demo #3: JDrop
http://jdrop.org
Upload JSON so you can run something on mobile and view/analyze results in desktop's large screen
Works with the bookmarklet tools part of Steve's package of mobile meta-tool

Last year I compared some CSS minifiers, namely YUICompressor, CSSTidy (with "small" vs. "safe" settings), PHP PEAR's CSS lib and Minify (detailed results). Now that I've done some work on the YUICompressor and since there's a new kid on the block from Microsoft I thought I should give it another go.

I only compared CSSTidy with best compression, YUICompressor and MS' Ajaxmin. The verdict is still the same - CSSTidy looks the best, but there really isn't that big of a difference between the three (and virtually no difference between YUIC and Ajaxmin), especially if you take into account gzipping.

Still on average you get about 35% size reduction when you minify and 80% when you gzip the minified CSS. Meaning when all is done your users only download 19-20% of what they would normally do if you don't perform these simple optimizations.

Experiment

Source CSS files came from csszengarden.com - 213 CSS files.

For the YUICompressor I used the JS verison, on windows with WSH, described in the previous post.

For CSSTidy I used exactly the same code as the last year.

Lastly, Ajaxmin I ran with the default options:

$ AjaxMin.exe in.css -out out.css

Results

CSSTidy wins with bringing the size down to 63.29% of the original on avegare. The other two follow closely with 66.47% (ajaxmin) and 66.41% (YUI). After gzipping: 19.44% (tidy), 19.62% (ajaxmin) and 19.63% (yui). In other words YUI's cssmin performs ever so slightly better that Ajaxmin by 0.06% before gzipping, but after gzipping Ajaxmin wins with 0.01%.

Overall the results show that you shouldn't sweat too much over the choice of CSS minifier. They are all pretty close and you should pick the one that is most convenient for your own use and build process.

CSSTidy and Ajaxmin have parsers, so theoretically they have a much better chance. YUI's cssmin is a bunch of regular expressions and no proper parsing. In reality they are all performing really similarly. Part of the reason is that CSS is not as minifiable as JavaScript as most of it is long selectors and long prperty names (background-image, text-decoration, -webkit-transform-origin, anyone?), which are not renamed to shorter names the way JS variables are.

One random observation (and hint for Ajaxmin folks) is that cssmin strips selectors with empty rulesets while Ajaxmin does not. E.g. #selector {} is totally gone in cssmin but stays in ajaxmin.

Detailed results

There's a new version and now hosted on AMO (addons.mozilla.org). Get the new version there and it will take care of auto-updates in the future.

WFT

Web Testing Framework (WTF) is an extension to YSlow that tests for the following shady web dev practices:

• Use of <blink>
• Use of <marquee>
• Use of <font>
• Missing doctype
• Use of spacer GIFs
• Use of <a href="#"..> and <a href="javascript:...>

Gimme!

• The code is in GitHub
• To install click here

Note that this extension requires Firebug and YSlow2

Help

Please report any bugs here and also suggestions for more checks for what you think is bad practice (and is technically possible to test by a tool, as opposed to a human)

Thanks!

Thanks to Ryan Grove for inspiration with the naming of the new tool

Motivation

The motivation is mainly to demonstrate how easy it is to create new extensions and new checks to YSlow. They don't even need to be performance-related.

There are two basic concepts. 1. A rule is a type of check like "use gzip". 2. The pool of available rules can be combined into rulesets. The API is therefore just:

YSLOW.registerRule({...});
YSLOW.registerRuleset({...});

The first takes an object containing name, id, info about the new check and a lint() function that performs the check. The second accepts a config object - which rules go into the ruleset.

Check this file - that's all it takes to do a new extension.

The previous parts are here (building and testing) and here (what gets minified). Now let's see how YUI CSS min handles CSS hacks.

As you know CSS hacks often use errors in CSS parsers in browsers to target specific browser versions and supply additional rules to work around other issues in said browsers. That makes any CSS tool's job slightly more challenging. Not only does the tool have to avoid repeating the browsers errors, but also has to understand what browsers got wrong and support it too. Fun stuff. Isn't it a joy being a web developer?

So here are some hacks that are tested to work with the YUICopmpressor's CSS min.

Underscore/star hack

The simplest ever hack to target IE6 and IE7. In the example below normal browsers see 1px dropping _width and *width as invalid, IE7 ignores the *, drops the _width as invalid and sees 3pt, IE6 ignores the _ and sees _width as width, so it sees 2em.

CSS min doesn't parse and doesn't understand CSS properties, so it accepts pretty much any property.

Before:

#element {
    width: 1px;
    *width: 3pt;
    _width: 2em;
}

After:

#element{width:1px;*width:3pt;_width:2em}

Child selector hack

CSS min strips comments but there is this child selector hack people use to hide declarations from IE7 and below.

CSS min retains empty comments that immediately follow > (thanks go out to Chris Burroughs)

Before:

html >/**/ body p {
    color: blue; 
}

After:

html>/**/body p{color:blue}

IE5/Mac hack

This hack targets IE5/Mac, if anyone still worries about this browser. The hack is retained after minification, only it's minified.

Before:

/* Ignore the next rule in IE mac \*/
.selector {
   color: khaki;
}
/* Stop ignoring in IE mac */

After:

/*\*/.selector{color:khaki}/**/

Box model hack

This hack uses valid CSS and there's no special use of comments so it's retained.

Before:

#elem { 
    width: 100px; /* IE */
    voice-family: "\"}\""; 
    voice-family:inherit;
    width: 200px; /* others */
}
html>body #elem {
    width: 200px; /* others */
}

After:

#elem{width:100px;voice-family:"\"}\"";voice-family:inherit;width:200px}html>body #elem{width:200px}

Seems like the code highlighter chokes here though. It ain't easy

That's all, folks!

Thanks and please, feel free to suggest improvements and report bugs. Also play with the web UI of the JS-version here to see for yourself what it does to your code.

The first part is here. It was more about building the YUICompressor, writing and running test cases. Now let's see what the compressor does exactly to your CSS.

BTW, you can play with the web UI to see for yourself how the minifier works.

Stripping comments and white space

This is the bare minimum a minifier can do. And when it comes to CSS, this is also the place where ther biggest improvement comes from. In JS for example you can rename variables and save bytes, but in CSS the possibilities are more limited. No shorter way to say text-decoration, unfortunately.

So before:

/***** 
  classmates stuff
*****/
.classmates {
    /* after 10 years */
    weight: considerable;
}

After:

.classmates{weight:considerable}

Special comments

Stripping comments is nice but not always ok. Sometimes you need to retain copyright information. Use ! at the beginning of the comment to mark the comment as special.

Before:

/*!
  (c) copyright copyleft
*/
.classmates {
    /* after 10 years */
    weight: considerable;
}

After:

/*!
  (c) copyright copyleft
*/.classmates{weight:considerable}

Thanks to the charmingly insisting Billy Hoffman and the valid case he presented, the bang (!) itself is preserved too. This way you can safely double minify. Also lint tools (such as Zoompf, YSlow and PageSpeed) can see the ! and conclude that this comment is there intentionally, not because you forgot to minify.

Striping last semi-colon

The last semi-colon in a declaration block is out. So keep it in your source for maintenance purposes and let the minifier take care of stripping it out.

Before:

a {
  one: 1;
  two: 2;
}

After:

a{one:1;two:2}

Extra semi-colons

One semi-colon is all you need, so the minifier will strip an accidentally added one.

Before:

p :link { 
  ba: zinga;;;
  foo: bar;;;
}

After:

p :link{ba:zinga;foo:bar}

No empty declarations

Empty declaration blocks don't do anything, so why send them over the net?

Before:

.empty { ;}

After:

(nothing...)

Zero values

A zero is a zero. Zero pixels or % or centimeters or whatever, it's still zero. Also sometimes (when everything is a zero) you need just one zero instead of four, or three or two.

Before:

a { 
  margin: 0px 0pt 0em 0%;
  background-position: 0 0ex;
  padding: 0in 0cm 0mm 0pc
}

After:

a{margin:0;background-position:0 0;padding:0}

Floats

For values such as 0.something, the 0 is not needed.

Before:

::selection { 
  margin: 0.6px 0.333pt 1.2em 8.8cm;
}

After:

::selection{margin:.6px .333pt 1.2em 8.8cm}

Colors values

RGB color values are nice, but not the most concise form. Make them hex. Also AABBCC hex can be the shorter ABC. But don't touch RGBA and don't touch the IE filter values in quotes.

Before:

.color {
  me: rgb(123, 123, 123);
  impressed: #ffeedd;
  background: none repeat scroll 0 0 rgb(255, 0,0);
}

After:

.color{me:#7b7b7b;impressed:#fed;background:none repeat scroll 0 0 #f00}

Before:

.cantouch {
  alpha: rgba(1, 2, 3, 4);
  filter: chroma(color="#FFFFFF");
}

After (no color minification) :

.cantouch{alpha:rgba(1,2,3,4);filter:chroma(color="#FFFFFF")}

Single charsets

Only one charset is allowed per stylesheet. So, if there's more than one, strip it. It may happen when merging several stylesheets into one.

Before:

@charset "utf-8";
#foo {
  border-width: 1px;
}
 
/* second css, merged */
@charset "another one";
#bar {
  border-width: 10px;
}

After:

@charset "utf-8";#foo{border-width:1px}#bar{border-width:10px}

Alpha opacity

There's a shorter way to write opacity filter for IE.

So before:

code {
   -ms-filter: "PROGID:DXImageTransform.Microsoft.Alpha(Opacity=80)"; /* IE 8 */
   filter: progid:DXImageTransform.Microsoft.Alpha(Opacity=80);       /* IE 4-7 */
}

After:

code{-ms-filter:"alpha(opacity=80)";filter:alpha(opacity=80)}

There are more filters that could be shorten besides the opacity, but MSDN suggests the longer syntax should be used, So a bit more experimentation is needed here...

Thanks!

Whew, sort of a lengthy post. Thank you for reading and coming up next time... hacks

If you have ideas, comments, your bug reports are welcome

Honored to be a part of the YUI project, I am now helping with the maintenance of the CSSMin part of the YUICompressor. My changes are now part of the trunk on github, so I'm official. Next on the agenda is documenting the thing, so that's what I'll try to do here, maybe in a few posts. You know, divide and conquer.

PHP, Java and a JavaScript port

Originally written in PHP by Isaac Schlueter and ported to Java by Julien Lecomte, CSSMin got a JavaScript port by yours truly some time ago. Because, after all, JavaScript is the language of the web, isn't it?

You can play with the latest git version of the JS port online here.

I'm also happy to report that the JS port is now used in PageSpeed and YSlow (as you probably know Firefox extensions are written in JavaScript)

Building

If you want to play on your own with the source version of YUICompressor without waiting for the next release, you can build it like so:

1. Checkout or download the code from http://github.com/yui/yuicompressor/
2. Navigate to the root yuicompressor/ directory
3. Type ant and hit enter

In order for this to work you need a somewhat recent Java SDK installed and also Ant running. (On the Mac, just do port install apache-ant to get Ant)

This is for the Java version, the JS version needs no building, of course.

Tests

There's a bunch of new tests now (and if you want to contribute to the project, you can always write more tests and test cases for any bugs), you can run them with the suite script that Isaac wrote:

1. cd tests/
2. ./suite.sh

One thing I added (and loved it) is to run the tests using the JS port as well. Since the JS min part is using Mozilla's Rhino (slightly modified), Rhino is part of the code. So I'm using this already available JavaScript interpreter to run the JS port. Convenient.

The procedure to write new tests is simple:

1. Create source CSS file in the tests/ directory, e.g. new-test.css
2. Create a new file with the expected result and name it with a .min extension, e.g. new-test.css.min

You can use the handy-dandy online version to help with the tests creation.

Next time

With those details out of the way, the next time I'll talk more about the different things that CSSMin does to your CSS code. Thanks for reading!

So I've been thinking and talking to folks about this idea of having one-stop shop for all your minification needs. Minification of JS and CSS as well as image optimization helps site performance by reducing download sizes. This is good. But not a lot of people do it.

People don't do it, because it's a PITA It's simple enough, but with deadlines upon you and all that, you don't want to do an extra step. That's why having a build process helps, by automating this. But setting up a build process is yet another PITA. So it goes.

So my idea was to help busy designers and developers, that wouldn't invest their time researching which minifiers are good, downloading setting up, learning about the 10+ PNG optimization tools... That's how the the idea for the one-click OMG tool came about. (One-drag is more appropriate, come to think of it...) One tool that runs on all operating systems - Win, Mac, Linux - and delivers all minification and optimization tools you need as one package.

Running

Running the tool is as simple as drag/dropping a bunch of files and directories. Here I've dropped "wordpress" directory. The tool recursively looks into the dropped files for things it can optimize. More information here.

Download

Version 0.0.1 is here. It doesn't do image optimization, only JS and CSS minification, but please feel free to download and give it a shot. Unzip the package for your OS and run omg.exe (Windows), OMG.app (Mac), or the omg binary (Linux)

• Windows
• Mac
• Linux

Open source

The code is on GitHub. Fork and enjoy.

The developer's notes are there too - how to setup, run, package. Also a list of todos if you want to help.

Next?

This is just a preliminary version. Feel free to join, comment, suggest. Hate the name? Say so

Personally, looks like my plate is very full for the next moth or two, so I probably won't be actively working on the tool. I hope though the foundation is good enough and relatively documented, should be easy to pick up if anyone's interested in contributing.

Built with XUL

This has been a learning experience for me with XULRunner. I loved it. I love the idea of being able to create cross-OS desktop apps with JavaScript alone.

Behind the scenes, I'm using my JavaScript port of YUICompressor's CSSmin and Doug Crockford's JSMin. JSMin should be replaced with YUICompressor (or Google closure compiler) in the next release.

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 12 This post is part of the 2009 performance advent calendar experiment (12 articles down, 12 more to go). Stay tuned for the articles to come.

Let's continue the topic of reducing file sizes started with the previous post and talk about making images smaller.

Engineer's guide to smaller images

Just to set the frame of the discussion - this is not about using Photoshop or setting the quality of the JPEGs and so on. I realize that we, web developers, wear many hats - we're designers, client/server coders, Apache/Linux admins, database heros. But this post is not about using image programs and assumes that you or your designer has already created the images to be used on the site with the appropriate colors, quality and so on.

Now, repeat after me: you should never take an image from the designer and put it up on the web.

Most often this image is bigger than it should be. It's not the designer's fault, it's usually the software used to produce the image.

You shouldn't put an image up on your server before running it through a few tools. These tools are free, open source, cross-platform and can be run on the command line, hence scripted and run in batches over a large number of files - by you, or even better, automatically by a build deployment process. It's OK to batch-run those files without human intervention, because these tools simply optimize the files, they don't change the pixel information, so the "after" images look exactly like "before", only smaller.

Selecting the right file format

The first step towards leaner images is to select the correct file format. There are three options:

1. JPEG for photos. Photos contain millions of colors and smooth transitions of colors. Blue skies, clouds, sunset, your dog, lolcats - all photos.
2. GIF is for the occasional "loading..." progress animation. This is it, no other uses for GIFs.
3. PNG is for everything that's not a photo or an animation. That includes all icons, graphs, buttons, gradients, and what not. Any image with sharp transitions of colors. Think (but don't use for) text. Sharp transitions become "dirty" in JPEG.

PNG is an interesting topic for a follow up post, for now let's stop here. If it's not a photo, it should be PNG. An edge case is a screenshot for example. Depends on what's on the screen of course, but most often JPEG will give a smaller size if you can live with the artifacts around the sharp edges (like text).

Optimizing GIFs

Unfortunately many people still use GIFs even for non-animated images. That's a mistake. PNG is a superior format and yields smaller file sizes.

People still use GIFs because they think either that a/ GIF is smaller than PNG or b/ there's lack of support for PNG in browsers. These are misconceptions and I'll talk more about them tomorrow.

So, the way to optimize a GIF is to convert it to PNG.

You can use many tools to turn your GIFs to PNGs, including ImageMagick and OptiPNG.

# option 1: ImageMagick (if you know the filename)
$ convert logo.gif logo.png

# option 2: ImageMagick again (if you just convert all files in a directory)
$ mogrify -format png *.gif

# option 3: OptiPNG
$ optipng *.gif

These are just some of the options, I'm sure there are others.

After the conversion you can optimize the new PNGs like all other PNGs

Optimizing PNGs

There are various ways to write a PNG file. Unfortunately not all image editing programs do a good job at writing PNGs for the minimal file size.

Luckily, to fill the void, there's a great number of tools that excel in writing small PNGs. There are different ways to optimize a PNG:

1. Stripping out "chunks" - PNG is an extensible format. Extensions come in the form of chunks and most chunks are not needed for the web.
2. Reducing the number of colors and switching between PNG types - truecolor PNG, grayscale, palette...
3. Chosing the best "filter". Filters are a pre-compression step. You can compress any type of file, but when you know the file is an image, you can do better. Filters are for this purpose.
4. Optimizing the actual DEFLATE compression algorithm

Different tools specialize in one or more of these areas. So the more tools you run, the better the results will be. But you have to run at least one tool, always. You'll be surprised how unoptimized are most PNGs coming from common commercial image programs.

So, to optimize a PNG you shoul run as many of the following programs as possible:

# optipng (skip -o7 to run faster)
$ optipng -o7 my.png

# pngcrush (skip -brute to run faster)
$ pngcrush -rem alla -brute -reduce my.png my.png.temp
$ mv my.png.temp my.png

# pngout - closed source, non-windows binaries here
# (add parameter -s2 to run faster)
$ pngout my.png

# advpng (use -z2 to run faster)
$ advpng -z4 my.png

# deflopt - windows only
$ deflopt my.png

Other tools to note include PNGrewrite, PNGNQ and PNGquant, but they are limited because they deal only with PNG8 (256 colors) files. PNGNQ and PNGQuant are actually converters from truecolor to PNG8, so they are not guaranteed to be lossless. PNGreqwrute is safe to use, it will just silently fail if the file has more than 256 colors, so there's nothing to lose.

Oh, and another, excellent tool - PNGOptimizer, windows-only has both command line interface and a GUI.

PNGSlim for the hardcore PNG optimization

If you're really serious about optimizing your PNGs, the tool is called PNGSlim. It's a Windows-only batch file that runs pretty much all tools above and runs them (especially PNGOut) with all kinds of parameters, hundreds of times. So it can take a while to run.

Optimizing JPEGs

JPEG is a lossy format (you lose information every time you save it, even if you choose 100% quality), but there are some operations that can be done losslessly - such as tweaking comments and meta information, cropping, rotating to 90, 180, 270 degrees. The tool that does this magic is called JPEGTran and is likely already on your unix/linux box. If not - here's how to install it (for Windows - get the .exe here)

So to optimize a JPEG losslessly you remove the meta information and optimize the so called Huffman tables. For bigger JPEGs (bigger than 10K) you can also convert the image to progressive coding.

# strip meta and optimize
$ jpegtran -copy none source.jpg > destination.jpg

# strip meta and convert to progressive coding
$ jpegtran -copy none -progressive source.jpg > destination.jpg

# keep all meta but still optimize
$ jpegtran -copy all source.jpg > destination.jpg

Important note on stripping meta

Only strip meta information from images you own the rights for and have permission. Otherwise you're committing a crime. Photographers put important copyright information in meta markers.

Optimizing GIF animations

Remember - no GIFs other than animations. For animations, run GIFsicle (pronounced "yo' mama" ) berfore you put them up:

# GIFSicle
$ gifsicle -O2 source.gif > destination.gif 

More tools?

These are the core tools for image optimization. There's a number of wrapper tools that are more or less UIs on top of these, because there are people who don't like consoles (really?!). I'll list the few that I can think of, please comment if you know of others, especially for windows. It's nice to give nice UIs to give to designers so they can drag-drop optimize images too.

• smush.it, created by yours truly and Nicole Sullivan, now part of YSlow - runs pngcrush, jpegtran, gifsicle
• PageSpeed runs optipng, jpegtran
• PunyPNG - originally inspired by smush.it, but more advanced
• ImageOptim - Nice easy UI for Mac, runs most of the tools above (hope your company firewall doesn't block the site because of the domain name )
• PNGSquash another UI for Mac, runs advpng, pngcrush, optipng
• PNG Monster is for Windows, runs many PNG tools, you can drag/drop on it
• IrfanView - my favorite image viewer for Windows has a plugin to use PNGOut
• WP-Smushit is a WordPress plugin by Alex Dunae which sends all your image uploads to smush.it for optimization. Talk about easy!

More reading

Series of articles on YUIBlog:

• Choosing the right format
• 4 steps to size reduction
• Progressive JPEG

presentations:

• Image Optimization - 7 mistakes
• Image Weight Loss Clinic - a 5-minute "ignite" case study of optimizing a web site. Spoiler: 30.15% image size reduction. Transcript here.

and more:

• Derek Tonn's graphics optimization blog
• Sergey Chikuyonok's excellent articles on optimizing PNGs and optimizing JPEGs
• Installing PNG tools on the Mac
• The book "Even Faster Web Sites" has a chapter written by yours truly and Nicole Sullivan

Thanks!

Thank you for reading. Now you have a whole lot of tools/toys to install and play with. Image optimization is an easy way to improve performance, it's just running a bunch of tools. You don't need to worry that the quality will suffer (so the designer won't be disappointed in you ). So you can only win. You may win quite a bit, you may win just a little (anywhere between 5 to 30% savings is what I've seen on random live sites when I was working on and testing smush.it). The thing is you'll almost always win something.

And because it's human to forget to optimize the images before you push them live, do take the time to setup the optimization step as part of the automated deployment process.

And to summarize once again the steps of what this automated process would be:

1. Convert GIFs to PNG. Then relax and take a deep breath (instead of flaming the unfortunate soul who created the GIFs) while you string replace "gif" with "png" in all your styleshets
2. Run PNGs through optipng, pngcrush, pngout, any or all of the tools listed above
3. Run JPEGtran
4. Run GIFsicle

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 11 This post is part of the 2009 performance advent calendar experiment. Stay tuned for the next articles.

After removing all the extra HTTP requests you possibly can from your waterfall, it's time to make sure that those that are left are as small as they can be. Not only this makes your pages load faster, but it also helps you save on the bandwidth bill. Your weapons for fighting overweight component include: compression and minification of text-based files such as scripts and styles, recompression of some downloadable files, and zero-body components. (A follow-up post will talk about optimizing images.)

Gzipping plain text components

Hands down the easiest and at the same time quite effective optimization - turning on gzipping for all plain text components. It's almost a crime if you don't do it. Doesn't "cost" any development time, just a simple flip of a switch in Apache configuration. And the results could be surprisingly pleasant.

When Bill Scott joined Netflix, he noticed that gzip is not on. So they turned it on. And here's the result - the day they enabled it, the outbound traffic pretty much dropped in half (slides)

Gzip FAQ

How much improvement can you expect from gzip?

On average - 70% reduction of the file size!

Any drawbacks?

Well, there's a certain cost associated with the server compressing the response and the browser uncompressing it, but it's negligible compared to the benefits you get

Any browser quirks?

Sure, IE6, of course. But only in IE6 service pack 1 and fixed for after that. You can boldly ignore this edge case, but if you're extra paranoid you can disable gzip for this user agent

How to tell if it's on?

Run YSlow/PageSpeed and they'll will warn you if it's not on. If you don't have any of those tools just look at the HTTP headers with any other tool, e.g. Firebug, webpagetest.org. You should see the header:

Content-Encoding: gzip

provided, of course, that your browser claimed it supports compression by sending the header:

Accept-Encoding: gzip, deflate

What types of components should you gzip?

All text components:

• javascripts
• css
• plain text
• html, xml, including any other XML-based format such as SVG, also IE's .htc
• JSON responses from web service calls
• anything that's not a binary file...

You should also gzip @font-files like EOT, TTF, OTF, with the exception of WOFF. Average about 40% to be won there with font files.

How-to turn on gzipping

Ideally you need control over the Apache configuration. If not full control, at least most hosting providers will offer you ability to tweak configuration via .htaccess. If your host doesn't, well, change the host.

So just add this to .htaccess:

AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml application/javascript application/json

If you're on Apache before version 2 or your unfriendly host don't allow any access to configuration, not all is lost. You can make PHP do the gzipping for you. It's not ideal but the gzip benefits are so pronounced that it's worth the try. This article describes a number of different options for gzipping when dealing with uncooperative hosts.

Rezipping

As Billy Hoffman discovered, there's potential for file size reduction with common downloadable files, which are actually zip files in disguise. Such files include:

• Newer MS Office documents - DOCX, XLSX, PPTX
• Open Office documents - ODT, ODP, ODS
• JARs (Java Applets, anyone?)
• XPI Firefox extensions
• XAP - Silverlight applications

These ZIP files in disguise are usually not compressed with the maximum compression. If you allow such downloads from your website, consider recompressing them beforehand with maximum compression.

There could be anywhere from 1 to 30% size reduction to be won, definitely worth the try, especially since you can do it all on the command line, as part of the build process, etc. (re)Compress once, save bandwidth and offer faster downloads every time

15% uncompressed traffic

Tony Gentilcore of Google reported his findings that a significant chunk of their traffic is still sent uncompressed. Digging into it he realized there's a number of anti-virus software and firewalls that will mingle with the browser's Accept-Encoding header changing into the likes of:

Accept-Encoding: xxxx, deflxxx
Accept-Enxoding: gzip, deflate

Since this is an invalid header, the server will decide that the browser doesn't support gzip and send uncompressed response. And why would the retarded anti-virus program do it? Because it doesn't want to deal with decompression in order to examine the content. Probably not to slow down the experience? In doing so it actually hurts the user to a greater extend.

So compression is important, but unfortunately it's not always present. That's why minification helps - not only because compressing minified responses is even smaller, but because sometimes there is no compression despite your best efforts.

Minification

Minification means striping extra code from your programs that is not essential for execution. The code in question is comments, whitespace, etc from styles and scripts, but also renaming variables with shorter names, and various other optimizations.

This is best done with a tool, of course, and luckily there a number of tools to help.

Minifying JavaScript

Some of the tools to minify JavaScript include:

• YUICompressor
• Dojo ShrinkSafe
• Packer
• JSMin
• ... and the new kid on the block - Google's Closure compiler

How much size reduction can you expect from minification? To answer that I ran jQuery 1.3.2. through all the tools mentioned above (using hosted versions) and compared the sizes before/after and with/without gzipping the result of minification.

The table below lists the results. All the % figures are % of the original, so smaller is better. 29% means the file was reduced to 29% of its original version, or a saving of 71%

As you can see gzipping alone gives you about 70% savings, minification alone cuts script sizes with more than half and both combined (minifying then gzipping) can make your scripts 85% leaner. Verdict: do it. The concrete tool you use probably doesn't really matter all that much, pick anything you're comfortable with to run before deployment (or best, automatically during a build process)

Minifying CSS

In addition to the usual stripping of comments and whitespaces, more advanced CSS minification could include for example:

// before
#mhm {padding: 0px 0px 0px 0px;}
// after
#mhm{padding:0}

// before
#ha{background: #ff00ff;}
// after 
#ha{background:#f0f}
//...

A CSS minifier is much less powerful than a JS minifier, it cannot rename properties or reorganize them, because the order matters and for example text-decoration:underline cannot get any shorter than that.

There's not a lot of CSS minifiers, but here's a few I tested:

• YUI compressor - yes, the same YUI compressor that does JavaScript minification. I've actually ported the CSS minification part of it to JavaScript (it's in Java otherwise) some time ago. There's even an online form you can paste into to test. The CSS minifier is regular expression based
• Minify is a PHP based JS/CSS minification utility started by Ryan Grove. The CSS minifier part is also with regular expressions, I have the feeling it's also based on YUICompressor, at least initially
• CSSTidy - a parser and an optimizer written in PHP, but also with C version for desktop executable. There's also a hosted version. It's probably the most advanced optimizer in the list, being a parser it has a deeper understanding of the structure of the styleshets
• HTML_CSS from PEAR - not exactly an optimizer but more of a general purpose library for creating and updating stylesheets server-side in PHP. It can be used as a minifier, by simply reading, then printing the parsed structure, which strips spaces and comments as a side effect.

Trying to get an average figure of the potential benefits, I ran these tools on all stylesheets from csszengarden.com, collected simply like:

<?php
$urlt = "http://csszengarden.com/%s/%s.css";
for ($i = 1; $i < 214; $i++) {
  $id = str_pad($i, 3, "0", STR_PAD_LEFT);
  $url = sprintf($urlt, $id, $id);
  file_put_contents("$id.css", file_get_contents($url));
}
?>

3 files gave a 404, so I ran the tools above on the rest 210 files. CSSTidy ran twice - once with its safest settings (which even keep comments in) and then with the most aggressive. The "safe" way to use CSSTidy is like so:

<?php
// dependencies, instance
include 'class.csstidy.php';
$css = new csstidy();
 
// options
$css->set_cfg('preserve_css',true);
$css->load_template('high_compression');
 
// parse
$css->parse($source_css_code);
 
// result
$min = $css->print->plain();
?>

The aggressive minification is the same only without setting the preserve_css option.

Running Minify is simple:

<?php
// dependencies, instance
require 'CSS.php';
$minifier = new Minify_CSS();
 
// minify in one shot
$min = $minifier->minify($source_css_string_or_url);

As for PEAR::HTML_CSS, since it's not a minifier, you only need to parse the input and print the output.

<?php
require 'HTML/CSS.php';
 
$options = array(
    'xhtml' => false,
    'tab' => 0,
    'oneline' => true,
    'groupsfirst' => false,
    'allowduplicates' => true,
);
 
$css = new HTML_CSS($options);
$css->parseFile($input_filename);
$css->toFile($output_filename);
// ... or alternatively if you want the result as a string
// $minified = $css->toString();

So I ran those tools on the CSSZenGarden 200+ files and the full table of results is here, below are just the averages:

Again, the numbers are percentage of the original, so smaller is better. As you can see, on average gzip alone gives you 70% size reduction. The minification is not so successful as with JavaScript. Here even the best tool cannot reach 40% reduction (for JS it was usually over 50%). But nevertheless, gzip+minification on average gives you a reduction of 80% or more. Verdict: do it!

An important note here is that in CSS we deal with a lot of hacks. Since the browsers have parsing issues (which is what hacks often exploit), what about a poor minifier? How safe are the minifiers? Well, that's a subject for a separate study, but I know I can at least trust the YUICompressor, after all it's used by hundreds of Yahoo! developers daily and probably thousands non-Yahoos around the world. PEAR's HTML_CSS library also looks pretty safe because it has a simple parser that seems to tolerate all kinds of hacks. CSSTidy also claims to tolerate a lot of hacks, but given that the last version is two years old (maybe new hacks have surfaced meanwhile) and the fact that it's the most intelligent optimizer (knows about values, colors and so on) it should be approached with care.

204

Let's wrap up this lengthy posting with an honorable mention of the 204 No Content response (blogged before). It's the world's smallest componet, the one that has no body and a Content-Length of 0.

Often people use 1x1 GIFs for logging and tracking purposes and other types of requests that don't need a response. If you do this, you can return a 204 status code and no response body, only headers. Look no further that Google search results with your HTTP sniffer ON to see examples of 204 responses.

The way to send a 204 response from PHP is simply:

header("HTTP/1.0 204 No Content");

A 204 response saves just a little bit but, hey, every little bit helps.

And remember the mantra: every extra bit is a disservice to the user

Thank you for reading!

Stay tuned for the next article continuing the topic of reducing the component sizes as much as possible.

2010 update:
Lo, the Web Performance Advent Calendar hath moved

Dec 2 This is the second in the series of performance articles as part of my 2009 performance advent calendar experiment. Stay tuned for the next articles.

While theoretically you can speed up your site by just blindly following advice from this blog and other sources, it is much better to understand what's going on on the page and what you're dealing with. That's where the tools come in. Some tools give you insight about the network activities going on between the server and the browser (packet sniffers), some help you benchmark code execution on the client (profilers), some even give recommendations specific to performance improvements. You should aim at mastering as many of the tools as possible, because there's no single one that is The Tool. And that's not a bad thing, it's normal, because performance optimization is a multi-discipline activity touching a lot of different aspects of the the development process.

YSlow

(Full disclaimer: I helped with YSlow development and I was the architect of YSlow 2.0 so this fact may or may not have something to do with why YSlow is the first in the list. Plus, this is an unordered list.)

YSlow is an extension to Firebug, that:

• inspects the DOM
• hooks into the Net panel to listen to network activity and discover components that are not part of the DOM

Then the tool looks at the page and the components and tries to figure out how closely they match with Yahoo!'s performance best practices. Then you're given a list of findings with some advice and links for more information on how to improve.

PageSpeed

PageSpeed inspects the page and the components and checks it against conformance with Google's performance best practices.

In addition to that, PageSpeed has some quite advanced features like the Activity Panel which shows more detailed information on the page's, well, activity - such as the browser paint events, javascript parsing, execution, DNS lookups and so on. PageSpeed also tells you how many (and which) JavaScript functions were never called before onload so you can take some hints to lazy-load some of the JavaScript payload (after analysing, of course that the code is not needed in other browsers or other page use cases). Same with CSS - PageSpeed gives you a list of unused selectors so you can check whether you have leftovers from previous versions of the page.

MSFast

MSFast (from MySpace) inspects the page and helps answer many questions left open by YSlow and PageSpeed, such as:

• What's going on with IE?
• What's the memory and CPU footprint of your code?
• How does the page looks like (as in screenshots) while it's being loaded so you can see what the people with slower connections have to experience

PageTest

AOL's PageTest is an IE plugin but also a hosted service which is a great way to show your boss/client performance details without inconveniencing them with challenging download and installation activities. PageTest gives you a waterfall view of the page load and a checklist of things to improve, plus some screenshots of interesting moments during load and even a video - an excellent view of how the page looks like in slow speeds. The hosted service can show you the dial-up experience in 4 different places in the world.

DynaTrace's Ajax Edition

Dynatrace Ajax is a very detailed lower level tool that not only shows the waterfall of components downloads but also the rendering time, CPU consumption, JavaScript parsing and execution. The screenshot above is just the tip of the iceberg of the tool's plethora of views and insights. It's highly recommended. (free, registrationware)

Packet sniffers

A good packet sniffer is indispensable for inspecting the HTTP traffic and figuring out how the page loads and what the request/responses and their headers look like. Here's a list of recommended sniffers, each with something good on top of the others:

• IBM PageDetailer - a mature tool, somewhat simple which makes it a good start, requires registration to download
• Fiddler - very powerful, extensible
• HTTPWatch - (paid, but with a free version) integrates into the browser (both IE and FF) as a panel - very convenient to use. Extensible.
• Microsoft Visual Round Trip analyser (and an excellent writeup)- goes even lower into the packet level of the requests and draws a different view of the waterfall, one that visualizes the TCP packets and the TCP slow start. It also gives recommendations for performance improvements. Built on top of NetMon (Microsoft Network Monitor) to present the data in a more useful and friendly way.
• Charles proxy - the only non-windows tool in the list is an excellent packet sniffer for Mac

Time for a little rant - a more detailed view into the HTTP chunks is something that I think is important (will blog about it as part of this series) and missing from the current tools. HTTPWatch is the only tool that at least tells you the number of chunks and Fiddler prompts you to de-chunk HTTP responses when inspecting the body, which gives you a hint that the response was chunked. I hope to see more in that area, hopefully soon.

Thanks for reading

That concludes day 2 of the performance advent calendar. Hope you'll have fun installing and playing with new toys!

Did I miss a tool that should've been in the list? Let me know.

This posts is one of those "note to self" kinda posts. I just finished installing PHP and Apache on my Mac OS 10.5.6 and though I should document the experience should I (or you) need to do it again.

It could already be there

The default OS install came with goodies like ruby and php already there. So I could use php on the command line already. But it wasn't "hooked" to Apache for proper web development.

Also turns out Mac comes with some version of Apache, looks like it's disabled by default, but if it isn't, disable it from System Preferences / Sharing / Web Sharing.

Now let's start fresh with PHP5 and Apache 2, ignoring the PHP that's already there.

Mac ports prerequisite

Mac ports makes installing many software packages a breeze on the Mac. If you don't have it already, do set up Mac ports first.

Ready? Set? Go

1. Log into the mac ports prompt:

   $ sudo port
   Password:
   MacPorts 1.700
   Entering interactive mode... ("help" for help, "quit" to quit)
   [Users/stoyan] >

2. inside the Mac ports prompt simply do:

   [Users/stoyan] > install php5

3. This is it! Give it a bit of time to pull out all dependencies, including Apache2. The rest is just some configuration...
4. Start Apache and make it start when you power on the computer next time:

   $ sudo launchctl load -w /Library/LaunchDaemons/org.macports.apache2.plist

5. Test that Apache runs fine by pointing your browser to http://localhost/. You should see a page that says "It works!"
6. Tell Apache that PHP exists:

   $ sudo /opt/local/apache2/bin/apxs -a -e -n "php5" libphp5.so
   [activating module `php5' in /opt/local/apache2/conf/httpd.conf]
   

7. Create a php.ini file (PHP configuration) by copying the default .ini

   sudo cp /opt/local/etc/php.ini-dist /opt/local/etc/php.ini

8. If you need Apache stuff, like config files, error/access logs, htdocs... look around /opt/local/apache2. The web root for example is /opt/local/apache2/htdocs. I found it kinda convoluted so decided to move the web root to my home directory. So next two steps are optional.
9. I'll store all web apps and pages and scripts in a directory called /localhost in my home directory:

   mkdir ~/localhost

10. Time to edit the Apache config to tell it about the new location of the web root. Open httpd.conf

    sudo vi /opt/local/apache2/conf/httpd.conf

    Search for "DocumentRoot" and replace the current value /opt/local... with /Users/[your username]/localhost.
    The result would be like:

    DocumentRoot "/Users/stoyan/localhost"

    Then do the same further down in the config where it says <Directory..., so it should read:

    #
    # This should be changed to whatever you set DocumentRoot to.
    #
    <Directory "/Users/stoyan/localhost">
    

11. Apache also need to know that files that end with .php will be handled by the php module, so edit httpd.conf (the same one from the previous step). Search for "php" and you'll find:

    LoadModule php5_module        modules/libphp5.so

    Add one more line so it looks like:

    LoadModule php5_module        modules/libphp5.so
    AddHandler application/x-httpd-php .php

12. restart Apache (see below) and you're all done

start/stop/restart Apache

Here's how you start/stop/restart Apache:

• $ sudo /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper start

• $ sudo /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper stop

• $ sudo /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper restart

Or to stop and disable starting up every time you power on:

$ sudo launchctl unload -w /Library/LaunchDaemons/org.macports.apache2.plist

Verify that all is good

You can check that all is good by creating a PHP info script.

echo "<?php phpinfo(); ?>" > ~/localhost/test.php

Now point your browser to http://localhost/test.php. It should give you the php info page (a looong page of PHP-related information)

UPDATE: Ryan Grove has a better script to display the JSLint results. So basically follow the instructions here until you get to Step 2, point 5 (where you write the command to run JSLint). Then head over to Ryan's blog post to get the better script.

JSLint is an indispensable tool if you're serious about your JavaScript code quality. You can run it online for curiosity but for real development it has to be part of your coding environment and just a click/keystroke away.

While on PC I integrated JSLint with my text editor of choice - TextPad - and shared here. Now, ladies and gentlemen...[drum roll] I give you...[bzfghgang!] JSLint on the Mac!

Prerequisite: get Rhino running on your OSX

Don't worry, it's pretty straightforward, described here

Step 1: get JSLint

The Rhino version of JSLint is here. It's just one JS file. Find an appropriate place to copy it, I think ~/Library/JSLint is as good as any.

$ mkdir ~/Library/JSLint
$ curl http://jslint.com/rhino/jslint.js > ~/Library/JSLint/jslint.js

Test how it works from the command line:

$ java org.mozilla.javascript.tools.shell.Main ~/Library/JSLint/jslint.js myjavascript.js

Step 2: integrate with TextMate

TextMate extensions work their magic through the so called bundles. Here's what you do.

1. Select menu: Bundles / Bundle Editor / Edit Commands...
2. In the list of commands, expand JavaScript
3. Click the + sign found under the list, select New Command
4. type the name "jslint"
5. Replace the contents of the Command(s) text field with

   java org.mozilla.javascript.tools.shell.Main ~/Library/JSLint/jslint.js "$TM_FILEPATH"

6. In the Input: dropdown select "Entire Document", in the Output: "Show as Tool Tip" or "Show as HTML"
7. In Activation, click on Key Equivalent and then select a key combination you like, for example Command + L (L for Lint)
8. And this is it, refer to the screenshot below to compare with what you just did. Close the bundle editor window and you're done

Now test your new shiny tool. Open a javascript file and press Command+L. Here's a sample output:

And after fixing the missing semi-colon:

To quote http://www.mozilla.org/rhino/:

Rhino is an open-source implementation of JavaScript written entirely in Java. It is typically embedded into Java applications to provide scripting to end users.

Rhino allows you to use JavaScript:

• on the server-side, so you can ditch RoR, Perl, PH... well, keep PHP ... in favor of JavaScript
• on the command line, so you can shell scripts

Let's see how you can install Rhino on OSX.

Step 1 - download and unzip

Download the binary from the Rhino site and unzip to a temporary directory, say /tmp. On the command-line:

$ curl ftp://ftp.mozilla.org/pub/mozilla.org/js/rhino1_7R1.zip > /tmp/rhino.zip
$ cd /tmp
$ unzip rhino.zip

Now you have the file /tmp/rhino1_7R1/js.jar

Step 2: move js.jar where Java can find it

Your default Java install (comes "free" with OSX) will look for class libraries in a predefined directory ~/Library/Java/Extensions. This directory may not exists, so create it and move the js.jar there.

$ mkdir ~/Library/Java
$ mkdir ~/Library/Java/Extensions
$ mv /tmp/rhino1_7R1/js.jar ~/Library/Java/Extensions/

Step 3: Done! Now test it

That's all there is, your Rhino install is ready to use. To launch and test the Rhino shell try:

$ java org.mozilla.javascript.tools.shell.Main
Rhino 1.7 release 1 2008 03 06
js> print('hello!')
hello!
js> parseInt('123abc')
123
js> encodeURI('hola LA!')
hola%20LA!
js> for (var i = 0; i < 5; i++)
  > print('i is now ' + i)
i is now 0
i is now 1
i is now 2
i is now 3
i is now 4
js> quit()

Last example - create a script that reads the HTML source of my blog:

$ echo "print(readUrl('http://phpied.com'))" > read.js

now you have a script called read.js, let's run it:

$ java org.mozilla.javascript.tools.shell.Main read.js

Thanks for reading!

And happy JS-scripting!

ffmpeg is such an amazing tool, looks like it's for video what ImageMagick is for images. An all-powerful all-formats wicked cool command-line tool.

This blog post is an introduction to some of the MP3 capabilities of ffmpeg. I'll use ffmpeg to transform a blog post into a podcast-ready mp3 file. If you continue to read this longish post, here's what you can expect to see:

• using PHP DOM
• ffmpeg to convert to MP3
• ffmpeg to crop (slice) an MP3
• glue together several MP3s
• Mac's say command for TTS (text-to-speech)

PHP DOM to get some blog content

This blog's feed is at http://phpied.com/feed. Let's create a small PHP script to extract the title, date and content of the last blog post. We'll feed these to Mac's say command to read then aloud.

Location of the feed:

$file = 'http://phpied.com/feed/';

Load the feed into a DOM instance:

$dom = new DOMDocument;
$dom->loadXML(file_get_contents($file));

Access the node that contains the first item, i.e. the last blog post

$post = $dom->getElementsByTagName('item')->item(0);

The title and a friendly-formatted date:

$title = $post->getElementsByTagName('title')->item(0)->textContent;
$date = $post->getElementsByTagName('pubDate')->item(0)->textContent;
$date = strtotime($date);
$date = date('F jS, Y', $date);

Get the content:

$ns = 'http://purl.org/rss/1.0/modules/content/';
$content = $post->getElementsByTagNameNS($ns,'encoded')->item(0)->textContent;

Strip out HTML tags and entities:

$content = strip_tags($content);
$content = html_entity_decode($content);

(Since this content will be read aloud, HTML tags and entities will make no sense. Here we cound've done better job by doing something more special for list, using ALT tags to replace images and so on...)

echo $title, "\n\n", $date, "\n\n", $content;

OK, so now let's call the script from the command line and write the output to a file:

$ php feed.php > thepost.txt

Here's the result - thepost.txt

Using Mac's say for text-to-speech

You can make your Mac talk on the command line, like:

$ say test

and it will say the word "test"

The say command can also read text from text files and write to AIFF audio files. Let's read thepost.txt into an audio file.

$ say -f thepost.txt -o thepost.aiff

Since I'll make this look like it's a part of an ongoing series of podcasts, I'll add some music and I also need a greeting and goodbye spoken text. So:
$ say -o welcome.aiff Welcome to phpied.com podcast
$ say -o thatsallfolks.aiff That was all for today, join us next time on... phpied.com

OK, so now I have three AIFF files:

• thepost.aiff
• welcome.aiff
• thatsallfolks.aiff

Now I want to add some music before/after the podcast. I took four loops from Garage Band's library. Here they are:

• opener.mp3
• closer.mp3
• breaking-news.mp3
• squeeze-toy.mp3

Next?

Now I have a bunch of audio files. All I need to do is merge them, glue them together into one MP3. Glueing MP3 will be as easy as simply concatenating the files, using cat for example and them making a final pass through ffmpeg to correct dates and other meta data, so that the result looks like one single file, and not like a Frankenstein

In order for the concatenation to work, you only need to make sure all files are the same format, bitrate, etc.

Let's choose 22050 Hz, mono for the result. This means always add the options:

-ar 22050 -ac 1

to all calls to ffmpeg.

Let's get cracking.

ffmpeg to convert just about anything

The simplest use of ffmpeg is to convert from one file format to another, for example AVI to MPEG, WMV to FLV and what not. This is done like this for example:

$ ffmpeg -i input.avi output.flv

ffmpeg to get file information

It's useful to know what type of file we're dealing with, you can do this simply by omitting the output file:

$ ffmpeg -i input.avi

Let's check out one of the Garage Band loops:

$ ffmpeg -i opener.mp3 
FFmpeg version..... (more ffmpeg information)
Input #0, mp3, from 'opener.mp3':
  Duration: 00:00:12.6, start: 0.000000, bitrate: 191 kb/s
  Stream #0.0: Audio: mp3, 44100 Hz, stereo, 192 kb/s
Must supply at least one output file

Pretty good quality, more than I need. Plus, for some reason Garage Band added silence at the end of the files I exported, so let's cut it off.

ffmpeg to crop files

I want to remove trailing 5 seconds or so of each Garage Band loop. Here goes:

$ ffmpeg -i breaking-news.mp3 -ac 1 -ar 22050 -ss 0 -t 6 breaking-news-ok.mp3
$ ffmpeg -i opener.mp3 -ac 1 -ar 22050 -ss 0 -t 8 opener-ok.mp3
$ ffmpeg -i closer.mp3 -ac 1 -ar 22050 -ss 0 -t 33 closer-ok.mp3
$ ffmpeg -i squeeze-toy.mp3 -ac 1 -ar 22050 -ss 0 -t 2 squeeze-toy-ok.mp3

-i is the input file -ac is the number of channels (1 for mono) -ar is the rate, -ss is start, -t is length.

Now you can see how the meta information for the opener.mp3 has changed:

$ ffmpeg -i opener-ok.mp3 
Input #0, mp3, from 'opener-ok.mp3':
  Duration: 00:00:08.1, start: 0.000000, bitrate: 63 kb/s
  Stream #0.0: Audio: mp3, 22050 Hz, mono, 64 kb/s

Convert AIFF to MP3

Now let's convert the AIFF files from our TTS say command to MP3, keeping the same 22050 mono rate:

$ ffmpeg -i thepost.aiff -ac 1 -ar 22050 thepost.mp3
$ ffmpeg -i welcome.aiff -ac 1 -ar 22050 welcome.mp3
$ ffmpeg -i thatsallfolks.aiff -ac 1 -ar 22050 thatsallfolks.mp3

Here are the new MP3s:

• thepost.mp3
• welcome.mp3
• thatsallfolks.mp3

Glue the pieces with cat and ffmpeg

Now, last stage, let's glue all the pieces with cat which simply means append the next file at the end of the previous.

$ cat breaking-news-ok.mp3 welcome.mp3 opener-ok.mp3 thepost.mp3 
            squeeze-toy-ok.mp3 thatsallfolks.mp3 closer-ok.mp3 > pieces.together

Then make these pieces a proper MP3 file

$ ffmpeg -i pieces.together final.mp3

Well, that's all folks, here's the final result:
final.mp3

Since I switched to Mac I need to learn a bunch of stuff I was taking for granted for years. Recently I needed to upload a bigger file or two to my server and scp is not the tool for the job. So instead of starting a hunt for a decent free FTP client, I tried the good old command line.

Turns out it's much easier to FTP files from the command line than I would think.

Connecting to the server

$ ftp phpied.com
Connected to phpied.com.
220---------- Welcome to Pure-FTPd [TLS] ----------
220-You are user number 1 of 50 allowed.
220-Local time is now 02:17. Server port: 21.
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
Name (phpied.com:stoyan): myusername
331 User myusername OK. Password required
Password:

After supplying your credentials you're in and you have an inviting ftp> prompt:

230 OK. Current restricted directory is /
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

So what do you do here? You can list available commands with
ftp> help

You can navigate with cd (good old "change directory")

ftp> cd www/phpied.com/files
250 OK. Current directory is /www/phpied.com/files

You can list the contents of a remote directory with ls or dir.

And finally you can copy files from your desktop to the remote machine. You don't copy files, you PUT them. You're rewarded with a nice progress indicator.

ftp> put src.zip 
local: src.zip remote: src.zip
229 Entering Extended Passive Mode (|||59237|)
150 Opening BINARY mode data connection for src.zip
 10% |***                                  |   511 KB   64.00 KB/s    01:09 ETA

later...

ftp> put src.zip 
local: src.zip remote: src.zip
229 Entering Extended Passive Mode (|||59237|)
150 Opening BINARY mode data connection for src.zip
 93% |**********************************   |  4607 KB   45.62 KB/s    00:07 ETA

finally...

ftp> put src.zip 
local: src.zip remote: src.zip
229 Entering Extended Passive Mode (|||59237|)
150 Opening BINARY mode data connection for src.zip
100% |*************************************|  4938 KB   44.73 KB/s    00:00 ETA
226 Transfer complete.
5056569 bytes sent in 01:53 (43.55 KB/s)

And then, bye-bye:

ftp> quit
221 Goodbye.

JPEGtran is cool because it lets you optimize JPEG images losslessly by:

1. Stripping meta data (meta is sometimes bulky and useless for web display)
2. Optimizing Huffman tables or
3. Convert a JPEG to progressive encoding

From my experience 1 is more important than 2 or 3 and 3 gives better results than 2 for images over 10K

Installation

I never had to install jpegtran before because all unix/linux machines I've touched already have it. And on windows you just copy a binary somewhere in your path.

Well, I got this MacBook now and it doesn't have jpegtran so had to figure it out myself. Here's how you can do it, worked for me on Mac OS should work on any unix/linux too.

BTW, jpegtran is part of a package of few tools known as libjpeg, so you'll be installing a few programs not only jpegtran.

1. Get the source code from here. It's the file called jpegsrc.v6b.tar.gz. Using cURL you can download like:
   curl http://www.ijg.org/files/jpegsrc.v6b.tar.gz > /tmp/libjpeg.tar.gz
2. Uncompres the package, e.g. tar -xzvf /tmp/libjpeg.tar.gz
3. go to the directory that contains the uncompressed code, e.g. cd /tmp/jpeg-6b
4. ./configure
5. sudo make install

Done.

You can test your shiny new set of tools like this and get some help information about the various options:

> jpegtran -h
> cjpeg -h
> djpeg -h
> rdjpgcom -h
> wrjpgcom -h

You also test by optimizing my book cover from Amazon like:

curl http://ecx.images-amazon.com/images/I/41ckBp3bBUL._SL500_AA240_.jpg > oojs.jpg
jpegtran -copy none -progressive oojs.jpg > oojs-opt.jpg

This gives you 10% smaller file with not a pixel of quality loss. Not bad, eh, for a minute of work, or less.

... but doesn't write PNG8 with alpha transparency, unfortunately.

This comment on the YUI blog got me all excited by the possibility of having another designers tool other than Fireworks that creates PNG8 (palette PNGs) with alpha transparency.

Overall Paint.NET is a very simple and friendly program (as a non-designer I'm often intimidated by Photoshop's plethora of features) and I can see how I can use it for my future (rare, I hope) design needs It's pretty impressive, free and from what I read, it has a dedicated community, plus a number of plugins (hmm, a smush.it plugin would be nice... one day).

I tried saving a PNG8 with alpha but unfortunately this feature is not (yet!) supported. I created an image with a sold red square and blue gradient that fades to 100% transparency. When saving as 8-bit (in order to create a palette PNG) the semi transparent pixels are all gone. It treats palette PNGs like GIFs (like Photoshop does).

Default (produces truecolor PNG):

With 8 bit option selected:

ExifTool looks like a very promising tool to fiddle with all sorts of JPEG metadata (needed for smush.it) but first I had to make sure I can install it on Dreamhost. Although installation didn't go as described on the exiftool site (since I don't have sudo access on Dreamhost), it's still installable and it's actually pretty easy.

1. ssh to your DH box and go to the directory that will contain the tool, in my case it's my home directory
2. Download the latest version of the code, e.g.
   wget http://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-7.59.tar.gz

3. Uncompress the code archive
   tar -xf Image-ExifTool-7.59.tar.gz

4. Delete the original archive
   rm Image-ExifTool-7.59.tar.gz

5. Rename the newly created Image-ExifTool-7.59 to something shorter to type, e.g. "et"
   mv Image-ExifTool-7.59 et
6. All done! You can now run it from any directory and show help info like
   ~/et/exiftool - h

You can also test the new installation with some of the images found in the exiftool test directory, like so:

$ ~/et/exiftool ~/et/t/images/IPTC-XMP.jpg

ExifTool Version Number         : 7.59
File Name                       : IPTC-XMP.jpg
File Size                       : 20 kB
File Modification Date/Time     : 2005:12:31 13:05:50-08:00
File Type                       : JPEG
MIME Type                       : image/jpeg
JFIF Version                    : 1.02
Exif Byte Order                 : Little-endian (Intel, II)
Image Description               : A witty caption
Make                            : FUJIFILM
Camera Model Name               : FinePix2400Zoom
Orientation                     : Horizontal (normal)
X Resolution                    : 72
Y Resolution                    : 72
Resolution Unit                 : inches
Software                        : Adobe Photoshop 7.0
Modify Date                     : 2004:02:26 09:36:46
Artist                          : Phil Harvey
Y Cb Cr Positioning             : Co-sited
Copyright                       : Copyright 2004 Phil Harvey
F Number                        : 3.5
Exposure Program                : Program AE
ISO                             : 100
Exif Version                    : 0210
Date/Time Original              : 2001:05:19 18:36:41
Create Date                     : 2001:05:19 18:36:41
Components Configuration        : YCbCr
Compressed Bits Per Pixel       : 1.6
...
... [snip]
...
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:4:4 (1 1)
Aperture                        : 3.5
Image Size                      : 100x80
Shutter Speed                   : 1/64
Focal Length                    : 6.0 mm
Light Value                     : 9.6

Smush.it is getting more and more buzz all over the internets. Now there's even a song about it! Me and Nicole are pretty busy answering email, but a little slow to document the thing, I though I should at least shed some light on how the tool works by using some of the presentations.

What the tool does can be summarized in these steps:

• Turn GIFs into PNG8. Results are reported only if there's a saving, the file name then becomes source.gif.png. Smush.it uses imagemagick to do the conversion and then pngcrush to crush the pngs
• Crush PNGs using pngcrush
• Strip JPEG metadata and make them progressive, using jpegtran
• GIF animations: use gifsicle to remove pixels that don't change from one frame to another

This has been documented here on developer.yahoo.com together with the command line tools and options.

So all the tool does is run the appropriate command for each file type. Easy as that All the tools mentioned are free open-source and available on all operating systems, including Windows.

Here are some presentations on slideshare that might explain things a little more:

1. High-perf web sites - PHP Quebec Montreal, March 2008
2. 7 mistakes in image optimization - O'Reilly's Velocity, SFO, June 2008
3. Ajax Experience, Boston, earlier this week. Draft 1, Draft 2. The final and shortest version is below. It doesn't say much but the talk was just 5 minutes and included a demo. It's weird how little you can say in 5 minutes, I mean just "hello, my name, ... welcome to blah, blah..." is 20 seconds

Happy smushing!

"Smush it! Smush it real good..." - hothardware.com

Since me and Nicole announced smush.it yesterday at Ajax Experience and thanks to Christian Heilmann posting it on Ajaxian and Yahoo Developer Network, this thing seems to have exploded! It's all over the blogosphere, twitter-sphere and every other sphere.

BTW, Chris never seizes to amaze - he posted the video on Ajaxian at 11:01 am and our talk was from 10:30 to 10:35 am

Last night, about 12 hours after the announcement, I checked the directory that stores the results and it had over 10 000 entries. There's one entry for every smush.it run, which means for example every page you smush using the FF extension. If you have 5 images on a page on average, this means over 50 000K smushed images in a day, nice!

We've received great feadback and great suggestions, keep'em coming. People are already making smush.it part of their dev/build process. People are already seeing response time performance improvement.

I'm really looking forward to releasing the official API, open source the code, the command-line version and all the fun stuff. We were just too busy trying to come up with something presentable for the Ajax conference.

The URL again: http://smushit.com

So, what's a smu?

Some time ago I posted the slides from my Velocity 2008 talk on Slideshare and since the slides have tables, all the tables came out with the wrong font and all messed up and misplaced. Instead of trying to figure out what the heck is wrong with Powerpoint and how to work around it, I thought I could probably convert the ppt to pdf, since Slideshare accepts pdfs too.

Now I'm sure there are probably several tools out there that do PDF-2-PPT, some maybe commercial, but I love imagemagick and the command line, so I had to try it myself. Here's how to do it if ever need to.

1. Export all slides from PPT into individual PNG images. In powerpont, go File | Save As..., then select PNG. When promted, say that you want all slides, not only the current one.
2. Rename the first 9 files. The previous step created a number of files like Slide1.PNG, SLide2.PNG....Slide56.PNG. Now you need to rename the first 9 to pad the numbers with a zero, like Slide01.PNG and so on, this will help with the sorting.
3. convert Slide*.PNG show.pdf

And that's that.

Now this has some drawbacks: the links are no longer links in the PDF, the file size is pretty big and the font is kinda bad, but hey, it worked for me.

HTTPFox is an interesting Firefox extension for monitoring the HTTP traffic, obviously inspired by the IE-only commercial HttpWatch.

HTTPFox shows some stuff that are missing from Firebug's Net Panel, such as requests for favicons and such. There's also a little search box that lets you filter the list of components. Pretty cool too is the ability to select and copy the list of components.

Funny today I played with a little export feature in the Net Panel, logged here, demo here.