phpied.com

getUserMedia() is a proposal for one of the most desired device APIs that can give HTML and JS access to the user's camera and microphone. It's already available in Chrome Canary in the form of navigator.webkitGetUserMedia(). It's also available in Opera without a prefix.

In part #1 of this miniseries I talked about manipulating pixels in canvas, let's do the same but this time using video data from your own webcam instead of a static image.

Demo

I've only tested in Chrome so if you want to see the demo, you need to:
1. install Canary
2. go to chrome://flags and Enable PeerConnection

If you don't want to go through the trouble, here's a snapshot of what you're missing: a little video element showing your video camera stream and 4 canvas elements where the image data is manipulated in some way.

And the demo

Hooking up the cam

Getting a video stream is pretty straightforward:

navigator.webkitGetUserMedia(
  {video: true},
  iCanHazStream,
  miserableFailure
);

You declare what type of media you want (video in this case) and provide success and failure callbacks. The browser then prompts the user to allow access:

If the user allows, your success callback is called.

Here's mine:

function iCanHazStream(stream) {
  var url = webkitURL.createObjectURL(stream);
  $('video').src = url;
  webkitRequestAnimationFrame(paintOnCanvas);
}

I have a <video id="video"> element on the page and I set its src to be the stream URL. (In Opera you assign the stream directly, not some made up URL. In webkit the URL ends up being something like http://www.phpied.com/files/canvas/blob:http%3A//www.phpied.com/c0d155b9-f4f8-4c4f-b2bc-694de68d74f2. Anyway, not terribly important)

So this is all you need to do in order to display the camera stream in a VIDEO element. Easy right?

Then I have another function paintOnCanvas() which I schedule with the new requestAnimationFrame hotness instead of old school setInterval()

Setting up the canvas

For the image manipulation I'm using the same CanvasImage() constructor from part #1.

During page load I initialize 4 canvas elements with a placeholder image.

var transformadores = [
  new CanvasImage($('canvas1'), 'color-bars.png'),
  new CanvasImage($('canvas2'), 'color-bars.png'),
  new CanvasImage($('canvas3'), 'color-bars.png'),
  new CanvasImage($('canvas4'), 'color-bars.png')
];

And I have 4 simple pixel manipulators like you saw already:

var manipuladors = [
  {
    name: 'negative',
    cb: function(r, g, b) {
      return [255 - r, 255 - g, 255 - b, 255];
    }
  },
  {
    name: 'max blue',
    cb: function(r, g, b) {
      return [r, g, 255, 255];
    }
  },
  {
    name: 'max red',
    cb: function(r, g, b) {
      return [255, g, b, 255];
    }
  },
  {
    name: 'noise',
    cb: function(r, g, b) {
      var rand =  (0.5 - Math.random()) * 50;
      return [r + rand, g + rand, b + rand, 255];
    },
    factor: '(0 - 500+)'
  }
];

Painting on the canvas

Finally, the paintOnCanvas() function. Here's what happens:

function paintOnCanvas() {
  var transformador = transformadores[0];
  transformador.context.drawImage(
    $('video'), 0, 0, 
    transformador.image.width, transformador.image.height
  );
  var data = transformador.getData();
  for (var i = 0; i < 4; i++) {
    transformador = transformadores[i];
    transformador.original = data;
    transformador.transform(manipuladors[i].cb);
  }
  webkitRequestAnimationFrame(paintOnCanvas);
}

First we need to take the image data from the video element and draw it on a canvas. Then read the image data from the canvas, play with it and paint it back. This seems like a hassle, there might be an easier way to get image data from the video or the stream without going stream-video-canvas, but I don't know it. In any event I only do it once for the first canvas, then remember this data and use it for all the 4 canvases.

It's surprisingly easy to draw a video data in canvas, just using context.drawImage(video_dom_element, ...). From there I read the image data into data and loop through the 4 canvas instances, transforming the image using one of the manipulators I have set up.

Once again, for your entertainment, the demo is right here.

In the previous post I talked about manipulating and changing pixels in an image (using JavaScript and canvas) one at a time. We took a single pixel and messed around with its R, G, B or A values.

This time let's look into taking account not only the single pixel but the pixels around it. This allows you to do all kinds of effects, the most popular being emboss, edge detection, blur and sharpen.

The demo page is here

Theory

The type of manipulation we'll consider is called image convolution using a 3x3 matrix. You take 9 pixels from the image: the current pixel you're changing and the 8 immediately around it.

In other words you want to change the RGB values for the pixel in the middle based on its own value and those around it.

Let's say we have some sample values (given in red for R, blue for B and green for G in this figure):

Remember this manipulation was called convolution matrix. So you need a matrix. Below is an example of one such matrix (used in the blur effect)

Now you take one of the channels, say R for example. You take each of the 9 R values you have and multiply it by the corresponding number in the matrix. Then sum the nine numbers.

1*1 + 2*2 + 5*1 + 11*2 + 10*4 + 20*2 + 1*1 + 10*2 + 1*1 =
 1  +  4  + 5   +   22 +  40  +  40  +  1  +  20  +  1  =
                      134 

In addition to the matrix we also have a divisor and an offset, both optional. If there's no divisor (meaning it's 1, not 0), the result for Red we're looking for is 134. As you can see 134 is pretty far off from the original value of 10. But the blur effect has a divisor of 16. So the new value for red is 8.375

If the convolution asked for an offset, you add it to the end result.

Then you repeat the same for Green and Blue. You can do alpha if you want but for regular images it has constant 255 value so you'll do a lot of math and end up with 255.

You may have noticed that the divisor 16 is also the sum of the numbers in the matrix;

1 + 2 + 1 + 2 + 4 + 2 + 1 + 2 + 1 = 16

This way the result image is as bright as the original. If you have an unbalanced matrix you'll get a darker or a lighter image.

The offset is 0 most of the time, but not always. The emboss effect has offset 127 for example.

Demo matrices

My demo uses the most popular matrices out there. You can search the web for other matrices and play with them. None of them define a divisor because it's the sum of their elements, but the API I'll show you lets you use your custom divisor.

Without further ado, here are the matrices I used defined as an array of JavaScript objects:

var matrices = [
  {
    name: 'mean removal (sharpen)',
    data:
     [[-1, -1, -1],
      [-1,  9, -1],
      [-1, -1, -1]]
  },
  {
    name: 'sharpen',
    data:
     [[ 0, -2,  0],
      [-2, 11, -2],
      [ 0, -2,  0]]
  },
  {
    name: 'blur',
    data:
     [[ 1,  2,  1],
      [ 2,  4,  2],
      [ 1,  2,  1]]
  },
  {
    name: 'emboss',
    data:
     [[ 2,  0,  0],
      [ 0, -1,  0],
      [ 0,  0, -1]],
    offset: 127,
  },
  {
    name: 'emboss subtle',
    data:
     [[ 1,  1, -1],
      [ 1,  3, -1],
      [ 1, -1, -1]],
  },
  {
    name: 'edge detect',
    data:
     [[ 1,  1,  1],
      [ 1, -7,  1],
      [ 1,  1,  1]],
  },
  {
    name: 'edge detect 2',
    data:
     [[-5,  0,  0],
      [ 0,  0,  0],
      [ 0,  0,  5]],
  }
];

Results

Original

Blur

Sharpen

Edge detect

Edge 2

Emboss

Emboss (subtle)

Mean removal (sharpen a lot)

The API

The API is the same as in the previous post, same constructor and all, just adding a new method called convolve(). This is where the magic happens.

You use this method like so:

transformador.convolve([
  [1,2,1],
  [2,4,2],
  [1,2,1]
], 16, 0);

Again, 16 is optional as the method will figure it out if you omit and offset is optional too. Actually you can go to the demo and play in the console to see what happens with a different divisor, e.g.

transformador.convolve([[1,2,1],[2,4,2],[1,2,1]], 10);

or

transformador.convolve([[1,2,1],[2,4,2],[1,2,1]], 20);

convolve()

Some comments on how convolve() was implemented in this demo.

The big picture:

CanvasImage.prototype.convolve = function(matrix, divisor, offset) {
  // ...
};

Handle arguments: flat matrix is easier to work with and figure out the divisor if missing. How 'bout that array reduce, eh? ES5 ftw.

  var m = [].concat(matrix[0], matrix[1], matrix[2]); // flatten
  if (!divisor) {
    divisor = m.reduce(function(a, b) {return a + b;}) || 1; // sum
  }

Some vars more or less the same as the last time in the transform() method:

  var olddata = this.original;
  var oldpx = olddata.data;
  var newdata = this.context.createImageData(olddata);
  var newpx = newdata.data
  var len = newpx.length;
  var res = 0;
  var w = this.image.width;

Then a loop through all the image data, filter out every 4th element (because we ignore Alpha channel) and write the new image data to the canvas.

  for (var i = 0; i < len; i++) {
    if ((i + 1) % 4 === 0) {
      newpx[i] = oldpx[i];
      continue;
    }
 
    // 
    // magic...
    //
  }
  this.setData(newdata);

Remember that canvas image data is one long array where 0 is R for pixel #1, 1 is B, 2 is G, 3 is Alpha, 4 is R for pixel #2 and so on. This is different than more other code examples you'll in different languages where there are two loops in order to touch every pixel: one from 0 to width and an inner one from 0 to height.

And finally, the "magic" part:

    res = 0;
    var these = [
      oldpx[i - w * 4 - 4] || oldpx[i],
      oldpx[i - w * 4]     || oldpx[i],
      oldpx[i - w * 4 + 4] || oldpx[i],
      oldpx[i - 4]         || oldpx[i],
      oldpx[i],
      oldpx[i + 4]         || oldpx[i],
      oldpx[i + w * 4 - 4] || oldpx[i],
      oldpx[i + w * 4]     || oldpx[i],
      oldpx[i + w * 4 + 4] || oldpx[i]
    ];
    for (var j = 0; j < 9; j++) {
      res += these[j] * m[j];
    }
    res /= divisor;
    if (offset) {
      res += offset;
    }
    newpx[i] = res;

these are the pixels we want to inspect. oldpx[i] is the one in the middle which we're changing to newpx[i]. Also note how we default all pixels to oldpx[i]. This is to deal with the boundary pixels: tho top and bottom rows of pixels and the left and right columns. Because the pixel in position 0x0 has no pixels above it or to the left. Then we loop through these and multiply by the corresponding value in the matrix. Finally divide and offset, if required.

Thanks!

Thanks for reading, and now go play with the demo in the console. An easy template to start is:

transformador.convolve([[1,0,0],[0,0,0],[0,0,-1]], 1, 127); 

If you want to apply convolutions on top of each other, you can reset the original image data to the current.

transformador.original = transformador.getData();

Done it before with PHP, but now that JavaScript is all-powerful, let's see how we can manipulate images in an HTML <canvas>.

The demo is here.

Pixel manipulation

The simplest way to fiddle with image data is to take each pixel and change the value of one or more of its channels: red, green, blue and alpha (transparency), also known as R, G, B and A for short. The results may vary from mostly useless to "wow, that sepia was easy, I'm totally doing it for all my thumbnails!"

A trivial example is to just swap some values. E.g. take the amount of B and assign it to G.

rgb(100, 50, 30, 255) becomes
rgb(100, 30, 50, 255)

Callbacks

For the purpose of this demo I've isolated the part that deals with the canvas API (loading an image file, reading and writing pixels) from the actual pixel manipulation. The manipulation itself is in the form of simple callback functions.

The example above (B to G and G to B) is written as

function (r, g, b) {
  return [r, b, g, 255];
}

Here we ignore alpha, because we don't need it and hardcode it to 255. Here's the result:

from the original:

Let's say you want to touch alpha and make an image partially transparent. Here's the callback:

function (r, g, b, a, factor) {
  return [r, g, b, factor];
}

Here we let the user specify a factor in other words how transparent should the image be. We use this as a value for the alpha channel.

The result of calling this with factor of 111:

And the most complicated form of callback is taking into account where this pixel is in the overall image. Here's a gradient:

function (r, g, b, a, factor, i) {
  var total = this.original.data.length;  
  return [r, g, b, factor + 255 * (total - i) / total];
}

And the result of calling this with factor of 111:

this refers to an object of our own design which you'll see in a bit. It keeps some info in case the callbacks need as is the case above.

The canvas part

For the canvas stuff I started by copying an old post to load an image into a canvas. This constructor was born:

function CanvasImage(canvas, src) {
  // load image in canvas
  var context = canvas.getContext('2d');
  var i = new Image();
  var that = this;
  i.onload = function(){
    canvas.width = i.width;
    canvas.height = i.height;
    context.drawImage(i, 0, 0, i.width, i.height);
 
    // remember the original pixels
    that.original = that.getData();
  };
  i.src = src;
  
  // cache these
  this.context = context;
  this.image = i;
}

You use this by passing a reference to a canvas element somewhere on the page and a URL to an image. The image has to be on the same domain for the image data part to work.

var transformador = new CanvasImage(
  $('canvas'),
  '/wp-content/uploads/2008/05/zlati-nathalie.jpg'
);

The constructor creates a new Image object and once it's loaded it draws it into the canvas. Then it remembers some stuff for later such as the canvas context, the image object and the original image data. `this` is the same `this` that pixel manipulator callbacks have access to.

Then you have three simple methods to get, set and reset the image data into the canvas:

CanvasImage.prototype.getData = function() {
  return this.context.getImageData(0, 0, this.image.width, this.image.height);
};
 
CanvasImage.prototype.setData = function(data) {
  return this.context.putImageData(data, 0, 0);
};
 
CanvasImage.prototype.reset = function() {
  this.setData(this.original);
}

The "brains" of the whole thing is the transform() method. It takes the pixel manipulator callback and a factor which is essentially a configuration setting for the manipulator. Then it loops through all pixels, passes olddata rgba channel values to the callback and uses the return values from the callback as newdata. At the end the new data is written back to canvas.

CanvasImage.prototype.transform = function(fn, factor) {
  var olddata = this.original;
  var oldpx = olddata.data;
  var newdata = this.context.createImageData(olddata);
  var newpx = newdata.data
  var res = [];
  var len = newpx.length;
  for (var i = 0; i < len; i += 4) {
   res = fn.call(this, oldpx[i], oldpx[i+1], oldpx[i+2], oldpx[i+3], factor, i);
   newpx[i]   = res[0]; // r
   newpx[i+1] = res[1]; // g
   newpx[i+2] = res[2]; // b
   newpx[i+3] = res[3]; // a
  }
  this.setData(newdata);
};

Pretty simple, right? The only confusing part could be the i += 4 increment in the loop. See, the data returned by canvas' getImageData().data is an array with 4 elements for each pixel.

Imagine an image with only two pixels - red and blue and no transparency. The data for this image is

[
  255, 0, 0, 255,
  0, 0, 255, 255
]

And this is it. You use the objects created by ImageCanvas constructor like so:

transformador.transform(function(r, g, b, a, factor, i) {
  // image magic here
  return [r, g, b, a];
}, factor);

Feel free to play in the console with this on the demo page

Callbacks (contd.)

The rest of the code for the demo is just the setting up pixel-fiddling callbacks and creating some UI to use them. Let's check out a few.

Greyscale

Grey is equal amounts of r, g, b. So you can simply average the three values to get a single value:

var agv = (r + g + b) / 3;

This is good enough, but there's a secret formula that does a better job for humans, because it takes into account our species' sensitivity to the different channels. The result:

function(r, g, b) {
  var avg = 0.3  * r + 0.59 * g + 0.11 * b;
  return [avg, avg, avg, 255];
}

Sepia

A lazy sepia is to make a greyscale version and then add some color to it - equal amounts of rgb to each pixel. I add 100 to red and 50 to green and I like it, but you can play with different values.

function(r, g, b) {
  var avg = 0.3  * r + 0.59 * g + 0.11 * b;
  return [avg + 100, avg + 50, avg, 255];
}

There's a second sepia callback which is supposed to be better, but I don't like it as much.

Negative (invert)

Subtracting each channel from 255 gives you a negated image:

function(r, g, b) {
  return [255 - r, 255 - g, 255 - b, 255];
}

Noise

Noise is kind of fun, you take a random value between -factor and factor and add it to each channel:

function(r, g, b, a, factor) {
  var rand =  (0.5 - Math.random()) * factor;
  return [r + rand, g + rand, b + rand, 255];
}

Noise with factor of 55:

Your turn

I'm leaving some of these manipulator callbacks for your source exploration and then to your imagination. What can you think of?

Go nuts with those callbacks in the console! Your basic "template" again is:

transformador.transform(function(r, g, b, a, factor, i) {
  // image magic here...
  return [r, g, b, a];
});

Try e.g. - blue to 255. Or make it black and white (not greyscale, make each pixel either 0,0,0 or 255,255,255)

Today our family went to the yearly photo session with the girls. We took one shot that can be looked normally, as well as upside down, so I was wondering can you flip an image using a canvas tag. Turns out, yes, you can and it's pretty easy.

» Demo is here.

How to load an image in a canvas tag?

Start unpretentiously with an empty canvas tag:

<canvas id="canvas"></canvas>

Now the javascript. Two variables to store a handle to the canvas element and the 2D context of the canvas:

var can = document.getElementById('canvas');
var ctx = can.getContext('2d');

Now let's load an image into the canvas. Using the new Image() constructor you can create an image object, then set its src property to point to the location of the image file. Then set an onload handler for the image which is an anonymous function to be called when the image is done loading. There you put the image inside the canvas using the drawImage() method of the canvas context.

var img = new Image();
img.onload = function(){
    can.width = img.width;
    can.height = img.height;
    ctx.drawImage(img, 0, 0, img.width, img.height);
}
img.src = 'zlati-nathalie.jpg';

You can also notice how the dimensions of the canvas are adjusted to match the dimensions of the image.

How to flip the image upside down

The canvas context provides a rotate() method. The rotation always happens around the top left corner of the image, so we first translate() the image to the bottom right. This way when the image is rotated, it fits back into the canvas. (There is also a one pixel correction, I have no idea why, just saw that the image wasn't flipping exactly otherwise). Assigning this functionality to the onclick:

can.onclick = function() {
    ctx.translate(img.width-1, img.height-1);
    ctx.rotate(Math.PI);
    ctx.drawImage(img, 0, 0, img.width, img.height);
};

C'est tout! Once again, the demo is here.

This is very flattering: Greg Houston took my script for DIY canvas pie and added tooltips and better colors logic. Here's the result, it's really nice, built with some MooTools. The tooltips are not supported in <canvas> but Creg used an image that overlays the pie and set the tooltips with an image map. Clever, isn't it?

I just used random colors for the pieces of the pie, but Greg knows better. Based on a tutorial by Jim Bumgardner, he was able to achieve much more pleasing, yet reliable results. Funny thing that Jim works in Yahoo! two floors above me (used to be even on the same floor).

Check the demo and the post, cool stuff, works in IE too!

UPDATE: Translation in Brazilian Portuguese here, thanks Maujor!

OK, so you have an HTML table. Let's turn it into a pie chart with a bit of javascript.

We'll be using the canvas tag, so the browser has to support it. For IE - well, you still have the table. That's why we'll call it progressive enhancement. Unobtrusive too. Here's a screenshot:

» The demo is here (refresh for new colors)

Here are the ingredients to the recipe:

1. One <canvas> tag
2. One <table> full of data
3. javascript to get the data from the table
4. javascript to plot the data on the canvas

One canvas tag

<canvas id="canvas" width="300" height="300"></canvas>

One table full of data

This is a bare bone unstyled old school table.

<table id="mydata">
    <tr>       <th>Lang</th><th>Value</th> </tr>
    <tr><td>JavaScript</td>  <td>100</td>  </tr>
    <tr><td>       CSS</td>  <td>200</td>  </tr>
    <tr><td>      HTML</td>  <td>300</td>  </tr>
    <tr><td>       PHP</td>  <td> 50</td>  </tr>
    <tr><td>     MySQL</td>  <td> 30</td>  </tr>
    <tr><td>    Apache</td>  <td> 10</td>  </tr>
    <tr><td>     Linux</td>  <td> 30</td>  </tr>
</table>

javascript to get the data from the table

First, some setup. Let's tell the script which is the ID of the data table, the ID of the canvas tag and which column contains the data:

// source data table and canvas tag
var data_table = document.getElementById('mydata');
var canvas = document.getElementById('canvas');
var td_index = 1; // which TD contains the data

Next, select all table rows, then loop through the rows, selecting all TDs. Add the data we need to a data array. While at it, run a total of the data in the column and also create an array of random colors. Paint each row with the selected color. (we'll see the actual getColor() in a bit.)

var tds, data = [], color, colors = [], value = 0, total = 0;
var trs = data_table.getElementsByTagName('tr'); // all TRs
for (var i = 0; i < trs.length; i++) {
    tds = trs[i].getElementsByTagName('td'); // all TDs
 
    if (tds.length === 0) continue; //  no TDs here, move on
 
    // get the value, update total
    value  = parseFloat(tds[td_index].innerHTML);
    data[data.length] = value;
    total += value;
 
    // random color
    color = getColor();
    colors[colors.length] = color; // save for later
    trs[i].style.backgroundColor = color; // color this TR
}

javascript to plot the data on the canvas

Time for the fun part, the drawing! First, we need to create a context object. Then figure out the raduis of the pie and the center, all based on the width/height pf the canvas tag:

// get canvas context, determine radius and center
var ctx = canvas.getContext('2d');
var canvas_size = [canvas.width, canvas.height];
var radius = Math.min(canvas_size[0], canvas_size[1]) / 2;
var center = [canvas_size[0]/2, canvas_size[1]/2];

Next, let's loop through data and paint pieces of the pie. To draw a piece of pie, you basically need to call these methods of the context object:

• beginPath() - to start the piece of the pie
• moveTo() - to set the pencil in the center
• arc() - draw a piece of a circle
• lineTo() - finish the circle piece with a line back to the center
• closePath() and fill() but set the fill color first.

Here's the actual code for this part, hopefully the comments help:

var sofar = 0; // keep track of progress
// loop the data[]
for (var piece in data) {
 
    var thisvalue = data[piece] / total;
 
    ctx.beginPath();
    ctx.moveTo(center[0], center[1]); // center of the pie
    ctx.arc(  // draw next arc
        center[0],
        center[1],
        radius,
        Math.PI * (- 0.5 + 2 * sofar), // -0.5 sets set the start to be top
        Math.PI * (- 0.5 + 2 * (sofar + thisvalue)),
        false
    );
 
    ctx.lineTo(center[0], center[1]); // line back to the center
    ctx.closePath();
    ctx.fillStyle = colors[piece];    // color
    ctx.fill();
 
    sofar += thisvalue; // increment progress tracker
}

utility

Here's the function that gives a random color:

    // utility - generates random color
    function getColor() {
        var rgb = [];
        for (var i = 0; i < 3; i++) {
            rgb[i] = Math.round(100 * Math.random() + 155) ; // [155-255] = lighter colors
        }
        return 'rgb(' + rgb.join(',') + ')';
    }

C'est tout! Enjoy your pie

UPDATE: Comment by Zoltan below, if you use Explorer Canvas, you can make this work in IE with only this:
<!--[if IE]><script type="text/javascript"
src="/path/to/excanvas.js"></script><![endif]-->