Canvas pixels #2: convolution matrix

June 11th, 2012. Tagged: (x)HTML(5), canvas, images, JavaScript

In the previous post I talked about manipulating and changing pixels in an image (using JavaScript and canvas) one at a time. We took a single pixel and messed around with its R, G, B or A values.

This time let's look into taking account not only the single pixel but the pixels around it. This allows you to do all kinds of effects, the most popular being emboss, edge detection, blur and sharpen.

The demo page is here

Theory

The type of manipulation we'll consider is called image convolution using a 3x3 matrix. You take 9 pixels from the image: the current pixel you're changing and the 8 immediately around it.

In other words you want to change the RGB values for the pixel in the middle based on its own value and those around it.

Let's say we have some sample values (given in red for R, blue for B and green for G in this figure):

Remember this manipulation was called convolution matrix. So you need a matrix. Below is an example of one such matrix (used in the blur effect)

1,2,1,2,4,2,1,2,

Now you take one of the channels, say R for example. You take each of the 9 R values you have and multiply it by the corresponding number in the matrix. Then sum the nine numbers.

1,2,1,2,4,2,1,2,

1*1 + 2*2 + 5*1 + 11*2 + 10*4 + 20*2 + 1*1 + 10*2 + 1*1 =
 1  +  4  + 5   +   22 +  40  +  40  +  1  +  20  +  1  =
                      134 

In addition to the matrix we also have a divisor and an offset, both optional. If there's no divisor (meaning it's 1, not 0), the result for Red we're looking for is 134. As you can see 134 is pretty far off from the original value of 10. But the blur effect has a divisor of 16. So the new value for red is 8.375

If the convolution asked for an offset, you add it to the end result.

Then you repeat the same for Green and Blue. You can do alpha if you want but for regular images it has constant 255 value so you'll do a lot of math and end up with 255.

You may have noticed that the divisor 16 is also the sum of the numbers in the matrix;

1 + 2 + 1 + 2 + 4 + 2 + 1 + 2 + 1 = 16

This way the result image is as bright as the original. If you have an unbalanced matrix you'll get a darker or a lighter image.

The offset is 0 most of the time, but not always. The emboss effect has offset 127 for example.

Demo matrices

My demo uses the most popular matrices out there. You can search the web for other matrices and play with them. None of them define a divisor because it's the sum of their elements, but the API I'll show you lets you use your custom divisor.

Without further ado, here are the matrices I used defined as an array of JavaScript objects:

var matrices = [
  {
    name: 'mean removal (sharpen)',
    data:
     [[-1, -1, -1],
      [-1,  9, -1],
      [-1, -1, -1]]
  },
  {
    name: 'sharpen',
    data:
     [[ 0, -2,  0],
      [-2, 11, -2],
      [ 0, -2,  0]]
  },
  {
    name: 'blur',
    data:
     [[ 1,  2,  1],
      [ 2,  4,  2],
      [ 1,  2,  1]]
  },
  {
    name: 'emboss',
    data:
     [[ 2,  0,  0],
      [ 0, -1,  0],
      [ 0,  0, -1]],
    offset: 127,
  },
  {
    name: 'emboss subtle',
    data:
     [[ 1,  1, -1],
      [ 1,  3, -1],
      [ 1, -1, -1]],
  },
  {
    name: 'edge detect',
    data:
     [[ 1,  1,  1],
      [ 1, -7,  1],
      [ 1,  1,  1]],
  },
  {
    name: 'edge detect 2',
    data:
     [[-5,  0,  0],
      [ 0,  0,  0],
      [ 0,  0,  5]],
  }
];

Results

Original

Blur

Sharpen

Edge detect

Edge 2

Emboss

Emboss (subtle)

Mean removal (sharpen a lot)

The API

The API is the same as in the previous post, same constructor and all, just adding a new method called convolve(). This is where the magic happens.

You use this method like so:

transformador.convolve([
  [1,2,1],
  [2,4,2],
  [1,2,1]
], 16, 0);

Again, 16 is optional as the method will figure it out if you omit and offset is optional too. Actually you can go to the demo and play in the console to see what happens with a different divisor, e.g.

transformador.convolve([[1,2,1],[2,4,2],[1,2,1]], 10);

or

transformador.convolve([[1,2,1],[2,4,2],[1,2,1]], 20);

convolve()

Some comments on how convolve() was implemented in this demo.

The big picture:

CanvasImage.prototype.convolve = function(matrix, divisor, offset) {
  // ...
};

Handle arguments: flat matrix is easier to work with and figure out the divisor if missing. How 'bout that array reduce, eh? ES5 ftw.

  var m = [].concat(matrix[0], matrix[1], matrix[2]); // flatten
  if (!divisor) {
    divisor = m.reduce(function(a, b) {return a + b;}) || 1; // sum
  }

Some vars more or less the same as the last time in the transform() method:

  var olddata = this.original;
  var oldpx = olddata.data;
  var newdata = this.context.createImageData(olddata);
  var newpx = newdata.data
  var len = newpx.length;
  var res = 0;
  var w = this.image.width;

Then a loop through all the image data, filter out every 4th element (because we ignore Alpha channel) and write the new image data to the canvas.

  for (var i = 0; i < len; i++) {
    if ((i + 1) % 4 === 0) {
      newpx[i] = oldpx[i];
      continue;
    }
 
    // 
    // magic...
    //
  }
  this.setData(newdata);

Remember that canvas image data is one long array where 0 is R for pixel #1, 1 is B, 2 is G, 3 is Alpha, 4 is R for pixel #2 and so on. This is different than more other code examples you'll in different languages where there are two loops in order to touch every pixel: one from 0 to width and an inner one from 0 to height.

And finally, the "magic" part:

    res = 0;
    var these = [
      oldpx[i - w * 4 - 4] || oldpx[i],
      oldpx[i - w * 4]     || oldpx[i],
      oldpx[i - w * 4 + 4] || oldpx[i],
      oldpx[i - 4]         || oldpx[i],
      oldpx[i],
      oldpx[i + 4]         || oldpx[i],
      oldpx[i + w * 4 - 4] || oldpx[i],
      oldpx[i + w * 4]     || oldpx[i],
      oldpx[i + w * 4 + 4] || oldpx[i]
    ];
    for (var j = 0; j < 9; j++) {
      res += these[j] * m[j];
    }
    res /= divisor;
    if (offset) {
      res += offset;
    }
    newpx[i] = res;

these are the pixels we want to inspect. oldpx[i] is the one in the middle which we're changing to newpx[i]. Also note how we default all pixels to oldpx[i]. This is to deal with the boundary pixels: tho top and bottom rows of pixels and the left and right columns. Because the pixel in position 0x0 has no pixels above it or to the left. Then we loop through these and multiply by the corresponding value in the matrix. Finally divide and offset, if required.

Thanks!

Thanks for reading, and now go play with the demo in the console. An easy template to start is:

transformador.convolve([[1,0,0],[0,0,0],[0,0,-1]], 1, 127); 

If you want to apply convolutions on top of each other, you can reset the original image data to the current.

transformador.original = transformador.getData();

Tell your friends about this post: Facebook, Twitter, Google+

11 Responses

  1. Very nice introduction to convolutions and its applications in image manipulation. I have one minor nitpick though: convolve is the verb form of convolution. Convolute means something else entirely.

  2. thanks, didn’t know that, fixing…

  3. Nice article, thanks!

    I wrote a matrices library for JS a while back; probably overkill for something like this, but still could come in useful: https://github.com/callumacrae/MatrixJS

  4. is there somthing wrong in the “these” matrix?
    For example, px[i] is at the left edge,
    px[i - 4] is the px in row above px[i] and at the right side.

  5. [...] manipulation in canvas. Also, part 2 and part [...]

  6. You already know therefore significantly in terms of this topic, produced me personally imagine it from so many various angles. Its like men and women are not interested unless it’s one thing to accomplish with Lady gaga! Your individual stuffs excellent. Always care for it up!

  7. Struggling to get your car paid off? Are you mad about the last deal you
    got? Perhaps you’re looking for an automobile now, and you’re not sure what to do differently.
    You’re in the same boat with many other people. Keep reading to find out information regarding what to do next time you enter a dealership.

  8. Xu Meng While listening to the words of Liu Yiyi some unhappy. But think of the four hundred thousand take their hard-Friends of Stony Point without boondoggle feel pleased, and he is not making money bank today. The money hatred is turned brain add up year after year, is not easy, all sweat ah, even to the beloved woman also will be distressed.

  9. Yiyi shocked, his hands over his mouth and replaced later awakes to cover eyes: You, you quickly get dressed.

  10. This specific technology supplies plenty of lumination pertaining to safety measures at nighttime, enabling youngsters to study and also making it possible for family to soundly walk for you to outhouses at nighttime. The 1st of those Troubles seemed to be sorted out throughout November ’07 through a good oil industry outsider which employed his / her skills in this real industry to come up with this receiving resolution. John Davis, a good InnoCentive Solver from your Middle Usa, seemed to be granted $20,500 pertaining to his / her creative resolution..
    UGG ֩` ȥå http://brianbain.com/ugg-northlandtimeshares/northlandtimeshares-ugg60.html

  11. It’s very effortless to find out any topic on net as compared to textbooks, as I
    found this paragraph at this web site.

Leave a Reply