How to use machine learning in web pages:
a tutorial for clustering with ML.js

ML.js is a javascript library developed as a part of the MLweb project and aimed at providing machine learning capabilities for web applications.

This short tutorial will show how to build a small and simple example application of the clustering functions. Further information can be found in the main documentation.

The goal is to create a web page that classifies the uploaded images into a number of groups (see the live demo). This problem can be tackled with unsupervised learning techniques and here, we will apply the K-means method.

HTML part

First, we need to load the machine learning library, typically in the head:
<html>
<head>
	<meta charset="UTF-8">
	<script src="http://mlweb.loria.fr/ml.js"> </script>
</head>
Then, the document body merely contains a button for importing images, another one to launch the clustering, a select to choose the number of groups and a div to show the images:
<body>
	<h1>Very basic image clustering demo</h1>

	<input id="imagefile" type="file" onchange="loadImages();" multiple > 
	<button onclick="clusterimages();">Cluster images</button>
	in 
	<select id="selectK" onchange="clusterimages();">
		<option value="2">2</option>
		<option value="3">3</option>
		<option value="4">4</option>
		<option value="5">5</option>
		<option value="6">6</option>
		<option value="7">7</option>					
	</select>
	groups 
	
	<div id="images" style="position:relative;"></div>
	
	<canvas id="canvas" width="4" height="4" style="display: none;"></canvas>
</body>
</html>
We also included an invisible canvas which will be used to downsample the images.

Javascript code

The javascript code contains two functions: one to load the images and another one to cluster the images. Loading the images is out of the scope of this tutorial (many javascript tutorials will cover that). What is more interesting is how we transform the images.

Machine learning techniques require that each data instance in the data set is represented by a list of numbers of a given fixed length. This could be easily obtained since the javascript imageData corresponds to the values (between 0 and 255) of red, green, blue and alpha (rgba) at each pixel. However, the number of pixels may vary from one image to the other and thus we would end up having data instances represented by lists of numbers of various length. In order to work with images of various resolutions, we need to transform them.
Here, we use a very simple transformation: every image will be reduced to a thumbnail of 4x4 pixels and represented by the colors of these 16 pixels. This is a very rough approximation, but can be sufficient to cluster the images in different groups.

var data = new Array();	// will store the data set 

function loadImages( ) {
	for ( var i=0; i < imagefile.files.length ; i++) {
	
		var reader = new FileReader();
	
		// prepare callback for end of read:
		reader.onload = function (e) {
			var imgdata = new Uint8Array(  e.target.result );
	   		var blob = new Blob( [ imgdata ], { type: "image/jpeg" } );
	   		var url = URL.createObjectURL( blob ) ;

	   		var img = new Image(); 
	   		img.onload = function () {
	   		
	   			// Draw image on the canvas while reducing resolution to 4x4 pixels
				var ctx = canvas.getContext("2d");
   				ctx.drawImage(img, 0,0, img.width, img.height, 0,0, canvas.width, canvas.height);
				
				// Get the image data (rgba of each of the 16 pixels)
				var rgb = ctx.getImageData(0,0,canvas.width, canvas.height) ;
				
				// and store it in the data set:
				data.push( rgb.data );
				
				// Display the image  
		   		img.id = "img" + (data.length-1);
		   		img.style.width = "100px";
		   		images.appendChild(img); 
	   		};
	   		img.style.border = "10px solid white";
	   		img.src = url;	
		}
		reader.readAsArrayBuffer(imagefile.files[i]);	
	}
}

Note that reducing resolution to 4x4 pixels is easily done by drawing the image on a 4x4 canvas. The getImageData() function then returns an Array of length 64 containing the red, green, blue and alpha values of all the 16 pixels. The variable data is an Array of as many Arrays as uploaded images.

Next, we can ask ML.js to cluster the images with

	// Recover the desired number of groups
	var K = parseInt(selectK.value);

	// Transform the data to a suitable Matrix for ML.js functions
	var X = array2mat(data);
	
	// Cluster the data into K groups with the KMEANS method
	clustering = kmeans ( X, K );
The result is an object in which the field clustering.labels is an Array of numbers between 0 and K-1 such that clustering.labels[i] is the index of the group to which was assigned the i-th row of the matrix X (representing the i-th image). Thus, we can group the images in different columns as follows:
function clusterimages() {
	// Recover the desired number of groups
	var K = parseInt(selectK.value);

	// Transform the data to a suitable Matrix for ML.js functions
	var X = array2mat(data);
	
	// Cluster the data into K groups with the KMEANS method
	clustering = kmeans ( X, K );
	
	// Prepare columns
	var toppos = new Array(K);	// toppos[j] = top position of next image in group j
	for ( var j=0; j < K; j++)
		toppos[j] = 10;
		
	// Display the clustering result 
	for ( var i=0; i < X.length; i++) {
		
		// Each image of index i...
		var img = document.getElementById("img" + i);
		
		// is labeled by a group index between 0 and K-1:
		var group = clustering.labels[i]; 
		
		// So we position the image in the corresponding group column 
		img.style.position = "absolute";
		img.style.left = (10 + group * 130) + "px";
		img.style.top = toppos[group] + "px";		
		toppos[group] += 30 + img.height;
	}
}
The result of kmeans also has a field clustering.centers which is a Matrix storing the centers of the groups, i.e., the average image for each group. So, we could for instance color the image border with the average image color in its group, as is done in the complete source code.

Final notes

Here, the computations are not very demanding: the data set is small (unless you upload thousands of images) and the K-means method is one of the most simple ones. Thus, we could implement the machine learning part in the current scope, without blocking the browser for too long. However, in many situations, we should compute in a background lab by replacing

	clustering = kmeans ( X, K );	
	
	// Prepare columns
	...
by
var lab = new MLlab(); // this should be a global variable
...
	lab.load(X, "X");            // load data in the matrix X
	
	// Run kmeans in the lab and use the result in a callback
	lab.exec("kmeans(X, " + K + ")", function ( clustering ) {	
		
		// Prepare columns
		...
		
		} );