For a one paragraph intro to SVMs and the kernel-trick I wanted a a graphic that I’ve seen in a book (although forgotten where, perhaps in Pattern Classification?):
Simple idea — show some 2D data points that are not linearly separable, then transform to 3D somehow, and show that they are. I found nothing on google (at least nothing that was high enough resolution to reuse, so I wrote some lines of python with pylab and matplotlib:
import math
import pylab
import scipy
def vlen(v):
return math.sqrt(scipy.vdot(v,v))
p=scipy.randn(100,2)
a=scipy.array([x for x in p if vlen(x)>1.3 and vlen(x)<2])
b=scipy.array([x for x in p if vlen(x)<0.8])
pylab.scatter(a[:,0], a[:,1], s=30, c="blue")
pylab.scatter(b[:,0], b[:,1], s=50, c="red", marker='s')
pylab.savefig("linear.png")
fig = pylab.figure()
from mpl_toolkits.mplot3d import Axes3D
ax = Axes3D(fig)
ax.view_init(30,-110)
ax.scatter3D(map(vlen,a), a[:,0], a[:,1], s=30, c="blue")
ax.scatter3D(map(vlen,b), b[:,0], b[:,1], s=50, marker="s", c="red")
pylab.savefig("tranformed.png")
pylab.show()
Take — adapt — use for anything you like, you can rotate the 3D plot in the window that is shown and you can save the figures as PDF etc. Unfortunately, the sizing of markers in the 3d plot is not yet implemented in the latest matplotlib (0.99.1.2-3), so this only looks good with the latest SVN build.
i just wanted to say thank you, didn’t think one could illustrate the kernel trick quite so easily.
Posted by thomas on August 1st, 2010.
Thanks Thomas! If you liked this – you may be amazed at what some kernels can do, see here:
http://home.comcast.net/~tom.fawcett/public_html/ML-gallery/pages/
Posted by gromgull on August 1st, 2010.