Asking too much of “compression”

Dude has an immediate stage presence. Charming German accent, cool deadpan delivery. He starts off staring at the audience, and launches into a story which turns out to be a joke. The gist (sorry, this blog is not the best venue for delivery of jokes) is that there are three prisoners facing death, and they are asked their last wishes. The first says something to establish the joke rule of threes. The second, a German, says, “I vant to give a speech!” The last, an Englishman, says, “I want to be executed before the German.” Ba-da-boom!

He’s still telling some jokes and building up to his talk. I heard people complaining over the break about people who took too long to get into their talks. But the topic here, among other things, is humor. One way or another, I’m liking this guy. Now he’s showing a slide that says is his “take-home message”: “(Human) Unsupervised Intelligent Agent.” (I guess we’re meant to insert verbs in there, like on CNN headlines?)

Okay, so he’s now outlining the technical specs of a human as a computer. “It” has the capacity to store a lifetime (say, 100 years) of sensory input, at a compressed bit rate. He’s detailing the “compression algorithm” we have, which he says gets better as individual learn — meaning, as we learn what data to save. He says what matters is not the number of bits we’re saving, but the change in the number of bits we need to save as our learning algorithm gets better.

So far, he’s said a whole lotta’ nothin’, but he says this is all we need to understand in order to get the rest of his talk on music, humor, etc. He’s talking about a scenario with a robot sitting in a dark room. This is boring, he says, because it’s completely compressible and there’s no change. But if you hear music, there are more bits and you can compress it more, so there’s more discovery going on in the process of compressing more.

He’s talking about art, now, such as caricatures, where talented artists discover how to represent famous faces in just a few figures (he claims one artist can represent President Obama’s face in five lines, though he’s not showing it to us). Science and art, he says, are all about distilling this “essence” and discovering how to compress more. Now he’s tying this back into entropy and artificial neural nets, which try to discover ways of compressing or matching data sets in previously unknown efficient ways.

Schmidhuber’s claim is thought-provoking, to say the least. Certainly one of the most distinguishing features of intelligence is its ability to “distill the essence” of phenomenon so that they can be explained in more simple ways. He talked in particular about representing visual data (which I found especially interesting because I’m such a big fan of PNG and vector-based encoding of images, because they’re lossless and massively more efficient than pixel-based JPGs).

But this strikes me as a great example of metaphor gone awry. First of all, the metaphor itself is only so good. He only mentioned “distilling the essence” and simplicity once; mostly, he’s been talking again and again about bit rates, reward optimizers, and compression. Even if we could perfectly replicate intelligence based on these principles, getting into strong claims about how “this is really what’s going” (which is implicit in all of this) is absurd, because these are all recent inventions of engineering. But really, the applicability of the metaphor itself is weak. We might notice some useful correspondences between the system he’s describing and the way our own intelligence works. But there is in his talk a casual presumption that we’ve somehow gotten at some deeper scientific truth at how intelligence works, rather than having just, as A.I. folks like to say, produced an interesting heuristic.

The other point that needs to be made is that the aspect of intelligence for which Schmidhuber has produced a metaphor is just one part of intelligence, and not even close to the essence of it. Hardly all of intelligence, creativity, and humor, is “figuring out the essence behind things.” That implies a stripping down (a “compression,” as he would call it) that is in fact the opposite of what most creativity and humor is about. Where is the compression in this?:

“Autumn Rhythm”, Jackson Pollock