There must be a Pony in there somewhere

BRT has talked often about neural nets and their ability to learn by example, a notion that tends to be discussed in the abstract, until now.

If we choose higher-level layers, which identify more sophisticated features in images, complex features or even whole objects tend to emerge. Again, we just start with an existing image and give it to our neural net. We ask the network: “Whatever you see there, I want more of it!” This creates a feedback loop: if a cloud looks a little bit like a bird, the network will make it look more like a bird. This in turn will make the network recognize the bird even more strongly on the next pass and so forth, until a highly detailed bird appears, seemingly out of nowhere.

The results are intriguing—even a relatively simple neural network can be used to over-interpret an image, just like as children we enjoyed watching clouds and interpreting the random shapes. This network was trained mostly on images of animals, so naturally it tends to interpret shapes as animals. But because the data is stored at such a high abstraction, the results are an interesting remix of these learned features.

Of course, we can do more than cloud watching with this technique. We can apply it to any kind of image. The results vary quite a bit with the kind of image, because the features that are entered bias the network towards certain interpretations. For example, horizon lines tend to get filled with towers and pagodas. Rocks and trees turn into buildings. Birds and insects appear in images of leaves.

In looking at these amazing images, one sees the beginnings of true AI, where interpreting reality via neural nets resembles, in indirect fashion, the paintings of Dali, Hieronymus Bosch or better yet, Chagall, where daydreams rule in unusual yet recognizable forms.

I and the Village - Chagall

When you let the system roam, images emerge similar to Van Gogh, electric, evocative and powerful.

The Starry Night - Van Gogh

If we apply the algorithm iteratively on its own outputs and apply some zooming after each iteration, we get an endless stream of new impressions, exploring the set of things the network knows about. We can even start this process from a random-noise image, so that the result becomes purely the result of the neural network, as seen in the following images:

There must be a Pony in there somewhere - James Kirkwood Jr.