PhotoSketch: Taking image composition to the next level

We don't often write about work in Computer Vision but when we do we always present something that is likely to blow your mind (see VideoTrace and Adobe's Interactive Video Editing.)

Enter PhotoSketch, work that was recently published in ACM's Transactions on Graphics. The application takes as input a rough, hand drawn sketch of a scene and then generates a realistic image using information found on the Internet. The results are magnificent to say the least considering the difficulty in filtering the vast amount of noise in online images. The application requires that the user supplies labels with each object sketched; these are used to search for relevant images online. The object outline along with automated segmentation algorithms are used to extract specific objects from the photos found, e,g, a dog, an elephant, a car, a Frisbee etc. Another mechanism is used to blend the segmented images together and generate appropriate shadows. The results are jaw dropping to say the least considering how little input from the user is required.

The video below explains the process in more detail.