Software can automatically critique composition of digital photographs

The software performs a pixel-by-pixel analysis to extract features from a photograph and then uses statistical analysis to classify and compare it to highly aesthetic pictures. Credit: Patrick Manesll

Everyone may be a critic, but now Penn State researchers are paving a way for machines to get in on the act. However, the researchers add that their photo-analysis algorithm is designed to offer constructive feedback, not to replace photographers.

The researchers have developed an algorithm that analyzes the arrangement of visual elements -- the composition -- of digital photographs. It also offers feedback about the perceived composition of the photograph and provides examples of similarly composed pictures of high aesthetic value, said James Wang, professor of information sciences and technology. Wang and colleagues recently received a patent for the system.

"If you think about aesthetics, everything is about composition," said Wang. "You can look into colors, or textures, or shapes, but, if you boil it down, you eventually have to consider all of these elements as part of composition."

Training a machine to become an art critic is not easy, according to the researchers. A machine must be trained with examples of highly regarded photographs in order to recognize good compositional elements, said Jia Li, professor of statistics, who worked with Wang.

Li added that the software the team developed relies on psychological theories of human vision as well as on the opinions of people to help classify images. For example, the researchers considered work done in Gestalt psychology, which focuses on how people perceive shape and form, in the development of the software.

A group of graduate students helped develop the algorithm by manually labeling hundreds of images from a collection of pictures on the website photo.net. The composition of each picture is categorized into horizontal, vertical, centered, diagonal and textured types. These labels, then, serve as the training dataset that teaches the algorithm how to classify photographs.

"Aesthetics is, of course, subjective," said Li. "What one person might find pleasing may not be pleasing to another, but there are elements that many good photos have in common."

The software performs a pixel-by-pixel analysis to extract features from a photograph and then uses statistical analysis to classify and compare it to highly aesthetic pictures.

Often, photos may have a blend of compositional elements. The software can detect and indicate cases when there may be several compositional options for the photographer.

"For example, a group of people could look at a photograph of, let's say, a canyon, and some would see it as a vertical composition, some people could see it as a diagonal composition, and some might even say it's a horizontal picture," Wang said. "So, sometimes, composition is not well defined."

Wang said he doubts the application will replace photographers any time soon. He hopes, instead, that it creates better photographers.

"Our goal from the beginning was to help photographers," said Wang. "If you are an amateur photographer then, potentially, a computer can analyze your photograph's composition and help you improve it."

The software can be installed on a server that can be accessed by a mobile phone. The researchers expect that, as smart phones become more powerful, the application will be able to rest on the phone. Eventually, as smart phones gain more memory and power, the software would not need to be stored on a server, but could be added to the phone itself.

"It's not even necessary to store very high resolution images for the system to work, so putting the software on a mobile phone is possible," said Li. "The system was developed so that it doesn't consume a lot of memory."

Wang and Li worked with Lei Yao, a former doctoral student in information sciences and technology.

The National Science Foundation partially supported this work.

Contacts:

Matt Swayne, mls29@psu.edu, Work Phone: 814-865-9481

A'ndrea Elyse Messer, aem1@psu.edu, Work Phone: 814-865-9481