How an Algorithm Draws Unexpected Connections Between Works of Art
Given a starting image, the artificial intelligence can identify objects that match its colors, textures and themes
/https://tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/2a/e0/2ae0befe-d600-42c9-b089-b0c5ef57d785/2020_aug10_saint-swan.jpg)
A new artificial intelligence algorithm called MosAIc draws unexpected connections between seemingly disparate works of art: for instance, a piece of Persian glassware that resembles a naturalist’s 18th-century illustration of a blue crane, and an 18th-century Dutch garment that pairs well with a woodwork doll of a mother and child.
Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) collaborated with Microsoft to create the tool, which draws inspiration from a recent exhibition at Amsterdam’s Rijksmuseum, according to a statement. The show, titled “Rembrandt and Velázquez,” featured 60 works by Dutch and Spanish Old Masters (including the two who lent the exhibition its name), juxtaposed to highlight similarities in the artists’ lighting, colors and symbols.
MosAIc focuses specifically on objects housed in the collections of New York’s Metropolitan Museum of Art and the Rijksmuseum. Using machine learning and an image-retrieval system, the A.I. can spot connections between works from different cultures, artists and mediums.
“We have used this approach to find incredibly interesting and thought-provoking pairs of images,” Mark Hamilton, who studies artificial intelligence at MIT, tells artnet News’ Sarah Cascone.
The software may help museums curate future exhibitions, but it certainly won’t replace experts: As Hamilton adds, “Art historians and curators can provide a much deeper understanding and historical context of these matches.”
One pair of paintings highlighted in “Rembrandt and Velázquez” provided significant inspiration for the project. Spaniard Francisco de Zurbarán’s The Martyrdom of Saint Serapion shows the titular saint with his arms raised, while Dutch artist Jan Asselijn’s The Threatened Swan depicts a bird widely interpreted as a representation of political martyr Johan de Witt. The paintings both center on sacrificial figures clad in white—a symbol of innocence—and standing in similar poses.
“These two artists did not have a correspondence or meet each other during their lives, yet their paintings hinted at a rich, latent structure that underlies both of their works,” says Hamilton in the statement.
To identify relevant couplings, users introduce an image, like that of the blue-and-white Double Face Banyan, or men’s robe. Then, they need only ask the software to find the musical instrument that most closely resembles the garment. The result? An 18th-century Delft violin made of white earthenware and covered with cobalt blue illustrations.
The most similar piece of ceramic artwork, meanwhile, is a Chinese figurine. As Daphne Leprince-Ringuet points out for ZDNet, this connection can be linked to “the flow of porcelain and iconography” between China and the Netherlands during the 16th through 20th centuries. A paper describing how the software connects paintings is available as a preprint, which means it has not yet been peer reviewed, and a web version of the tool is currently in beta testing.
MosAIc is built around a “k-nearest neighbors” (KNN) algorithm, which is typically used to group similar items for tasks like recommending products online, per ZDNet. Usually, KNN algorithms require programs to conduct a time-consuming search through an entire database or ask users to choose filters for the data they’d like to explore, raising the possibility of inadvertently excluding results of interest. But the team behind MosAIc built the A.I. to apply filters as it searches through the trove of artwork instead of applying filters before it begins. The researchers describe this approach a “conditional KNN tree,” in which the search starts at the trunk and follows promising paths while pruning fruitless branches.
“Our method is good at building a particular type of exhibition: unlikely pairs of art that span barriers and share common structure,” Hamilton tells artnet News. “We hope this approach can be used as a tool to help art historians find new patterns in history and gather evidence to support their hypotheses.”