Neural Style Transfer Lab

About This Project

Neural style transfer is a technique rooted in the seminal 2015 paper by Gatys, Ecker, and Bethge, "A Neural Algorithm of Artistic Style." The approach leverages a pre-trained convolutional neural network (typically VGG-19) to decompose images into content representations and style representations. Content is captured by the higher-layer feature maps of the CNN, which encode the semantic structure of the image, objects, and spatial arrangement, while style is captured by the Gram matrices of the lower-layer feature maps, encoding texture, color correlations, and brushstroke patterns across spatial scales.

The optimization process begins with a noise image (or a copy of the content image) and iteratively adjusts pixel values to minimize a weighted combination of content loss and style loss. Content loss measures the mean squared error between the feature representations of the generated image and the original content image at a chosen layer (e.g., conv4_2), ensuring the output preserves the original scene structure. Style loss measures the mean squared error between the Gram matrices of the generated image and the style image across multiple layers (e.g., conv1_1 through conv5_1), encouraging the output to adopt the textures and color palette of the chosen artistic style.

The interactive demo above simulates this process procedurally: the content image is a generated cityscape, and each style applies characteristic brushstroke patterns, color palette transformations, and texture overlays that progressively blend into the content. While the demo runs in the browser using p5.js for visualization, a production implementation would use PyTorch with a pre-trained VGG-19 backbone and the L-BFGS optimizer for pixel-space optimization.