TensorFlow Style Transfer

Style Transfer

This is a specific area of ML where you extract a style (which doesn't have to be a literal art style) and then attempt to generate (aka "optimize") a target (in this case, an image) based on the data model.

One of the most straightforward examples is via TensorFlow's generative style transfer tutorial.

Alex2Alex

For an example, let's try to extract a style from Alex Grey artwork, and apply it to a photo of Alex Trebek.

Optimization

Without any changes at all, here's how we can "stylize" the photo of Alex Trebek, using the style detected from Alex Grey:

import tensorflow_hub as hub
hub_model = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2')
stylized_image = hub_model(tf.constant(content_image), tf.constant(style_image))[0]
tensor_to_image(stylized_image)

John Carpenter "They Live!" vibes

There's really no finish line in my mind for this, so it makes sense to keep going. Next we will extract "features" from using VGG19 from Keras, which leaves us with a list of layer names. Using a few of these layers, we define the content and style, then we build a VGG19 model (note: just in case you're wondering what most of this means, I am too).

Once we've created the model, we can start to "optimize" where we keep running the image through our model until we get something we want. Here's what it looks like after just a few (3) steps, You can see some of the style starting to transfer, but it's very subtle:

After 100 steps:

And here's what it looks like after running 1,000 steps:

This actually looks pretty good so far, but there is some artifacting in places. One technique is to run an edge-detection function on the before and after images, and get a total variation loss number (in this case it was 144733.16)

Now we can re-run the optimization while taking into consideration the total variation loss. There's a lot going on in these images, so for example take a look at Alex's left hand (bottom right of the image).

The following is using a weight of 30:

Here's using a weight of 100:

Here's using a weight of 0:

Go try it yourself at TensorFlow's generative style transfer tutorial.