Implementing DeepDream using Tensorflow: Dreamify Images using Deep Learning

DeepDream is one of the interesting applications of deep learning in computer vision. It is actually a computer program that was developed by Google engineer Alexender Mordvintsev. This program uses a convolutional neural network to find and enhance image patterns using algorithmic pareidolia. It creates dream-like hallucinogenic visualizations in the processed images. This model has been applied in the field of art history. Also, it has been used by researchers to allow users to explore virtual reality environments to mimic the experience of psychoactive substances.

In this article, we will demonstrate the implementation of DeepDream program to dreamify images. The pre-trained deep convolutional neural network used in the program will learn the patterns of an image that will be visualized in the processed image. First, the input image will be passed to the InceptionNet after that the gradient of the image will be calculated with respect to the activations of a particular layer of the network. Finally, the image is modified to increase the activations, and the patterns seen by the network are enhanced. In this way, a dream-like image is created.

Implementation

First of all, it needs a TensorFlow backend. 

import tensorflow as tf

import numpy as np
import matplotlib as mpl
import IPython.display as display
import PIL.Image
from tensorflow.keras.preprocessing import image

url = 'https://wallpaperaccess.com/full/1267119.jpg'

# Download an image and read it into a NumPy array.
def get_image(url, max_dim=None):
  name = url.split('/')[-1]
  image_path = tf.keras.utils.get_file(name, origin=url)
  img = PIL.Image.open(image_path)
  if max_dim:
    img.thumbnail((max_dim, max_dim))
  return np.array(img)

# Normalize an image
def deprocess(img):
  img = 255*(img + 1.0)/2.0
  return tf.cast(img, tf.uint8)

# Display an image
def show_img(img):
  display.display(PIL.Image.fromarray(np.array(img)))

# Downsizing the image makes it easier to work with.
original_img = get_image(url, max_dim=500)
show_img(original_img)

base_model = tf.keras.applications.InceptionV3(include_top=False, weights='imagenet')

# Maximize the activations of these layers
names = ['mixed3', 'mixed5']
layers = [base_model.get_layer(name).output for name in names]

# Create the feature extraction model
dream_model = tf.keras.Model(inputs=base_model.input, outputs=layers)

def get_loss(img, model):
  # Pass forward the image through the model to retrieve the activations.
  # Converts the image into a batch of size 1.
  img_batch = tf.expand_dims(img, axis=0)
  layer_activations = model(img_batch)
  if len(layer_activations) == 1:
    layer_activations = [layer_activations]

  losses = []
  for act in layer_activations:
    loss = tf.math.reduce_mean(act)
    losses.append(loss)

  return  tf.reduce_sum(losses)

In the DeepDream, the loss is maximized (generally, it is minimized) using the gradient descent. Once the loss is obtained for the chosen layers, the gradient is calculated with respect to the images and added to the original image. Adding the gradients to the image enhances the patterns seen by the network.

In the next step, the DeepDream is defined via a class that will be instantiated to create the model.

class DeepDream(tf.Module):
  def __init__(self, model):
    self.model = model

  @tf.function(
      input_signature=(
        tf.TensorSpec(shape=[None,None,3], dtype=tf.float32),
        tf.TensorSpec(shape=[], dtype=tf.int32),
        tf.TensorSpec(shape=[], dtype=tf.float32),)
  )
  def __call__(self, img, steps, step_size):
      print("Tracing")
      loss = tf.constant(0.0)
      for n in tf.range(steps):
        with tf.GradientTape() as tape:
          # This needs gradients relative to `img`
          # `GradientTape` only watches `tf.Variable`s by default
          tape.watch(img)
          loss = get_loss(img, self.model)

        # Calculate the gradient of the loss with respect to the pixels of the input image.
        gradients = tape.gradient(loss, img)

        # Normalize the gradients.
        gradients /= tf.math.reduce_std(gradients) + 1e-8 

        img = img + gradients*step_size
        img = tf.clip_by_value(img, -1, 1)

      return loss, img

deepdream = DeepDream(dream_model)

def run_deep_dream(img, steps=100, step_size=0.01):
  # Convert from uint8 to the range expected by the model.
  img = tf.keras.applications.inception_v3.preprocess_input(img)
  img = tf.convert_to_tensor(img)
  step_size = tf.convert_to_tensor(step_size)
  steps_remaining = steps
  step = 0
  while steps_remaining:
    if steps_remaining>100:
      run_steps = tf.constant(100)
    else:
      run_steps = tf.constant(steps_remaining)
    steps_remaining -= run_steps
    step += run_steps

    loss, img = deepdream(img, run_steps, tf.constant(step_size))

     display.clear_output(wait=True)
    show(deprocess(img))
    print ("Step {}, loss {}".format(step, loss))

  result = deprocess(img)
  display.clear_output(wait=True)
  show(result)

  return result

dream_img = run_deep_dream(img=original_img, steps=100, step_size=0.01)

As the resulting image is not much good and having a low resolution and noise, it will be fine-tuned using the octave that means an approach to perform the previous gradient ascent approach, then increase the size of the image and repeat this process for multiple octaves.

OCTAVE_SCALE = 1.30

img = tf.constant(np.array(original_img))
base_shape = tf.shape(img)[:-1]
float_base_shape = tf.cast(base_shape, tf.float32)

for n in range(-2, 3):
  new_shape = tf.cast(float_base_shape*(OCTAVE_SCALE**n), tf.int32)
  img = tf.image.resize(img, new_shape).numpy()
  img = run_deep_dream_simple(img=img, steps=50, step_size=0.01)

display.clear_output(wait=True)
img = tf.image.resize(img, base_shape)
img = tf.image.convert_image_dtype(img/255.0, dtype=tf.uint8)
show(img)

As we can see above, the dream-like image has been generated by the DeepDream program. For more clarity in the patterns, you can take other pictures with much better quality, and by tuning the parameters like octave scales, activations, etc. 

1. DeepDream, tutorial by TensorFlow.
2. DeepDream, Wikipedia.