Forem: Gideon

Unlocking Artistry: A Guide to Neural Style Transfer

Gideon — Thu, 31 Aug 2023 06:40:23 +0000

Imagine having the ability to recreate Leonardo da Vinci's artistic style in your own photographs. To make your images appear convincingly as if da Vinci himself painted them – sounds quite remarkable, doesn't it? In this article, we'll delve into the world of Neural Style Transfer (NST) and show you how to apply it to images, videos, and even real-time video streams.

Content

What is Neural Style Transfer?
How does it work?
Neural Style Transfer Use Cases
How to Stylize Images
How to Stylize Videos
Real-Time Video Stylization
Conclusion

What is Neural Style Transfer?

Neural Style Transfer is a captivating technique that involves blending the artistic style of one image with the content of another, all while preserving the core essence of the content image. This process involves supplying the model with two input images: one representing the desired style and another portraying the content to be transformed.

How does it work?

Neural Style Transfer might seem like a magical feat in the realm of computer science, specifically within the field of machine learning. Think of it as teaching a computer to mimic the artistic techniques of various renowned painters. Let's break down the process into simpler terms:

Neural Networks: Picture a neural network, a sophisticated computer program that learns patterns from images. This network can identify shapes, colors, and objects present in images.
Content and Style Images: For this magic to unfold, you need two images – a content image and a style image. The content image represents the image you want to transform, while the style image is the masterpiece whose artistic style you wish to apply.
Layers of Understanding: The neural network operates in layers, much like how our brain processes different aspects step by step. Each layer focuses on distinct elements in the images, like lines, textures, or colors.
Finding the Perfect Blend: The key lies in adjusting the content image to retain its essence while adopting the stylistic elements of the style image. The neural network does this by altering the pixels of the content image bit by bit.
Achieving Balance: As the network modifies the content image, it continuously compares it with the style image. The goal is to make the content image mirror the style image's artistic expression while preserving the underlying content.
Iterations: This process iterates repeatedly, similar to a painter refining their artwork. With each iteration, the content image gets closer to harmonizing the desired content and admired style.
Revealing the Magic: After several iterations, the content image emerges with an appearance as though it were painted using the chosen style. It's akin to the computer mastering the fusion of two images in a truly artistic manner.

Neural Style Transfer Use Cases

The applications of Neural Style Transfer span a wide spectrum of creativity and innovation:

Artistic Rendering and Filters: NST serves as a digital canvas, transforming photographs into captivating artworks inspired by diverse artists and art movements.
Fashion and Design: Designers can experiment with unique visual styles, inspiring fresh ideas for textiles, clothing, and interior design.
Entertainment and Media: In cinema, gaming, and virtual reality, NST breathes life into virtual worlds by applying distinct visual aesthetics that evoke various eras, genres, or artistic sensibilities.
Educational Enrichment: Complex concepts find a new dimension as NST contributes to scientific illustrations and diagrams, making learning engaging and relatable.
Architectural Visualization: Architects and real estate developers can showcase designs with diverse artistic styles, helping clients envision projects from novel perspectives.

How to Stylize Images

To embark on the journey of stylizing images, follow these steps:

Begin by installing the required libraries using the following terminal command:

   pip install tensorflow tensorflow_hub matplotlib numpy --upgrade

Next, import the necessary libraries and create a function that leverages TensorFlow's fast style transfer capabilities.

import matplotlib.pyplot as plt
import tensorflow as tf
import tensorflow_hub as hub
import numpy as np

def preprocess_and_view_image(content_image_path, style_image_path, save_path=None):
    content_image = plt.imread(content_image_path)
    style_image = plt.imread(style_image_path)
    content_image = content_image.astype(np.float32)[np.newaxis, ...] / 255.
    style_image = style_image.astype(np.float32)[np.newaxis, ...] / 255.

    style_image = tf.image.resize(style_image, [256, 256])
    hub_module = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2')
    outputs = hub_module(tf.constant(content_image), tf.constant(style_image))
    stylized_image = outputs[0]
    stylized_image_array = stylized_image.numpy()
    stylized_image_array = np.clip(stylized_image_array, 0, 1)

    plt.figure(figsize=(15, 5))
    plt.subplot(131)
    plt.imshow(content_image[0])
    plt.title('Content Image')
    plt.axis(False)
    plt.subplot(132)
    plt.imshow(style_image[0])
    plt.title('Style Image')
    plt.axis(False)
    plt.subplot(133)
    plt.imshow(stylized_image_array[0])
    plt.title('Stylized Image')
    plt.axis(False)
    plt.show()

    if save_path is not None:
        plt.imsave(save_path, stylized_image_array[0])

preprocess_and_view_image('content_image.jpg', 'style_image.jpg', 'output_image.jpg')

Use the function to preprocess and view your stylized image:

    preprocess_and_view_image('content_image.jpg', 'style_image.jpg', 'output_image.jpg')

Output:

How to Stylize Videos

Moving beyond images, you can extend the magic of Neural Style Transfer to videos. Let's explore how to stylize videos frame by frame:

Install Dependencies:
Before diving in, ensure you have the necessary libraries. You'll need OpenCV for video processing and TensorFlow for style transfer.

   pip install opencv-python-headless tensorflow tensorflow_hub numpy

Import Libraries: We start by importing the necessary libraries. OpenCV (cv2) is used for video processing, TensorFlow (tf) is employed for style transfer, and numpy is used for numerical computations.

   import cv2
   import tensorflow as tf
   import tensorflow_hub as hub
   import numpy as np

Preprocess Video: This function, preprocess_video, takes a video file path, a style image path, and an optional output path for the stylized video. It captures the video and applies style transfer to each frame.

   def preprocess_video(video_path, style_image_path, output_path=None):
       cap = cv2.VideoCapture(video_path)
       style_image = plt.imread(style_image_path)

Preprocessing Style Image: Before applying style transfer, the style image is loaded, and its pixel values are normalized to the range [0, 1]. It's also resized to match the dimensions used in the style transfer model.

       style_image = style_image.astype(np.float32)[np.newaxis, ...] / 255.
       style_image = tf.image.resize(style_image, [256, 256])
       hub_module = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2')

Applying Style Transfer: Within the video processing loop, each frame is converted to the appropriate format for style transfer. The style transfer model from TensorFlow Hub is used to stylize each frame, which is then post-processed.

       while cap.isOpened():
           ret, frame = cap.read()
           if not ret:
               break

Stylizing and Post-processing: Each frame is preprocessed and stylized using the TensorFlow Hub model. The stylized frame is then converted back to BGR format for displaying or saving.

           stylized_output = hub_module(tf.constant(frame), tf.constant(style_image, dtype=tf.float32))
           stylized_video_output = stylized_output[0].numpy()[0]
           stylized_video_output = (stylized_video_output * 255).astype(np.uint8)
           stylized_video_output = cv2.cvtColor(stylized_video_output, cv2.COLOR_RGB2BGR)

Displaying or Saving: Depending on the presence of an output path, the stylized frame is either displayed using cv2.imshow or saved as part of the stylized video using out.write.

          if output_path is not None:
               out.write(stylized_video_output)
           else:
               cv2.imshow('Stylized Video', stylized_video_output)
               if cv2.waitKey(1) & 0xFF == ord('q'):
                   break
       cap.release()
       if output_path is not None:
           out.release()
       cv2.destroyAllWindows()

Applying the Function: To stylize a video, use the preprocess_video function with the input video path, style image path, and an optional output path.

   preprocess_video('input_video.mp4', 'style_image.jpg', 'output_video.mp4')

Output:

Real-Time Video Stylization

Taking it a step further, let's achieve real-time video stylization using your computer's camera:

Initialize Webcam:
Begin by capturing video from your webcam.

   def real_time_stylization(style_image_path):
       cap = cv2.VideoCapture(0)  # 0 refers to the default camera
       style_image = plt.imread(style_image_path)
       style_image = style_image.astype(np.float32)[np.newaxis, ...] / 255.
       style_image = tf.image.resize(style_image, [256, 256])
       hub_module = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2')

       while cap.isOpened():
           ret, frame = cap.read()
           if not ret:
               break

           frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
           frame = frame.astype(np.float32)[np.newaxis, ...] / 255
           frame = tf.image.resize(frame, [256, 256])

           stylized_output = hub_module(tf.constant(frame), tf.constant(style_image, dtype=tf.float32))
           stylized_video_output = stylized_output[0].numpy()[0]

           stylized_video_output = (stylized_video_output * 255).astype(np.uint8)
           stylized_video_output = cv2.cvtColor(stylized_video_output, cv2.COLOR_RGB2BGR)

           cv2.imshow('Real-Time Stylization', stylized_video_output)
           if cv2.waitKey(1) & 0xFF == ord('q'):
               break

       cap.release()
       cv2.destroyAllWindows()

Apply the Function:
Use the real_time_stylization function to view real-time stylization through your webcam.

   real_time_stylization('style_image.jpg')

Conclusion

Our voyage through the realm of Neural Style Transfer (NST) has opened the doors to a world where commonplace images and videos can be transformed into mesmerizing masterpieces. By seamlessly blending the artistic essence of one image with the content of another, NST elevates creativity and innovation to unprecedented heights. Let's summarize the key waypoints on our journey:

Comprehending NST: Our journey began with an introduction to Neural Style Transfer, an ingenious technique that harmonizes the artistic flair of one image with the substance of another, conjuring enchanting outcomes.
Unveiling the Process: We delved into the intricacies of NST, unveiling the mechanics that enable a computer to emulate diverse artists' techniques. From understanding neural networks to unraveling iterative refinement, we demystified the wizardry behind this technique.
The Tapestry of Applications: Our exploration extended to the multifaceted applications of NST. From infusing artistic renderings and inspiring fashion designs to animating cinematic visuals and enriching education, the horizons of NST span as wide as the imagination.
Crafting Image Marvels: Our journey deepened as we explored the practical implementation of NST on images. Armed with the tools to merge content and style seamlessly, you're empowered to fashion your own captivating artworks.
Animating Visual Narratives: Venturing further, we harnessed NST's magic for videos. By imbuing each frame with distinct stylistic elements, we witnessed how NST transmutes moving sequences into aesthetic visual stories.
Live Stylization Spectacle: Our odyssey culminated in real-time video stylization. Through your webcam, you gained a front-row seat to the captivating fusion of content and style, unfolding before your eyes in an instant.

As you embark on your personal journey with Neural Style Transfer, always remember that innovation recognizes no confines. Whether you're an artist seeking fresh perspectives or a technologist pushing the boundaries of visual creativity, NST provides a blank canvas where imagination commands the stage.

Set forth, experiment boldly, and let your artistic expression thrive through the boundless potential of Neural Style Transfer.

Find the code repository on GitHub and connect with me on LinkedIn.

Unleashing the Power of Computer Vision: Your First CNN in TensorFlow Made Simple!

Gideon — Wed, 26 Jul 2023 14:48:21 +0000

What is a Convolutional Neural Network?

A Convolutional Neural Networks are multilayered Artificial Neural Networks which are used for detecting features in images.

Why was CNN built and what problems does it solve?

Convolutional Neural Networks (CNNs) were built to help computers "see" and understand images more effectively. Before CNNs, computers struggled to recognize objects or patterns in pictures because images contain a huge amount of data. CNNs were designed to mimic how the human brain processes images.

Imagine looking at a picture of a cat. When you see the image, your brain identifies different parts of the cat, like its ears, eyes, and nose. CNNs work similarly. They use specialized layers to look for small patterns in the image, like edges or shapes, and then combine them to recognize more complex features like a cat's face.

By using CNNs, computers can automatically learn what's important in an image without needing humans to tell them. This makes it possible for computers to do tasks like identifying objects, detecting diseases in medical images, or even driving cars on their own. CNNs have revolutionized computer vision and made many amazing technologies possible.

Here are some problems that it solves:

Image classification: CNNs can be trained to classify images into different categories, such as cats, dogs, cars, and people.
Object detection: CNNs can be trained to detect objects in images. This is useful for applications such as self-driving cars and facial recognition.
Segmentation: CNNs can be used to segment images, which means dividing them into different parts. This is useful for applications such as medical image analysis and scene understanding.
Super-resolution: CNNs can be used to improve the resolution of images. This is useful for applications such as image restoration and video compression.

Classifying Food Images with CNN

I will show you how to create a simple CNN models with TensorFlow using the food101 dataset. Here, i will show you hands-on how to create this CNN model with these steps.

Step 1: Install TensorFlow

pip install tensorflow

We check if the tensorflow project is properly installed:

import tensorflow as tf
print(tf.__version__)

Step 2: We load the dataset into our local machine

import zipfile

!wget https://storage.googleapis.com/ztm_tf_course/food_vision/pizza_steak.zip

# Unzip the file
zip_ref=zipfile.ZipFile('pizza_steak.zip')
zip_ref.extractall()
zip_ref.close()

zip_ref = zipfile.ZipFile('pizza_steak.zip'). This line creates a ZipFile object named zip_ref, which allows us to work with the contents of the ZIP file named "pizza_steak.zip"

The zipfile.ZipFile() function takes the name of the ZIP file as an argument and opens it for reading.

zip_ref.extractall() line extracts all the contents of the ZIP file into the current working directory.

The extractall() method is called on the zip_ref object to perform the extraction.

The zip_ref.close() After the extraction is complete, this line closes the ZipFile object to free up system resources.

It is a good practice to close the ZipFile object after you're done working with it.

Step 3: We preprocess our images by converting them into numerical representations. This enables our model to effectively detect features in the images.

import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator

#set the seed
tf.random.set_seed(42)
train_datagen = ImageDataGenerator(rescale=1./255)
valid_datagen = ImageDataGenerator(rescale=1./255)

#Setup paths to our data directories
train_dir = 'pizza_steak/train'
test_dir = 'pizza_steak/test'

# Import data from directories and turn it into batches
train_data =train_datagen.flow_from_directory(directory=train_dir,
                                              batch_size=32,
                                              target_size=(224,224),
                                              class_mode='binary',
                                              seed=42)
valid_data = valid_datagen.flow_from_directory(directory=test_dir,
                                               batch_size=32,
                                               target_size=(224,224),
                                               class_mode='binary',
                                               seed=42)

To prepare our data for training, we use the flow_from_directory method on the data generators. For the training data, we create batches of 32 images each, resizing them to a target size of 224x224 pixels. The class_mode is set to 'binary' since we are performing a binary classification (pizza or steak). The same process is applied to the validation data.

By dividing the data into batches, we efficiently feed the numerical representations of the images into our neural network, enabling it to learn patterns from the data during the training process

Step 4: Construct the CNN model with the capability to identify distinctive features within our images.

# Build a CNN Model
model_1= tf.keras.Sequential([
    tf.keras.layers.Conv2D(filters=10,
                           kernel_size=3,
                           activation='relu',
                           input_shape=(224,224,3)),
    tf.keras.layers.Conv2D(10,3,activation='relu'),
    tf.keras.layers.MaxPool2D(pool_size=2,
                              padding='valid'),
    tf.keras.layers.Conv2D(10,3,activation='relu'),
    tf.keras.layers.Conv2D(10,3,activation='relu'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(1,activation='sigmoid')
])

# Compile our model
model_1.compile(loss=tf.keras.losses.binary_crossentropy,
                optimizer=tf.keras.optimizers.Adam(),
                metrics=['accuracy'])
# Fitting our Model
model_1.fit(train_data,
          epochs=5,
          steps_per_epoch=len(train_data),
          validation_data=valid_data,
          validation_steps=len(valid_data))

Once we train our model, the obtained accuracy is 0.8920 for the training dataset, while the accuracy for the validation dataset is 0.8600. Additionally, the loss values for both the train and validation datasets are 0.2982 and 0.3116, respectively.

To enhance our model's performance and decrease its losses, several approaches can be employed. This includes increasing the number of epochs during training, exploring different optimizers, and augmenting the dataset with more data. Each of these strategies plays a role in fine-tuning the model and improving its overall accuracy and efficiency.

Conclusion

As we conclude this article, I hope you found the journey into the world of Convolutional Neural Networks insightful and exciting. If you've enjoyed exploring the power of CNNs for image classification, then you'll definitely want to catch me on my next adventure - a captivating style transfer project!

In the upcoming article, we'll dive into the fascinating realm of neural style transfer, where we blend the artistry of different images to create mesmerizing masterpieces. You'll witness how CNNs can transform ordinary photographs into extraordinary works of art, seamlessly merging the style of one image with the content of another.

Be prepared to embark on an artistic journey filled with creativity and innovation. Whether you're a developer, an artist, or simply a curious mind, the magic of style transfer will captivate you.

So stay tuned, as we uncover the secrets behind this enchanting process in our next article. Until then, keep exploring the incredible possibilities of deep learning and computer vision!