AI-Based Super Resolution Upscaling

AI-Based Super Resolution Upscaling

Author: Ali Nikrang, Johannes Pöll

Version 0.5, March 2020

Abstract

This guideline aims to give an overview of the state-of-the art software implementations for video and image upscaling. The guideline focuses on the methods that use AI in the background and are available as stand-alone applications or plug-ins for frequently used image and video processing software. The target group of the guideline is artists without any previous experiences in the field of artificial intelligence.

Introduction

Upscaling of digital images (or video frames) refers to increasing the resolution of the original images. This can be very useful when artists work with (older) low-resolution materials that need to be used in a high-resolution output file. There are different approaches to upscaling. Some of them have already been integrated into commonly used image editing programs such as Adobe Photoshop or Adobe After Effects. Some others are available as stand-alone applications. In the context of this guide, we only focus on methods based on artificial intelligence and give an introduction to the employment of Deep Learning technology for video upscaling. The target group of this guide are artists who have no previous experiences in the field of Deep Learning systems and want to use deep learning technology with as less effort as possible.

We will explain three AI-based models for upscaling videos that can be used by artists without great effort.

First, we focus on Waifux2x-Caffe, which is a highly rated and recommended implementation of Waifu2x. The interface contains several options that can be used for magnifying the size of images as well as for reducing the compression artifacts of images.

It comes with several pre-trained models that can be divided into two main categories: photo and illustration. The choice of the pre-trained model depends strongly on the input content. For art-like content (such as anime), an illustration model is recommended, while a photo model is for camera contents.

Preserve Details 2.0 resampling is an upscaling algorithm introduced in Adobe Photoshop CC 2018. It is Adobe’s most advanced upscaling technology used in a variety of Adobe CC apps like Photoshop or After Effects (Detail-preserving Upscale). It utilizes artificial intelligence to interpolate more effectively. 

In its very simple Photoshop CC interface it allows users to change the pixel size of an image to the desired higher resolution. The Reduce Noise option can be used to get better results.

Video upscales using Adobe CC After Effects, however, are a bit more tricky. Encoding/decoding video files and upsizing them should be different processes. For this reason, we render source videos into image sequences beforehand, as well as audio as a separate sound file. Then the image sequence is imported in a new Adobe After Effects CC project and a new composition is created. After importing the extracted audio and matching it with the image sequence, you can start the upscaling process. 

To our knowledge, the only out-of-the-box AI-driven upscaling desktop solution available on the market is Topaz Gigapixel AI for single image upscaling and Topaz Video Enhance AI for video upscaling.

Although Topaz Video Enhance AI supports direct upscaling from an encoded video file, we recommend upsizing from image sequences for better results. The processing menu lets you choose between three AI models for upsampling according to the nature of your original source footage. 

Using Topaz Video Enhance AI shows very promising upscaling results. Nonetheless it is a newly released stand-alone application with some limitations to keep in mind.