Multi-functional Super Resolution using Xilinx Kria KV260

Introduction

For security cameras, more than 62,000 security cameras have been installed in the capital Bangkok, Thailand [1]. The retrieved frames from most cameras are of low quality, making it difficult to identify the appearance and patterns of objects in the scene. However, these low quality frames may be beneficial for limited capacity storage and transfer rate. Consequently, the performance of current camera quality may lead to the risk of social behavior and classification.

To achieve the value-added in the current scenario, the image enhancement or reconstruction is the priority of choice for most security cameras [2] and product recognition [3]-[4]. Single image super resolution (SISR) is one of the image enhancements that is used to enlarge the low to high quality image. This approach has significantly improved and recovered the unclear image, which is applied to minimize the risk of security and false classification or recognition.

The state-of-the-art SISRs are designed for full image enhancement, which has high cost and computation complexity. Generally, the enlargement of the image has been selected or automatically detected on a specific object of interest in the scene. For that reason, we propose a low-cost and low-complexity multi-functional super resolution system to apply in real-time applications.

[1] https://www.nationthailand.com/in-focus/40012998

[2] Lyu, K., Pan, S., Li, Y., & Zhang, Z. (2022). JSENet: A Deep Convolutional Neural Network for Joint Image Super-Resolution and Enhancement. Neurocomputing.

[3] Yang, X., Wu, W., Liu, K., Kim, P. W., Sangaiah, A. K., & Jeon, G. (2018). Long-distance object recognition with image super resolution: A comparative study. IEEE Access, 6, 13429-13438.

[4] Shamsolmoali, P., Celebi, M. E., & Wang, R. (2020). Deep learning approaches for real-time image super-resolution. Neural Computing and Applications, 32(18), 14519-14520.

The detail implementation steps are as follows:

Installation of Vitis

To install Vitis in pc please follow the steps from https://github.com/Xilinx/Vitis-AI

Next, install Vitis AI Library v2.0 from the link below https://github.com/Xilinx/Vitis-AI/blob/master/demo/Vitis-AI-Library/README.md

To set up the cross-compiler in PC, please follow the step one and set up the target on the Kria KV260, please follow the step two

https://github.com/Xilinx/Vitis-AI/tree/master/setup/mpsoc/VART#step1-setup-cross-compiler

Installation of Image and Video Test Kria Board and Run Vitis AI Library Examples

https://github.com/Xilinx/Vitis-AI/blob/master/tools/Vitis-AI-Library/README.md#running-vitis-ai-library-examples

Implementation of RCAN in demo code

Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu, "Image Super-Resolution Using Very Deep Residual Channel Attention Networks", ECCV 2018

Model description light-wieght RCAN for scale=2x. Compared with the original version, the changes include:

- Remove all Channel-attention (CA) modules

- Using less Residual blocks: --n_resgroups 3 --n_resblocks 2 --n_feats 32

On PC, we can get the model by downloading from the Vitis-AI repository. Clink on the link below

https://github.com/Xilinx/Vitis-AI/tree/master/models/AI-Model-Zoo/model-list/tf_rcan_DIV2K_360_640_0.98_86.95G_2.0

In file model.yaml

--

description: Pruned_RCAN for Single Image Super-resolution.

input size: 360*640

float ops: 86.95G

task: super-resolution

framework: tensorflow

prune: '0.98'

version: 2.0

files:

- name: tf_rcan_DIV2K_360_640_0.98_86.95G_2.0

type: float & quantized

board: GPU

download link: https://www.xilinx.com/bin/public/openDownload?filename=tf_rcan_DIV2K_360_640_0.98_86.95G_2.0.zip

checksum: 9b162e9b632bd00cb984ce571a089deb

- name: rcan_pruned_tf

type: xmodel

board: zcu102 & zcu104 & kv260

download link: https://www.xilinx.com/bin/public/openDownload?filename=rcan_pruned_tf-zcu102_zcu104_kv260-r2.0.0.tar.gz

checksum: 82452edf520936e91f6dda7633b110c8

- name: rcan_pruned_tf

type: xmodel

board: vck190

download link: https://www.xilinx.com/bin/public/openDownload?filename=rcan_pruned_tf-vck190-r2.0.0.tar.gz

checksum: 7d4b7020bf8f319657b8ee04a9e409c3

license: https://github.com/Xilinx/Vitis-AI/blob/master/LICENSE

--

We download board: GPU download link: https://www.xilinx.com/bin/public/openDownload?filename=tf_rcan_DIV2K_360_640_0.98_86.95G_2.0.zip

Will get "tf_rcan_DIV2K_360_640_0.98_86.95G_2.0.zip"

In readme.md file tell detail to retain, test and convert model to KV260.

--

# RCAN: Image Super-Resolution Using Very Deep Residual Channel Attention Networks

## Contents

1. [Environment](#Environment)

2. [Preparation](#Preparation)

3. [Test/Train](#Test/Train)

4. [Performance](#Performance)

5. [Model information](#Model information)

6. [Quantize](#Quantize)

## Environment

1. Environment requirement

- Python 3.6

- Tensorflow 1.15.1

- Pillow 6.0.0

- numpy 1.15.0

- scikit-image 0.15.0

2. Installation

- Create virtual envrionment and activate it (without docker):

```shell

conda create -n tf_rcan python=3.6

conda activate tf_rcan

```

- Activate virtual envrionment (with docker):

```shell

conda vitis-ai-tensorflow

```

- Install all the python dependencies using pip:

```shell

pip install -r requirements.txt

```

Note: If you are in the released Docker env, there is no need to create virtual envrionment.

## Preparation

1. Datasets description

- [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/)

- [Benchmarks](https://cv.snu.ac.kr/research/EDSR/benchmark.tar)

2. Dataset diretory structure

+ data

+ DIV2K

+ benchmark

## Test/Train

1. Model description

light-wieght RCAN for scale=2x. Compared with the original version, the changes include:

- Remove all Channel-attention (CA) moduel

- Using less Residual blocks: --n_resgroups 3 --n_resblocks 2 --n_feats 32

2. Evaluation

```

# perform evaluation on 4 benchmarks: Set5, Set14, B100, Urabn100

sh run_eval.sh

```

3. Training

```

# perform training on DIV2K dataset

sh run_train.sh

```

4. Testing on your own dataset

```

# set the test dataset path as your own directory

sh run_demo.sh

```

## Performance

- Note: the Flops is calculated with the output resolution is 720x1280

### Model_info

1. Data preprocess

```

data channel order: BGR(0~255)

input = input / 255.0

```

2. System Environment

The operation and accuracy provided above are verified in Ubuntu16.04.10, cuda-9.0, Driver Version: 460.32.03, GPU NVDIA P100

--

To implement it in the board, we have to build source files from the link below.

https://github.com/Xilinx/Vitis-AI/tree/master/demo/Vitis-AI-Library/samples/rcan

We use putty to command in kria board and Winscp to transfer file from PC.

We implement SR from model type int8 from rcan_pruned_tf.xmodel and rcan_pruned_tf.prototxt by command ./test_jpeg_rcan rcan_pruned_tf womanx2.png

The results should be similar to the one show in picture.

In terminal, show performance of at fps7.56.

The next step of the project is to convert the model in order to run on the PetaLinux 2021.1 platform. In addition, a GUI will be designed and integrated in the system by using Qt. The system will then be tuned for quality enhancement of surveillance videos.