Image enhancement and post-processing for low resolution compressed video

Myoungjin Kim, Beomsu Kim, Mincheol Hong

(School of Electronic Engineering, Soongsil University, Seoul156-743, Korea)

Image enhancement and post-processing for low resolution compressed video

Myoungjin Kim, Beomsu Kim, Mincheol Hong

(School of Electronic Engineering, Soongsil University, Seoul156-743, Korea)

This research paper recommends the point spread function (PSF) forecasting technique based on the projection onto convex set (POCS) and regularization to acquire low resolution images. As the environment for the production of user created contents (UCC) videos (one of the contents on the Internet) becomes widespread, resolution reduction and image distortion occurs, failing to satisfy users who desire high quality images. Accordingly, this research neutralizes the coding artifact through POCS and regularization processes by: 1) factoring the local characteristics of the image when it comes to the noise that results during the discrete cosine transform (DCT) and quantization process; and 2) removing the blocking and ring phenomena which are problems with the existing video compression. Moreover, this research forecasts the point spread function to obtain low resolution images using the above-mentioned methods. Thus, a method is suggested for minimizing the errors found among the forecasting interpolation pixels. Low-resolution image quality obtained through the experiment demonstrates that significant enhancement was made on the visual level compared to the original image.

discrete cosine transform (DCT); projection onto convex set (POCS); image resolution

0 Introduction

As wired and wireless communication improves, it becomes possible to actively share various multimedia information. Since it is now possible to transmit large amounts of data over the Internet, sharing user created contents (UCC) has consolidated its position as a medium in everyday life. However, limitations in data transmission capacity hinder resolution and image quality for users of these UCC videos. Accordingly, video compression using resolution and compression rates at a temporal and spatial level that reduces data transmission rates is necessary when the ability to provide multimedia information is limited.

These video encoding methods use block-based discrete cosine transform (DCT)[1]. However, these BDCT encoding methods produce elements that obstruct image quality, such as blocking phenomena (blocking artifacts) and ring phenomena (ringing artifacts). In particular, the loss of DC and low frequency substance leads to the blocking phenomenon, which is discontinuous at the block boundary, while loss of high frequency substance produces ring phenomenon, which is a wave phenomenon near the boundary. When the compression rate is higher, substances that are lost increase, and blocking and ring phenomena get aggravated. Image quality gets decreased even more[1,2].

When this type of decreased low-resolution video is viewed on the end device at high-resolution, decreased quality becomes amplified; thus users are bound to feel visual discomfort. To resolve this discomfort, there is a need to remove the blocking and ring phenomena that occur in low-resolution videos.

Accordingly, this research seeks to remove blocking and ring phenomena, and enhance image resolution. The research factors the image’s local characteristics when it comes to the noise resulting from compressing the video in order to study techniques for removing the coding artifact.

For the remainder of this paper, section 1 describes our proposed algorithms, and section 2 discusses the results and presents a conclusion.

1 Proposed method

To remove quantization noise, this research seeks to remove blocking and ring phenomena by forecasting quantization parameter (QP) value. The forecast required extracting the image information from the compressed low-resolution video. Video with noise removed as described above is converted to high resolution through space resolution conversion. Fig.1 demonstrates the overall flow.

Fig.1 Overall flow chart of the proposed algorithm

Projection onto convex set (POCS) is defined by using quantization step size and the difference between each of the pixels located at the block’s inside and boundary. This is used to remove the blocking phenomenon. In general, the difference in the pixels at the block’s boundary shows a high correlation with the difference in the pixels inside the block. Average value is used for the difference in the pixels near the block's boundary to decide the projection set's limitations.

In Fig.2, the block's boundary lies between p0and q0. The average value of differences with neighboring pixels p0and q0situated at the block’s boundary, average value of difference for all neighboring pixels including p0and q0, and the value of difference at the block’s boundary are as shown in Eqs.(1) to (3). MPD is the element that indicates the level of activity at the nearby pixels. The larger the MPD is the more the applicable domain can be regarded as complex. On the other hand, the domain is flat when the MPD is smaller.

BD=|p0-q0|.

(3)

When the difference among the neighboring pixels at the block's boundary is greater, the applicable domain is the domain with a significant activity level. Thus, BD concerns the value of differences at the block’s boundary when it becomes wider. In contrast, when the difference among the neighboring pixels at the block’s boundary is smaller, the applicable domain has a low activity level. Then the scope is inactive when the difference at the block’s boundary becomes narrower.

Fig.2 Block’s boundary

A regularized filter technique for removing a compressed video’s quantization noise is significantly affected by the quantization coefficient QP. When QP is greater, the degree of image blocking increases. If QP is greater, it is necessary to set up the convex set with a wider scope. If QP is small, then one must set up a convex set that has narrow scope. Moreover, pixels are projected by defining POCS following QP in order to prevent saturation that results when filtered pixel values deviate from the specific scope.

As for the representative methods for enhancing an image's spatial resolution, there are super resolution image re-configuration and interpolation techniques. Super resolution entails estimating the relative movement of the low-resolution image that can be used as a reference. Then their sub-pixel unit movements can be used to obtain a sub-pixel value[3]. However, a wrong estimation of the sub-pixel movement can impair the image, and there are only few instances in which images that include encoding unit movements are obtained for the same scenes. Thus, it is difficult to apply to diverse areas in practice.

Image interpolation is the technique for obtaining low-resolution images from one given sheet of low-resolution images. Active research was conducted on adaptive interpolation. This technique operates in an adaptive manner by factoring in the low-resolution image’s borderline domain so that high frequency elements such as the image's borderline can be maintained, while also handling simple techniques such as zero-order interpolation, bi-linear interpolation, spline interpolation, and so on.

Fig.3 shows that it is quite possible to obtain a low resolution image with little computation as well as by re-configuring the image amplified from one reference sheet. However, it is not possible to prevent the loss of high frequency substances such as information on the borderline (which is important for recognizing the image), owing to the blocking phenomenon and flattening phenomenon[3,4].

When an arbitrary image is obtained or stored, quality decreases due to the movement of the device that obtains the image, inaccuracy of focus, and scattering while on standby. In general, the size of the image’s distortion is expressed by

where y, x and n are the column vectors of MN×1 that are aligned according to the sequence of each step. This signifies the depleted image with noise, and original image and attached noise, while H refers to the point spread function of the spatially varying or spatially invariant that produces degradation of the space domain. This is expressed as the matrix of MN×MN size.

Fig.3 Undirected image interpolation resul

To recover and forecast the original image x from Eq.(6) that expresses the degradation, a regularization recuperation method is used. This is defined as

In the above equation, the first item on the right side represents the degree of reliability towards the data, while the second item shows the relaxation degree of the original image. C represents the 2-D high-frequency filter that is used in general. α is the variable for the regularization medium used to control the characteristics of the two items that have contradictory characteristics.

Regularization recuperation uses the information available in advance that the original image has relaxed characteristics. This type of information restricts the expression domain of the recuperated image in order to minimize the scope of forecasting error.

This research paper assumes that the low-resolution image and its distribution characteristics are the same, this is, the low resolution image’s characteristics are linear and the noise is added after passing through the space which is linear and invariant. Accordingly, H is obtained by following the sequence shown in Fig.4.

The recuperation image to be obtained from the regularization relaxation equation given in Eq.(7) can be obtained by replacing the value obtained from adopting Gradient to M(x) with the 0 vector. This is expressed as

xM(x)=-HT(y-Hx)+αCTCx=

Fig.4 Block diagram on the use of the low resolution image’s PSF through space-invariant PSF forecasting

Because it is usually impossible to obtain matrix H in an ideal manner, it is not easy to obtain a directly recuperated image from Eq.(6). Accordingly, point spread function (PSF) was forecasted as shown in Fig.4 to use in the above mentioned equation H. Moreover, repetition technique was used to recuperate an increasingly accurate image by solving the error resulting from the estimated value. The repetition value for this is defined as

xK+1=xK+β[HTy-(HTH+αCTC)xK],

Repetition value of Eq.(6) is the regular regularization recuperation method. It is possible to add adaptiveness by attaching restrictions to the above mentioned value. xkbecomes the recuperated image where the step is obtained, and it is possible to obtain a recuperated image that is increasingly closer to the original image due to the forecast PSF as the repetitive values increase.

Regularization medium variable α can be set using various methods. However, this research decides upon the regularization medium variable in each of the step’s repetition values and uses the method of using it in the subsequent repetition value, which is determined as

(8)

The method suggested by this research paper combines the gradient technique with projection technique. The repetition value, optimized through the gradient calculation, is obtained in the convex function. By projecting into the forcefully defined significant domain using the information obtained from the low-resolution image relationship, it is possible to know that the repetition value converges into the projection set defined by the information provided in advance and by the set defined by the gradient.

2 Experimental results and conclusion

To conduct this experiment, “Mobile” and “Stefan” images (300 frames, CIF size) were used. Each QP value (16, 19, 23, 26) and 30 fps was applied. Experimental results are shown in Table 1. Compressed video information and the image’s local information were used to forecast QP value with quantization noise removal technique using the POCS method. Regularized filter was applied to produce better results. Moreover, it was possible to obtain visually softer image.

Table 1 Comparison of PSNR performance following QP used in the image and forecasted QP

Table 2 gives the result of the proposed method that increases spatial resolution by leveraging PSF forecasting. It is a comparison of the images produced by using the Bicubic interpolation techniques for a given image. The resolution enhancement technique (Proposed 1) leverages the proposed space-invariant PSF forecasting technique to all the sequences. Proposed 2 maintaines the PSF obtained with the static characteristics of the proposed method into a consistent 5×5 size. Resolution enhancement (Proposed 3) uses the adaptive interpolation method and space-invariant PSF forecasting. Table 2 confirms that the Bicubic interpolation technique vs. average is enhanced by at least 1.5 dB and by 7.9% from PSNR aspect.

As shown in Tables 1 and 2, the method proposed by this paper can be used to convert a low-resolution video such as UCC to high resolution, which can help mitigate the fatigue resulting from poor image quality.

We hope our research on compressed video resolution and image enhancement will offer these benefits: removal of distortion in compressed videos owing to blocking and ring phenomena, and enabling regular users to provide multimedia information services through high quality imagery.

Table 2 Comparison of PSNR performance following QP used in the image and forecasted QP

[1] SHEN Mei-yin, Kuo C-C J. Review of postprocessing techniques for compression artifact removal. Journal of Visual Communication and Image Representation, 1998, 9(1): 2-14.

[2] LI Zhen, Delp E J. Blocking artifact reduction using a transform-domain markov random field moel. IEEE Transactions on Circuits and Systems for Video Technology, 2005, 15(12): 1583-1593.

[3] ZHAO Meng. Video enhancement using content-adaptive least mean square filter. PhD thesis. Eindhoven University of Technology, 2006.

[4] Chen H Y, Leou J J. A visual attention approach to image interpolation. In: Proceedings of IEEE International Conference on Multimedia and Expo, 2008: 169-172.

date： 2012-10-04

The MKE(the Ministry of Knowledge Economy), Korea, under the ITRC(Information Technology Research Center) support program supervised by the NIPA(National IT Industry Promotion Agency) (NIPA-2012-H0301-12-2006)

Myoungjin Kim (webzealer@ssu.ac.kr)

CLD number： TN911.73 Document code： A

1674-8042(2013)01-0030-04

10.3969/j.issn.1674-8042.2013.01.007