Expression Preserved Face Privacy Protection Based on Multi-mode Discriminant Analysis

Xiang Wang , Chen Xiong Qingqi Pei and Youyang Qu

Abstract: Most visual privacy protection methods only hide the identity information of the face images, but the expression, behavior and some other information, which are of great significant in the live broadcast and other scenarios, are also destroyed by the privacy protection process. To this end, this paper introduces a method to remove the identity information while preserving the expression information by performing multi-mode discriminant analysis on the images normalized with AAM algorithm. The face images are decomposed into mutually orthogonal subspaces corresponding to face attributes such as gender, race and expression, each of which owns related characteristic parameters. Then, the expression parameter is preserves to keep the facial expression information while others parameters, including gender and race, are modified to protect face privacy. The experiments show that this method yields well performance on both data utility and privacy protection.

Keywords: Privacy protection, multi-mode, discrimination, expression-preserving.

1 Introduction

With the proliferation of inexpensive video surveillance and Internet, an increasing number of cameras are applied to a variety of public places. These cameras are used to record, store and process large amounts of image data. And in dealing with the image data in different scenes, human’s privacy captured in the images is inevitably threatened. At the same time, in the rapid development of the diversification services of Internet technology, the users’ private information will be leaked. As a result, people are increasingly concerned about whether their privacy information is protected.

As the monitoring system and the Internet in people’s lives have become more and more popular, for how to solve security issues exist in these scenarios, people have put forward varies effective methods. Some researchers have proposed to remove face identification information from images in a scene where there is a security risk. There are two kinds of solutions: ad-hoc methods such as blurring and pixelation [Neustaedter, Greenberg and Boyle (2006)], as well as K-Same [Newton, Sweeney and Malin (2005)] and K-Same-M[Gross, Sweeney, Fernando et al. (2006)], and other conventional methods. Neustaedter et al. [Neustaedter and Greenberg (2003)] destroyed the identity information by blurring the face, Berger [Berger (2000)] employed the pixelation to carry out de-identification operation on the images and retain skin color; Newton et al. [Newton, Sweeney and Malin (2005)] proposed K-Same-Family, where the face image to be protected is replaced by the average image of its the most similar K face images so that the probability of identifying the protected face is no more than 1/K. However, these face de-identification methods are not to remove all the information of the images, but only to hide the identity information of the images, while the expression, behavior and other awareness information are not considered. Some researchers [Hudson and Smith (1996)] believed that there was a fundamental trade-off between data utility and privacy. When the image is modified, the effective information used to understand the image may be deleted, so the modified image may become lack of practicality, hence it is particularly important to put a balance between privacy and data utility. In the face de-identification methods,people are usually concerned about the private information than the utility of images, so in the general de-identification images, the expression of face and other effective information will be affected, and the information in the video broadcast, video surveillance and other scenes are indispensable. Therefore, considering the balance between privacy protection and data utility, the proposed method preserves the expression information of the original image while ensuring that the privacy information is protected,so that the processed face image remains utility without revealing the identity information.Applying the multi-mode discriminant analysis [Sim, Zhang, Li et al. (2009)], the images containing multiple attributes are decomposed into orthogonal subspaces, and the characteristic parameters of each attributes such as gender, race and expression are obtained. By changing the parameters corresponding to the attributes such as gender and race while maintaining the expression parameters the same, we can get the de-identification images whose known facial attributes have changed in addition to expression while the privacy is protected.

2 Related works

The rapid development of video surveillance and Internet has brought convenience for people’s lives, but has also caused a hidden danger for people’s privacy. More and more researchers have made a lot of important work to protect people’s privacy. In the field of data mining, image and computer vision, issues related to face de-identification have also been resolved. For example, Newton and Gross et al. [Newton, Sweeney and Malin(2005); Gross, Sweeney, Fernando et al. (2006)] studied the trade-off between privacy protection and data utility, and some of these ideas are effectively applied to face de-identification. In computer vision, earlier face de-identification methods are ad-hoc methods such as masking, blurring and pixelation. However, these techniques are easy to apply to any image, but the performance in the face de-identification is not guaranteed.

Newton et al. [Newton, Sweeney and Malin (2005)] proposed a k-Same algorithm based on k-anonymity for the above problem. The k-Same algorithm achieves the purpose of face de-identification by averaging the k images in the image set to obtain the average face image. The recognition rate of the face de-identification image obtained by the k-Same algorithm does not exceed 1/k, so the privacy is protected. However, although the privacy of the face images is protected, the k-Same algorithm destroys the utility of the original data, and the de-identification images will show fuzzy and ghosting phenomenon.In view of the shortcomings in k-Same algorithm, the researchers proposed an improved algorithm for k-Same. k-Same-Select algorithm [Gross, Airoldi, Malin et al. (2006)] was proposed to obtain the de-identification images by dividing the image set into separate subsets which contain different attribute information. Using the k-Same-Select algorithm,the privacy of face images is protected and the attribute of each subset is well preserved.The k-Same-M algorithm was proposed by Gross et al. [Gross, Sweeney, Fernando et al.(2006)], which used the AAM [Cootes, Edwards and Taylor (1998)] to get the shape and appearance parameters of the image, and applied the k-Same algorithm for the appearance parameters of k images. The AAM model in the k-Same-M algorithm effectively solved the misalignment of face images.

The initial face de-identification methods only consider the identity information of the face images, and more and more face de-identification researches work to consider the attribute information of the images. Gross et al. [Gross, Sweeney, Torre et al. (2008)]introduced a multifactorial model that can decompose the test image into identity factors and non-identity factors, then applied the de-identification method to identity factors. The de-identification image obtained by this method contains the expression information. But the method has poor performance in de-identification. Du et al. [Du, Yi, Blasch et al.(2014)] retained the race, sex, and age attributes, and k-Same was applied after measuring attributes feature and selecting the corresponding AAM parameters. Each attribute corresponds to an AAM model. Othman et al. [Othman and Ross (2014)] changed the age information while protecting the privacy information.

In this paper, we maintain the facial expression information while protecting the privacy of face images. By establishing the multi-mode subspaces, we can get different attributes of the face. Changing the attributes other than the expression and keeping the expression features not change to produce the face de-identification images while retaining the original expression.

3 A new approach based on MMDA

In this paper, we apply the AAM algorithm to normalize the shape and appearance of the training sets, and then decompose the images in multiple modes. By decomposing the different attributes of the face into the corresponding independent subspace, we get the attribute parameters of different classes under each attribute, then the attributes feature can be changed by the related parameters. In this paper, by changing the other facial attributes of face images while maintaining the same expression, the privacy of the face images are protected and the original expression are retained.attributes

Figure 2: The proposed method can protect privacy while remaining expression attribute

Figure 3: 20 objects selected from CK+ database containing gender, race and expression

Our training set contains 20 objects with size of 640∗490 selected from the Extended Cohn-Kanade database [Lucey, Cohn, Kanade et al. (2010)]. Each image in the training set has three attributes such as gender, race and expression. The gender attribute includes male and female, and race attribute includes Africa and Europe, and the expression includes five classes such as smile, angry, sad, fear and surprise. The number of images corresponding to the different classes of each attribute is the same.

3.1 Pre-processing and normalization

Before applying multi-model discriminant analysis, we need to eliminate the effect of factors such as angle, attitude and position in the face images, also pre-process and normalize the images. In this paper, we mark the key facial points and use AAM method to get normalized images.

Figure 4: Overview of the pre-processing

3.1.1 Locating landmarks

Given a face image, we first locate its landmarks. By obtaining the coordinates of the landmarks, we can get the shape information. In this paper, 68 landmarks including facial contours, eyes, nose and mouth are selected. Then the shape vectors are produced by forming the coordinates into vectors.

Figure 5: Locating the facial landmarks

3.1.2 Active appearance models

To normalize the image whose landmarks are located, we use the AAM to align the image and get shape-normalized appearance. By establishing the average shape model and the average appearance model respectively, and using the Procrustes algorithm [Dryden and Mardia (1998)] and PCA [Kirby and Sirovich (2002)], we can get the parameters p and λ which control the two models respectively. The shape model and appearance model are combined to obtain a mixed model. After completing the AAM modeling, each image in the sample set is normalized to get a standardized face, so that the effects of facial attitude, angle, position and other factors can be removed.

Figure 6: Normalizing a face image by shape model and appearance model

We first locate facial landmarks for the training set, and 68 landmarks are marked on the key parts of the face, such as the eyes, the nose, the mouth and the face contour. Each image is aligned with the AAM algorithm. Applying the appearance model to face images,the shape-normalized appearance can be obtained. After this, all the images in the training set are normalized, and each image is 100×100 in size.

Figure 7: Training set after processing and normalization

3.2 Decomposition and synthesis

In this paper, the face data is decomposed into multiple orthogonal subspaces by the MMDA algorithm. The face data is composed of multiple attributes such as gender, race and expression. Applying MMDA to each subspace, the orthogonal basis in each mode is calculated and the corresponding parameter is obtained. By changing these parameters,what can be affected is only the corresponding attribute. In order to protect the privacy of face image, we can selectively alter the facial attributes to change some features. At the same time, in order to ensure the integrity of the face, we retain the expression attribute,so as to protect the privacy of the face image, but left facial expression information unchanged.

Figure 8: MMDA projects vector X into three mutually orthogonal subspaces (represented by the blue axis), encoding gender, race, and expression, respectively

Changing the X parameter in the gender subspace will only affect the gender without changing the race or expression. This corresponds to moving X to A. Reconstructing Vector A reveals a new face image that only shows a gender change instead of race or expression.

MMDA

Define X as a training matrix of D×N, xiis the training vector corresponding to any normalized image in the training set, and ximultiply labeled has K attributes, i=1, …, N,each label represents a attribute such as Male, African, Smile, etc. Each attribute i has Ciclasses,,, …. For gender, there are two class labels, Male and Female; For race,there are two class labels, African and European; For expression, there are five class labels, Smile, Angry, Sad, Fear and Surprise. Without loss of generality, it is assumed that the overall mean of X is 0, if not, the overall mean is subtracted for each eigenvector.Now, the overall mean of the training samples is equal for each attribute.

Whitening

Given an initial set of training samples X, the sample set X for each attribute feature is whitened in order to reduce the correlation between the features.

We compute the global scatter matrixfor each attribute i,

then Eigen-decompose it to get eigenvector matrix U and eigenvalue matrix D,

To reduce the redundancy of sample data, we only retain none-zero eigenvalues in D and corresponding eigenvectors in U. Then the (N-1)×D matrix P can be computed by the matrix D and U,

then apply it to sample data X to get whitened data,=PTX, now the scatter matrix ofis the identify matrix I. Since the sample data for each attribute comes from the same training set, only one whitening operation is required.

Building Subspace

For each attribute, we firstly compute the between-class scatter matrixand the within-class scatter matrix,is the whitened class mean,

then maximize the Fisher Criterion,

Zhang et al. [Zhang and Sim (2006, 2007)] has proven that JF(Vi) is equal to the ratio, where λband λware the eigenvalues of,, respectively. As is shown that,λb+λw=1, so JF(Vi) can reach the maximum value by making the λbequal to 1, and the subspace Vican be obtained by keeping the eigenvectors corresponding to λb=1 in matrix Vi, the dimension of Viis Ci-1 and. It can be proven that the subspace obtained above is the most discriminant subspace.

Repeat the same step for all attributes to get subspaces V1, V2,… Vk, the dimension of all subspaces is, but the whitened data holds a much larger subspace of dimension, so there is a residual space V0which contains residual variations that do not exist in all the identify space, and residual dimension is r0=N−∑iCi+K−1.

To compute V0, we can apply Gram-Schmidt algorithm. For each vector∈,subtract its projection on the basis of each subspace, and orthogonally normalize the remaining component to get V0. The complete subspace V contains:

Each subspace in V is orthogonal to each other, (Vi)TVj=0, if i≠j.

Multi-mode decomposition

Assuming that the vector to be decomposed is∈, we can decompose it into each attribute to obtain the parameter vector y by using the trained orthogonal subspace V,

from vector y, {C1-1，C2-1，…，Ci-1} denote attributes respectively, that is, the first C1-1 parameters represent the first attribute feature, the C1to C2-1 parameters to be the second attribute, and the Cito Ci-1 ones show the i-th attribute. By controlling these, we can selectively change the corresponding facial characteristics.

The training set contains K=3 facial attributes, such as gender, race and expression,where gender includes both male and female labels, race has two labels of African and European, and expression is either smile, angry, sad, fear or surprised.

Decomposing any training datato get the parameter vector yi:

In the vector yi, parameter of each attribute can be obtained by a certain degree:

1) The gender scalar gi, can only take on two constant values: G1or G2, representing Male and Female, respectively.

2) The race scalar ri, can only take on two constants: R1or R2, representing African or European, respectively.

3) The vector ei, which represents expression, can only take on five constants: E1, E2,E3, E4, E5, meaning smile, angry, sad, fear or surprised.

4) The remaining vector sirepresents residual space.

In fact, these scalars and vectors contain the average characteristics of attributes such as Male, Female, African, etc. Therefore, facial appearance can be changed by changing corresponding parameters. Meanwhile, we can control the intensity of the change, for instance, we can set the expression parameter to σE3, and we will also get sad appearance of different degrees by varying σ.

Alteration and synthesis

Given a decomposed training data, the synthesis of face data is achieved by whitened matrix P and subspace V,

where Prreverses the whitening and PCA operation of P. We decompose and synthesize the training set, where Q=PrV.

In the obtained feature parameters, the first six parameters respectively represent the characteristics of gender, race and expression attributes, so that the corresponding column vectors q1, …, q6decode these facial attribute appearance, and the decoding information remaining in the residual space contains other face information in the image.

Thus, we can define these vectors as semantic faces, which form a semantic basis for the face image, and a linear combination of any vectors results in a face that contains different gender, race, and expression appearance. For example, for G1q1, an average male face can be obtained by the gender parameter and semantic face q1, and G2q1can create an average female face. By changing scalar g, such as 3G1or 3G2, we can get a more masculine or feminine face.

Figure 9: By changing the sole parameter of q1 (the gender Semantic Face), Male (left)and Female (right) faces are synthesized. Varying this parameter also makes the face appear more or less Masculine (or Feminine)

Face reduction

Finally, by decomposing and synthesizing the training data, a new face image can be obtained. In order to preserve the integrity of the original image, the synthesized face is restored to the original image using AAM algorithm, so that the resulting image contains non-face information of the original image.

Figure 10: Reduction of the altered face image

4 Privacy protection

Applying MMDA to a face image labeled different classes, we can decompose it into gender, race and expression attributes, then a new synthetic image can be obtained by selectively changing any of the known attribute without affecting other attributes. The synthetic face image can generate new features on attributes such as gender, race and expression, which make the synthetic face image different from the original image on these attribute. Therefore, we can protect the privacy of face images through different degrees by selectively changing the attribute characteristics.

Figure 11: Overview of the proposed method containing decomposition, alteration and Synthesis

In this paper, we change the gender and race attribute to obtain the new gender and race attribute feature, and do not change the expression features, so that the privacy of face images can be guaranteed while data utility is considered.

To protect privacy of the training data, we decompose each image in the training set by MMDA algorithm to get the parameter vector Y related to the attribute feature,

where yiT=[gi,ri,eiT,siT].

For a face image, facial gender and race feature will be transformed. The first and second parameter gi, riin yirepresent the gender and race attribute respectively. By changing giand riin each yito the opposite of the current value, such as gi=G1, ri=R1while transforming it to gi= σG2, ri= σR2where σ is a constant greater than 0, the protection of facial gender and race attribute is archived. While changing the gender and the race attribute, the parameter vector eiin yiis kept constant so that the retention of the facial expression attribute is realized.

Hence, we can get y¯iwhose other face attribute has changed except that the expression attribute is invariant by changing the parameter vector yiselectively, and applying synthesis and reduction to the reconstructed y¯ito get the final face image whose expression appearance is unchanged while the face of other attributes has been protected.

5 Experiments

5.1 Evaluation criteria

Using the change detectors (CDs) proposed by Sim et al. [Sim and Li (2015)], we can determine whether the relevant attributes of the face after the synthesis have been changed. The change detector is based on the Face++ attribute classifier, and accepts two inputs, the original image and the new synthetic image. When it judges that two images differ in the relevant attribute, the output is “Changed”, otherwise it outputs “Unchanged”.We build three CDs, one each for gender, race and expression.

Let β be the probability that a facial attribute is correctly changed, tpis the true positive rate which is the probability that the attribute in the input image is changed when the CD outputs “Changed”, while fpis the false positive probability that the attribute is not changed in the input image but the CD outputs “Changed”. α is the ratio of the observed changed times to the times the CD outputting “Changed”. From these definitions, we can see:

The probability β can be observed to analyze effectiveness of our method when changing the several attributes of the face image.

5.2 Experiments and analysis

Single attribute change

The gender, race and expression attributes are selected without changing other attributes feature respectively, and each attribute is changed with varying intensity σ from 0.5 to 2.5 as the interval is 0.5. All three change detectors are used to measure α, then the β is calculated.

Table 1: Actual “changed” rate (‘β’) values for single-attribute change. These show that our method is effective in changing an attribute (bold values) and retaining an attribute(values in normal font)

Table 2: Actual “changed” rate (‘β’) for multi-attribute change (G, R, E respectively represents Gender, Race and Expression)

We can observe the “Gender Change” rows in the Tab. 1, for example, only when the gender attribute is changed, the gender CD holds a larger β than other CDs at each intensity level, indicating that the gender attribute is indeed changed, while other face attribute remain unchanged. Looking at the other rows in the table, the conclusion is similar. Therefore, our algorithm shows that it is effective to change facial attributes as well as remain attributes by MMDA.

Multiple attribute change

Changing multiple facial attribute and measuring β for corresponding CDs. Similarly,when observing different β, it can be seen that when the multiple attributes are changed,the unaltered attribute is unaffected, and as the change intensity σ increases, the β of the changed attributes are getting larger, while the β of the remain attribute decreases.

Privacy based on expression unchanged

We select the gender and racial attributes to change with different intensity σ, and make no change to the expression attribute. As is shown in Tab. 2, the expression feature is not affected when other two attributes are changed. The Face++ matcher is used to authenticate the face images with unaltered expression, and the expression feature is classified by the attribute classifier. The results are as follows:

Figure 12: The result shows Recognition Accuracy and Expression Accuracy

From the experiments, it can be seen that the correct recognition rate of face is relatively low when the gender and race attributes are changed at the same time, and the result of expression classification is maintained at a high correct rate. This shows that when changing the gender and race attributes while retaining the same expression, the privacy of face image has been protected well and the expression appearance still remain unaffected, so the purpose of privacy and the face of non-privacy information retained can be achieved.

6 Conclusions

In this paper, we mainly solve the problem of protecting the facial privacy while preserving the expression information in the image. Through the orthogonal decomposition of the multi-attribute face image, we establish the independent subspace of each attribute, get the corresponding parameter, and selectively change the parameters of other attributes in addition to the expression to retain only expression appearance the same and gender and race attributes different from the original face. Concluded from the experiment results,our method has a good effect on the decomposition of face attributes, and performs well on the privacy protection of face images by changing different attribute features while expression feature is preserved.

Acknowledgements:This work was supported by the National Key Research and Development Program of China (No. 2016YFB0800601), the Key Basic Research Plan in Shaanxi Province (Grant No. 2017ZDXM-GY-014) and the Key Program of NSFC-Tongyong Union Foundation under Grant U1636209.