A classification method of building structures based on multi-feature fusion of UAV remote sensing images

Haoguo Du,Yanbo Cao,Fanghao Zhang,Jiangli Lv,Shurong Deng,Yongkun Lu,Shifang He,Yuanshuo Zhang,Qinkun Yu

Yunnan Earthquake Agency,Kunming,650224,Yunnan,China

Keywords:Remote sensing image Building structure classification Multi-feature fusion Object-oriented classification method Texture feature classification method DSM and DEM elevation classification method RGB threshold classification method

ABSTRACTIn order to improve the accuracy of building structure identification using remote sensing images,a building structure classification method based on multi-feature fusion of UAV remote sensing image is proposed in this paper.Three identification approaches of remote sensing images are integrated in this method:object-oriented,texture feature,and digital elevation based on DSM and DEM.So RGB threshold classification method is used to classify the identification results.The accuracy of building structure classification based on each feature and the multi-feature fusion are compared and analyzed.The results show that the building structure classification method is feasible and can accurately identify the structures in large-area remote sensing images.

1.Introduction

Previous earthquake disasters show that the collapse of houses is often a significant cause of casualties and property losses(Li and Li,2016;Zhang et al.,2020;Du et al,2021;Wang et al.,2015).One of the most important measures for earthquake disaster prevention is to carry out earthquake disaster risk assessment.The building structure classification method is able to quickly and effectively classify the building structure types in large areas,thereby guaranteeing the assessment accuracy(Zhou G et al.,2010;Du et al.,2021;Cao et al.,2016).

Based on relevant investigations,the classification method is considered to be consisted of 7 secondary methods,which are namely field survey,visual interpretation,spectral features,object-oriented classification,texture features,morphological characteristics,and multi-feature fusion.

1.1.Field survey

Lu et al.(2011)summarized 22 recent earthquake disaster assessment reports in Yunnan Province and gave statistics on per capita floor space of buildings,building area ratios of various systems,and building area ratios of various structure types in Yunnan Province.Du et al.(2019)conducted field survey in the urban area of Honghe County,Yunnan Province.The analysis includes construction age,structure type,fortification level,floor,floor height,area,appearance shape and topography of houses;this information are further used to conduct a three-dimensional simulation evaluation of earthquake hazards.

1.2.Visual interpretation

He et al.(2016) used UAVs to capture house image data from 12 survey points in Yunnan,and compared the data to the actual survey results.This method provides effective basic information for the seismic performance assessment of buildings.

1.3.Spectral feature

Fig.1.Flow chart of building structure identification method based on multi-feature fusion of UAV remote sensing images.

The fact that multi-spectral image building targets have different characteristics at different scales,Zhang et al.(2019) proposed a multi-spectral image building identification method based on improved FCN.Furthermore,Hajime and Fumio (2000) used a multi-scale classification method to obtain seismic damage information of buildings,and identified features such as brightness,saturation,and tone of remote sensing images.

1.4.Object-oriented classification

Selecting rubble as the major damaged object of building collapse,Dong et al.(2011) evaluated the accuracy of rubble classification using three methods:pixel-based method,object-based method,and objected-based method integrated with statistical tectures;Feng et al.(2014) used wavelet transform and Fourier descriptor to construct a multi-scale description model of the target shape,and proposed a new object-oriented classification method for high resolution remote sensing images.

1.5.Texture features

Xie et al.(2017) chose single buildings as the object,calculated the texture feature parameters and the spectral feature parameters.They further standardized the parameters that are positively related to the degree of damage,forming a feature parameter vector.As a result,information of the distribution and extent of building damage are obtained according to the visualization results of the vector.

1.6.Morphological characteristics

Chen et al.(2018) performed the identification results using UAV remote sensing images with the assist of regional geographic environmental characteristics,including building arrangement,floor space,and building shadows;Hu et al.(2014)proposed an enhanced morphological building index (EMBI) for the automatic extraction of high-resolution building images using the index and the geometric constraints of the ground features;Combined the object-oriented method and the morphology information,You et al.(2019) integrated image spectrum,geometric and context features,and finally proposed a hierarchical extraction method that produces high-resolution remote sensing images with building information.

Fig.2.Remote sensing image of Gusheng village.

1.7.Multi-feature fusion

Taking advantage of the changing vector analysis method,Li et al.(2019) calculated the spectrum,texture feature and morphological building index difference of the corresponding objects,and compared the pre-and post-damaged building status;Li et al.(2019) used high-resolution remote sensing images of the meizoseismal area before and after an earthquake,combined with texture and morphological features to extract building damage information,which was further analyzed through the detection of building change.

All the aforementioned methods have been proved to be effective but with different accuracy.According to (methods or results) which compares the building extraction accuracy results after different feature fusion (Li and Zhang,2016;Du et al.,2021),this paper used a higher accuracy classification methods taking advantage of object-oriented and texture features.Then,the digital elevation classification method based on DSM and DEM was integrated,and the results were classified by the RGB threshold classification method,to obtain a building structure classification method based on multi-feature fusion of UAV remote sensing images.The classification results were verified and analyzed by calculating the missed detection rate,misclassification rate and kappa coefficient value(Zhang et al,2017).

2.Research method and technical route

2.1.Technical route

Fig.1 is a flowchart of the building structure identification method.Data collection which uses UAV remote sensing technology to obtain images,DSM and DEM digital elevations in the study area (Du et al.,2018).Then we take data processing step to classify the remote sensing image map of the study area by object-oriented and texture feature classidication methods.In addition,the process utilizes DSM and DEM digital elevation classification methods to identify the number of building floors.After that,the multi-feature fusion which uses the RGB threshold classification method to perform threshold classification on the object-oriented,texture feature and digital elevation results based on DSM and DEM.It also performs multi-feature ratio transform (Brovey)fusion to obtain building structure classification results based on multi-feature fusion of UAV remote sensing image.The last step is comparative analysis,verifying the feasibility and practicality of the model by calculating the misclassification rate,missed detection rate and kappa coefficient of building structure identification.This process compares and analyzes the accuracy of building structure classification based on different features and multi-feature fusion.

2.2.Precision analysis

We used the misclassification rate,missed detection rate and kappa coefficient as indicators to measure classification accuracy.Kappa coefficient is used for agreement test and classification accuracy measurement shown as Eq.(1) and (2).And Table 1 exhibits the classification standard of kappa coefficient accuracy level.

Table 1 The classification of accuracy level.

Table 2 The relationship between number of floors and building structure.

Table 3 The classification results.

Table 4 Comparison of the accuracy of three building structure identification results.

Table 5 Comparison of the accuracy of multi-feature fusion classification results.

Where k is the kappa coefficient,the index value of classification accuracy;Pois the overall classification accuracy,which is obtained by dividing the sum of samples correctly classified of each type by the total number of samples;acis the number of real samples for each type;bcis the number of prediction samples for each type;n represents the total number of real samples.

2.3.Digital elevation classification method based on DSM and DEM

Fig.3.The samples of building structure:the sample of frame structure (a),the sample of brick-concrete structure (b),and the sample of brick-wood structure (c).

Fig.4.The results of object-oriented classification method.

Digital Surface Model(DSM)refers to a ground elevation model that includes the height information of bridges,trees and buildings on the ground.Compared with DSM,DEM only contains the elevation information of the terrain without other surface information.Therefore,DSM is a DEM-based model that further covers other surface elevation information besides the ground terrain.This study extracted building elevation information from DSM of the UAV remote sensing images,and used Eq.(3) to calculate the number of building floors.By summing up the rules between the building structure and the number of floors,large-size building structures can be classified.The calculation equation of floor number in a building is shown as follows,

Fig.5.Variation curves of texture feature of different building structures.

Fig.6.Building classification results based on DSM and DEM elevation.

Where Niis the number of floors of the i-th building,Hiis the DSM value of the i-th building,and hiis the DEM value of the location of the i-th building.The height of the floor is taken as a consolidated value of 2.5 m..

2.4.Object-oriented classification method

Classification is the most important step in the object-oriented classification method.We adopts the multi-resolution classification method(Chen et al.,2006).The combined cost function:

Fig.7.The results of RGB threshold classification method.

Fig.8.The results of object-oriented and RGB threshold classification methods (a),texture features and RGB threshold classification method (b),results obtained based on DSM and DEM elevation and RGB threshold classification method (c).

where w is the weighting factor with a range from 0 to 1.In this paper,we set the spectral weight w1to 0.9 and the shape weight w2to 0.1.

Where f represents the heterogeneity of the image area,w1and w2are the spectral weight and the weight of the compactness,respectively,hcolorand hshaperepresent the spectral and shape heterogeneity,n is the number of bands,Piis the weight of the i-th band,σiis the standard deviation of the spectral value of the object in the i-th band,N is the total number of pixels in the image area,N1and N2are the total number of pixels in different image areas,L is the total length of the rectangular boundary that contains the image area,L1and L2are the total length of different image areas,E is the actual boundary length of the image area,E1and E2are the boundary lengths of different image areas,u is the overall compactness of the image area,and v is the smoothness of the image area boundary.

Fig.9.The results of:visual interpretation and field survey(a),object-oriented classification(b),texture features classification(c),results obtained based on DSM and DEM elevation(d).

2.5.Classification method

For color images,the traditional threshold classification algorithm initially converts the color image into a grayscale image,which is subsequently set as the threshold and processed pixel by pixel.If the pixel grayscale value is less than or equal to the threshold,it is considered as the foreground and set to black.If the value is greater than the threshold,it is considered as the background and set to white.The conversion formula of the three primary colors of RGB to gray is:

Where Gray is the converted gray value;R,G,and B are the value of three converted colors,respectively.

2.6.Texture feature classification method

In this paper,the texture features are analyzed using the gray-level cooccurrence matrix,which is a statistical-based texture description method.The texture feature of the gray-level co-occurrence matrix was first proposed by Pesar esi and Benediktsso (2001).The gray-level co-occurrence matrix is a matrix function of pixel distance and angle.It reflects the gray-scale correlation of adjacent pixels through the probability distribution of 2 Gy pixels appearing in the image at the same time.The parameters reflecting the gray-scale characteristic of buildings mainly include the mean value,variance,evenness,contrast,dissimilarity,entropy,angular second moment,and correlation.Some of the algorithms are listed as follows:

The mean value can reflect the local change of the image gray level,and the calculation method is shown in Equa.10:

where mean is the average value of the texture features of the building,N is the number of image gray-level,Pi,jis the probability that a pixel with a gray-level of j appears after starting from a pixel with a gray-level of i.

Where entropy is the entropy of the texture feature of the building.Angular second moment is the measurement standard of the uniformity of the image grayscale distribution and the thickness of the texture.When the image texture is finer and the grayscale distribution is more uniform,the energy value is larger,and vice versa.The calculation method is as follows:

Fig.10.The results of visual interpretation and field survey (a),multi-feature fusion method for building identification (b).

Where ASM is the angular second moment of the building texture feature.

Correlation is also called homogeneity,which is used to measure the similarity degree of the gray level of an image horizontally or vertically.Therefore,the value reflects the local gray correlation:the larger the value,the greater the correlation.The calculation method is as follows:

Where correlation is the correlation of building texture features.

2.7.Multi-feature fusion classification method

There are a number of image fusion methods,among which the common ones include:HIS fusion,HSV transformation,high-pass filter(HPF) fusion,wavelet transform,Multiplicative inverse,Brovey transform,Color normalized (CN) transform,Principal Component (PC)transform and Gram-Schmidt.According to the comparative analysis of the classification accuracy of nine fusion methods detailed in Gao(2015),this paper adopted a relatively simple Brovey transform fusion method,which multiplies the normalized high resolution images of three multispectral bands.The calculation method is shown in Eq.(14):

3.Data source and preprocessing

3.1.Sample selection in the study area

The study area,Gusheng Village,Dali Bai Autonomous Prefecture,is flat and densely populated with relatively simple building structures that could be interpreted easily.Fig.2 is a remote sensing image of the study area acquired by UAV.

According to previous field surveys and analyses,building structures in this area can be classified into three types:frame,brick-concrete and brick-wood.Thirty samples were selected from the study area,as shown in Fig.3a,b,and c.Among these,the building numbers of frame,brickconcrete and brick-wood structure were 7,7,and 16,respectively.

3.2.Object-oriented classification

In this study,we use envi to extract the structure information of buildings based on the object-oriented method.We set the classification parameters according to the results from previous investigations (Hao et al.,2020) and repeated classification experiments,as:classification scale 30,shape feature 0.1,compact degree 0.7.Fig.4 shows that the object-oriented classification method can effectively distinguish buildings from ground,trees,and grass.However,a small number of misclassification or unobvious results still exist.The misclassification cases were mainly generated by the buildings with similar colors or shapes to ground,trees and grass;the unobvious cases were mainly associated with the connecting part between adjacent buildings.

3.3.Texture feature classification

Different texture features in UAV remote sensing images can be used to distinguish different types of buildings.The texture features extracted from the co-occurrence matrix are used to measure the entire image area.For the gray-level co-occurrence matrix in each direction,the above features can be calculated.For the gray-level co-occurrence matrix in four directions (0°,45°,90°,135°),each statistical attribute can generate a texture image or band,which can be applied to classification and spectral features.These statistical data not only reflect the significant differences in spatial characteristics of classification categories,but also are compatible with traditional classification algorithms based on probability models.Different statistical attributes can be selected as indicators according to different images and building information.The results show that the mean value,variance,evenness,contrast,entropy,and angular second moment of buildings with different structures are classified successfully,while the dissimilarity and correlation are not obvious for the classification of frame and brick-concrete structure buildings.Therefore,we select mean,variance,homogeneity,contrast,entropy,and angular second moment as the texture feature classification parameters.Fig.5 shows the variation curves of texture feature of different building structures.

3.4.Digital elevation classification based on DSM and DEM

Building height difference calculated by DSM-and DEM-based UAV remote sensing image is a stable and transferable building feature.Using a ground object as a reference,the building structure can be separated according to the height difference between the ground and the building.Fig.6 shows the classification results.Table 2 shows the corresponding relationship between the number of floors and the building structures obtained from the selected samples in the study area in Fig.3.

3.5.Classification based on RGB threshold

The RGB threshold classification method is adopted,because in the feature space,the classification cannot be successful at one time.As shown in Fig.7a,b,and c are the values of three primary colors(R,G,B)of different building structures (frame,brick-concrete,brick-wood).It can be seen that the distinction is not obvious.The corresponding d,e,f are the classification results of a,b,and c obtained based on the RGB threshold,and the structure of the building is distinguished well.As shown in Fig.8,red,green and blue lines respectively represent the number and distribution of the three types of structures:frame,brickconcrete,and brick-wood.

4.Results

Fig.9 shows the results of three classification methods of building structure classification.Table 3 shows the number of frame structure buildings,brick-concrete structure buildings and brick-wood structure buildings obtained from three classification methods using the same scale.

It can be seen from Table 3 that the results obtained by three classification methods are different.For the classification of frame structure and brick-concrete structure buildings,the results based on the DSM and DEM elevation classification method is the closest to the field survey results;for the classification of brick-wood structure buildings,the results based on the texture feature classification method is the closest to the field survey results;for the total number of buildings,the results based on the texture feature classification method is the closest to the field survey results.

5.Discussion

5.1.Comparative analysis

Table 4 shows the accuracy analysis of the three classification methods.Each classification method includes misclassification cases and/or missed detection cases.When using the texture feature classification method,the kappa coefficient is the largest,and the total missed detection rate is the lowest,indicating the best classification effect on the building structures in the study area.Such prominent effect is primarily due to the fact that the three types of building structures are distinctly different,especially the texture features.The texture features are particularly noticeable compared with surrounding objects,as reflected by the variation curves of different texture features in Fig.5.The kappa coefficients obtained by the other two classification methods (objectoriented classification method and DSM and DEM elevation classification method)are 0.82624 and 0.8185,respectively.These two methods show good results when identifying the overall building structures in the study area.However,the remote sensing image interpretation signs of some frame structures and brick-concrete structures are highly similar;as a result,the misclassification rate and missed detection rate of frame structures and brick-concrete structures are relatively higher than that of brick-wood structures.At the same time,the buildings and the surrounding environment may overlap each other.Consequently,water,vegetation,roads,and open spaces will be mistakenly classified as buildings,which will also affect the classification accuracy.

5.2.Multi-feature fusion method

In order to improve the accuracy of the building structure classification,we used Brovey Transform to combine the three aforementioned classification methods,and obtained a building structure classification method based on the multi-feature fusion of UAV remote sensing images.Fig.10 is the resulting map obtained using the multi-feature fusion building structure classification method.Table 5 shows the comparison between the classification results based on multi-feature fusion of remote sensing images and the field survey results.It can be seen that after multifeature fusion,the misclassification rate is 5.704%,the missed detection rate is 2.4955%,and the overall building structure classification kappa coefficient is 0.8702.Compared with the three identification methods,the classification effect of multi-feature fusion method is better and the accuracy is higher.

Due to the cultural and environmental differences,the building structures in different regions also show diversified appearances.Therefore,different features should be selected for fusion to classify the building structures through remote sensing images in different regions.In this paper,the building structure classification method based on multifeature fusion of UAV remote sensing images was proposed on the basis of three classification methods:object-oriented,texture features and DSM-and DEM-based digital elevation.The three methods were compared and analyzed,and the ratio transformation method was used for fusion to improve the classification accuracy of building structures.Among them,the object-oriented classification method sorted original images into image objects based on feature space,improving the image processing quality and analysis efficiency.The texture feature classification method selected 6 features of the building structure with better distinction based on the eigenvalue results of the samples (mean,variance,evenness,contrast,entropy,and angular second moment).This method can effectively classify large-area images with different texture features.Finally,the classification method based on the DSM and DEM digital elevation identified the building structures according to the regular pattern of the relationship between the number of floors and building structure type,leading to satisfied results.Although multifeature fusion achieved good results in building structure identification,its accuracy may still be affected by two factors.The first factor is the color of the surrounding environment.If the surrounding objects(e.g.water,vegetation,roads,open space) have the similar colors to the targeted building,they may be misidentified,as shown in Fig.8a,b and c.The second one is the morphological structure of a building.Some buildings in the study are fuzzy in images,leading to a relatively higher misclassification rate and missed detection rate of frame and brickconcrete buildings,as shown in Tables 4 and 5 For the purpose of reducing the influence of the above factors:(1)It is necessary to improve the building structure identification method to enhance the efficiency and accuracy.For example,the integration of oblique photography technology,multi-modal remote sensing data fusion technology,regional building structure characteristics or shadow characteristics may improve the efficiency and accuracy of building structure classification to a certain extent;(2) Different fusion methodscan be modified to further improve the accuracy of the identification,such as HIS fusion,HSV transformation,high-pass filter (HPF)fusion,etc.

6.Conclusions

We classified different building structures based on the UAV remote sensing images of Gusheng Village in Dali Bai Autonomous Prefecture using the object-oriented method,texture features method,and DSM and DEM elevation method,respectively.After the comparative analysis of the classification results,the ratio transformation (Brovey) method is used for the fusion of the three classification methods based on their advantages and disadvantages,and a multi-feature fusion building structure classification method is proposed.By comparing and analyzing the accuracy of different classification methods and multi-feature fusion classification method,the results show that the proposed method is characterized by high accuracy,strong feasibility and practicality in the classification of building structures in large-area remote sensing images.

Acknowledgment

We thank peer reviewers and editors for their valuable comments.Their suggestions improved the accuracy of the paper and are beneficial for the follow-up studies of this project.

This study is sponsored by National Key R&D Program of China(2018YFC1504504);Youth Foundation of Yunnan Earthquake Agency(2021K01);Project of Yunnan Earthquake Agency“Chuan bang dai”(CQ3-2021001).