Dynamic Vision Enabled Contactless Cross-Domain Machine Fault Diagnosis With Neuromorphic Computing

Xinrui Chen , Xiang Li , Shupeng Yu , Yaguo Lei ,Naipeng Li , and Bin Yang

Dear Editor,

This letter presents a novel dynamic vision enabled contactless cross-domain fault diagnosis method with neuromorphic computing.The event-based camera is adopted to capture the machine vibration states in the perspective of vision.A specially designed bio-inspired deep transfer spiking neural network (SNN) model is proposed for processing the event streams of visionary data, feature extraction and fault diagnosis.The proposed method can also extract domain-invariant features from different machine operating conditions without target-domain machine faulty data.Experiments on rotating machines are carried out for validations of the proposed method, and the proposed method is verified to be effective in contactless fault diagnosis.

Related work: Rotating machines are widely used in the field of industrial manufacturing [1], [2].Once the rotating machine fails, it will affect the overall performance of the mechanical equipment and even cause serious safety accidents.Therefore, it is particularly important to develop efficient fault diagnosis method for rotating machines [3], [4].

In the past years, a large number of machine learning-based methods have been proposed to solve the fault diagnosis problems of rotating machines [5]-[7].Not only that, deep learning-based methods are also developing rapidly [8]-[10].However, the existing methods for capturing the vibrations of rotating machines still have significant limitations in deployment in industrial applications.Recently, event-based cameras have been used to capture the vibrations of rotating machines.Different from traditional vision-based fault diagnosis tasks [11], event vision-based methods have more advantages, but there are very few related studies.Liet al.[12] used event-based cameras to conduct contactless fault diagnosis of rolling bearings and achieved reliable results.However, this method still uses traditional neural network as the feature extraction network and loses the temporal characteristics of the data.

The SNN is a special bio-inspired structure that can process the temporal characteristics contained in the data and has lower energy consumption.Therefore, this special structure has good application prospects in engineering [13], [14].Xuet al.[15] introduce attention mechanism into SNN for bearing fault diagnosis.Zhanget al.[16]completed end-to-end model training on the rolling bearing dataset using SNN with convolution.Although the existing researches on SNN have achieved some results, the advantages have not been fully demonstrated.

Problem statement: Intelligent fault diagnosis algorithms for rotating machines have achieved great success in recent years.The most popularly used signal for fault diagnosis is the vibration acceleration data collected from contact accelerometers.However, the contact accelerometers have significant limitations in deployment in industrial applications and the other existing contactless sensors are often costly or ineffective.Meanwhile, the common deep neural net-

Corresponding author: Xiang Li.

Citation: X.Chen, X.Li, S.Yu, Y.Lei, N.Li, and B.Yang, “Dynamic vision enabled contactless cross-domain machine fault diagnosis with neuromorphic computing,”IEEE/CAA J.Autom.Sinica, vol.11, no.3, pp.788-790, Mar.2024.

The authors are with the Key Laboratory of Education Ministry for Modern Design and Rotor-Bearing System, Xi’an Jiaotong University, Xi’an 710049,China (e-mail: chenxinrui@stu.xjtu.edu.cn; lixiang@xjtu.edu.cn; yushupeng@stu.xjtu.edu.cn; yaguolei@mail.xjtu.edu.cn; naipengli@mail.xjtu.edu.cn; binyang@xjtu.edu.cn).

Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/JAS.2023.124107 work-based methods have high requirements on computations.That makes the current mainstream intelligent fault diagnosis methods less applicable in the real engineering problems.In addition, it often occurs that the distribution of training data (source domain) and testing data (target domain) are inconsistent [17], and usually the trainable parts of the target domain do not contain faulty data.In order to solve the above problems, we propose a dynamic vision enabled contactless cross-domain method with neuromorphic computing for fault diagnosis of rotating machines.

Intelligent fault diagnosis method: In this letter, we use the event-based camera to capture vibrations.The event-based camera records the light and shade changes of the target area and outputs them in asynchronous event streams.The event streams can be represented as, The individual eventeican be expressed as

wheretirepresentsthe timewhenthei-theventoccurs,xiandyirepresentthex-axisand y-axispositionswheretheeventoccurs.pirepresents the polarity of the event,pi=1 indicates that the brightness of the position increases, andpi=-1 indicates that the brightness of the position decreases.The shape of a single sample is (Vt,Vx,Vy), whereVtis the time step of the event, which represents the time length of

the sample, andVxandVyare the lengths of the x and y-directional,respectively, which are related to the selected RoI.

After that, a bio-inspired SNN model is proposed for processing the event streams.Compared with the traditional neural network, the SNN has a more efficient and energy-saving structure.The SNN processes data in a completely new way, specifically, the spiking data are all in the form of spikes of 0 and 1.As one of the most popular spiking neurons, leaky integrate and fire (LIF) neurons are often used for computation of spiking data,

wheretrepresents the time step, τ is the time constant,uandOsrepresent the membrane potential and output of the LIF neuron,ur1is the resting potential of theLIF neuron membrane,ur2is the reset potential of the LIF neuron membrane, and ωmis them-th The weight of each synapse,Tωis the integration time window,tmnis theTωwindow,K(·) represents the delay kernel function,uthis the ignimoment when then-th pulse of them-th synapse is excited within the tion threshold.

In this letter, SNN with convolution (SCNN) is used as the feature extraction network.The parameters of the SNN model designed in this letter mainly refer to the structures of the existing traditional convolutional networks which have been widely proven to be effective.Specifically, the SNN proposed in this letter contains a bottleneck block for feature extraction, and a classifier for feature classification.As shown in Table 1, we first use two convolutional layers for feature extraction, and two max-pooling layers for feature compression.Next, the features are flattened and output through two linear layers for feature classification, and the activation function in SCNN is replaced by the spiking layers.

In rotating machine fault diagnosis, using effective domain adaptation methods to narrow the distribution difference between the source domain and the target domain often achieves satisfactory results [18].In this letter, the maximum mean difference (MMD) between the source domain and the target domain is first calculated.However,compared with conventional data types, the output of SNN has one more time dimensionVt.Therefore, this letter combines the time dimension of the SNN outputs,

wherevrepresents the feature value, andstrepresents the spikes of the SNN output at timet.Next, the MMD can be calculated,

Fig.1.Flowchart of the proposed contactless intelligent fault diagnosis method for rotating machines.

Table 1.The SNN Model Proposed in This Letter

where Hkrepresents the reproducing kernel Hilbert space (RKHS)with characteristic kernelk.Therefore, the objective function to calculate the distribution difference between the source domain and the target domain can be defined as

whereS f(j)andT f(j)represent thej-th layer features of the source domain and the target domain, respectively.In addition, this letter proposes a deep distance metric learning method inspired by [18], the metrics we use include inter-class separabilityLinterand intra-class compactness respectivelyLintra,

wheref(m)x(i)represents the characteristics of thei-th type of data after them-th layer.clsrepresents the number of categories.The objective function for metric learning can be defined as

In summary, this letter mainly proposes a novel dynamic vision enabled contactless cross-domain fault diagnosis method with neuromorphic computing, the overall fault diagnosis process is shown in Fig.1.The event-based camera is adopted to capture the vibrations of rotating machines, and a bio-inspired deep transfer SNN model is proposed for processing the event streams.

Experimental study: The method proposed in this letter is verified on the rolling bearing test rig, as shown in Fig.2.The test rig is driven by a motor and the motor drives the shaft through a coupling.The bearing model on the shaft is ER-16K, and the event-based camera is placed in front of the bearing, The event-based sensor used in this letter is Prophesee 3.1.

Fig.2.The appearance and structure of the rolling bearing test rig.

This experiment includes four types of states of rolling bearings in total, including healthy, outer ring fault (Outer), inner ring fault(Inner) and rolling element fault (Ball).The source domain data is collected at a rotational speed of 40 Hz, and the target domain data is collected at 30 Hz.The settings when generating samples are as follows.The number of time steps is 10, the time length of a single time step is 1ms, the RoI size is 30×30.In this experiment, a total of 1000 samples are generated for each class, the size of training sets is 800,and the size of testing sets is 200.

In order to verify the effectiveness of the method proposed in this letter, we have completed different methods at the same time as a comparison.Specifically, this leftter also studies the following methods.

1) CNN only: This method is a basic control experiment.We only use CNN as the basic feature extraction network, and the domain adaptation method is not used.

2) SNN only: In this method, the structure of the CNN network is converted to the SNN used in this letter, so as to form a contrast, and the domain adaptation method is not used for model training as well.

3) TCNN: This method uses the mainstream CNN network and combines the domain adaptation method, in order to verify the competitiveness of the method used in this letter.Therefore, except that we replaced the SNN in the proposed method with CNN, the other parts are exactly the same.

The training accuracies of the four methods are shown in Fig.3.Each type of method was conducted three times, and the experimental results are shown in Table 2.Fig.4 uses tSNE to intuitively show the distribution of features extracted by the four types of methods in the source and target data.Fig.5 is an intuitive comparison of the spiking outputs and the corresponding label.

From the experiments in this letter, it can be concluded that, first of all, the method proposed in this letter is highly competitive compared with the traditional CNN-based method.The average precision of the method proposed in this letter can reach 98.12%, slightly higher than 95.99% of the CNN-based method, and the convergence speed during training is faster.In addition, the SNN-based crossdomain fault diagnosis method used in this letter improves model performance under different rotating speeds.Compared to the methods without cross-domain algorithms, the average accuracy proposed method increases by around 13%.Moreover, the method proposed in this letter has a good application prospect.Due to the advantages of SNN, the proposed method can be much more energy efficient compared with traditional CNN-based methods.

Fig.3.The trend of testing accuracy with the number of epochs.

Table 2.Comparison of the Accuracies of Different Methods

Fig.4.The tSNE dimensionality reduction visualization diagram of different experimental results.

Fig.5.Comparison of ground truths and model prediction.

Conclusions: This letter presents a novel dynamic vision enabled contactless cross-domain fault diagnosis method with neuromorphic computing.The event-based camera is used to capture the vibrations of the rotating machines, and the vibration signals are processed by a specially designed SCNN.In addition, we propose a specially designed SNN-based cross-domain fault diagnosis method to achieve cross-domain fault diagnosis for rotating machines without faulty data from the target domain.Finally, the method is verified on the rolling bearing test rig.Compared with the current mainstream CNNbased fault diagnosis methods, the method in this letter has strong competitiveness, and proposes a very effective direction for contactless vision-based fault diagnosis.

Acknowledgments: This work was supported in part by the National Key R&D Program of China (2022YFB3402100) and the National Science Fund for Distinguished Young Scholars of China(52025056).