BCI+VR Rehabilitation Design of Closed-Loop Motor Imagery Based on the Degree of Drug Addiction

Xuelin Gu,Banghua Yang,*,Shouwei Gao,Honghao Gao,Linfeng Yan,Ding Xu,Wen Wang

1 School of Mechanical and Electrical Engineering and Automation,Shanghai University,Shanghai 200444,China

2 School of Computer Engineering and Science,Shanghai University,Shanghai 200444,China

3 Department of Radiology&Functional and Molecular Imaging Key Lab of Shaanxi Province,Tangdu Hospital,Fourth Military Medical University,Xi’an,Shaanxi 710038,China

4 Shanghai Drug Rehabilitation Administration Bureau,Shanghai 200080,China

Abstract: After abusing drugs for long, drug users will experience deteriorated self-control cognitive ability, and poor emotional regulation.This paper designs a closed-loop virtual-reality(VR),motorimagery (MI) rehabilitation training system based on brain-computer interface (BCI) (MI-BCI+VR), aiming to enhance the self-control, cognition, and emotional regulation of drug addicts via personalized rehabilitation schemes.This paper is composed of two parts.In the first part, data of 45 drug addicts (mild:15; moderate: 15; and severe: 15) is tested with electroencephalogram (EEG) and near-infrared spectroscopy (NIRS) equipment (EEG-NIRS) under the dual-mode, synchronous signal collection paradigm.Using these data sets, a dual-modal signal convolutional neural network (CNN) algorithm is then designed based on decision fusion to detect and classify the addiction degree.In the second part, the MIBCI+VR rehabilitation system is designed,optimizing the Filter Bank Common Spatial Pattern(FBCSP)algorithm used in MI, and realizing MI-EEG intention recognition.Eight VR rehabilitation scenes are devised,achieving the communication between MI-BCI and VR scene models.Ten subjects are selected to test the rehabilitation system offline and online, and the test accuracy verifies the feasibility of the system.In future,it is suggested to develop personalized rehabilitation programs and treatment cycles based on the addiction degree.

Keywords:drug addiction degree;brain-computer interface,motor imagery;virtual reality;drug addiction rehabilitation

I.INTRODUCTION

Drug use not only undermines health,but brings about many social problems.The mental and physical dependence on drugs is an essential obstacle that prevents the addicts from getting rid of drugs.Traditional drug rehabilitation schemes, such as psychological remedies,education correction,sports training,and labor,are not productive and often breed boredom.The drug therapies have severe side effects and often lead to a new form of dependence.The physical therapy Transcranial Magnetic Stimulation(TMS)is dangerous and not widely applicable.Based on the degree of drug addiction,this paper designs the MI-BCI+VR rehabilitation system, which allows for personalized treatment.

The Electroencephalogram(EEG)is a type of spontaneous bioelectrical signal captured and recorded on the surface of the scalp.It has the characteristics of being noninvasive,no radiation damage,and low cost[1–3].Near-infrared spectroscopy technology uses light injection and detection points to measure the hemodynamic function of brain tissue, and can record blood oxygen levels [4–6].EEG and NIRS measuring devices are small in size, and movable.The measurement process is non-invasive and non-radioactive.The combination of EEG and NIRS can better detect the functional activities of brains,and is therefore an ideal choice to measure the degree of drug addiction[7,8].

The BCI technology is supported by neuroplasticity.It works out by activating the neurons in dormant state based on BCI feedback, forming a nerve conduction pattern.BCI decodes the activity signal of the central nervous system, and converts the signal into different types of output[9,10].

MI provides better rehabilitation training for the drug addicts [11, 12].The VR technology is used to simulate a three-dimensional virtual world that is immersive,interactive and conceptual[13,14].

Connelly et al.developed a pneumatic data glove with VR technology, with which the patients play games in a virtual environment and recover the motion functions of the palm-wrist joints and the metacarpophalangeal joints[15].Robert Leeb et al.designed a wheelchair control virtual system with MI-BCI and VR technologies.By training EEG,the system is able to distinguish between the motor imagination state and the resting state,carries out the forward and stop commands, and has dialogues with objects in the virtual environment [16].Vourvopoulos et al.came up with the EEG-based BCI-VR system, which uses the MI paradigm for the post-stroke rehabilitation and function evaluation of upper limbs[17].Wen et al.noted that the BCI-VR rehabilitation schemes provide patients with better feedback and facilitate the recovery of brain function[18].Marin-Pardo et al.put forward the REINVENT virtual reality rehabilitation system,and the results showed that the system was well received among stroke patients[19,20].VR scene feedback does a better job in neural activation and is attractive among patients [21, 22].Wang et al.used audiovisual BCI in the clinical evaluation of patients suffering from disorders of consciousness[23].Bundy et al.proved that BCI rehabilitation can be carried out at home[24].

As the traditional drug rehabilitation scheme is not effective, the drug therapies have severe side effects,and the physical therapies could bring harm to the patients,this paper proposes the MI-BCI+VR rehabilitation treatment system,which is safe,fun and effective.In the first part of the paper, the data of 45 drug ad-dicts were collected (mild: 15, moderate: 15, severe:15) in line with the designed experimental paradigm,and an algorithm used to test the degree of drug addiction was proposed.In the second part,the FBCSP motor imagery EEG recognition algorithm is optimized.Eight VR rehabilitation scenes are devised, and the communication between MI-BCI and the VR system is achieved.The offline and online tests among subjects verify the feasibility of the MI-BCI+VR system.

II.MATERIALS AND METHODS

2.1 Subjects Selection Criteria

Study participant criteria: 1.Meet the Diagnostic and Statistical Manual of Mental Disorders; 2.patients within six months of a withdrawal period;3.junior high school education and above; 4.age 18–41 years; 5.voluntary study participation and willingness to sign the informed consent form.Exclusion criteria: 1.Severe cognitive dysfunction, inability to cooperate and complete project-related assessment and testing; 2.patients with severe physical diseases;3.patients with severe psychotic symptoms; 4.current participation in other psychological interventions and treatments; The study was conducted following the declaration of Helsinki and was approved by the Ethics Committee of Shanghai University (Approval No.ECSHU2020-071).

2.2 EEG Equipment Information

The detailed description of EEG equipment is shown in Table 1.

Table 1. EEG hardware parameters.

2.3 NIRSIT Equipment Introduction

The detailed description of fNIRS equipment is shown in Table 2.

Table 2. NIRSIT hardware parameters.

2.4 NIRSIT Channel and Functional Area Division

The wearing method of EEG-NIRS is shown in figure 1.Considering the advantages of dual-mode signal,starting from the actual scientific rationality,combined with the hardware characteristics of EEG and NIRS(EEG-NIRS),the design conforms to the experimental paradigm of synchronous acquisition of two devices.We can get more comprehensive brain function information of drug addicts.

Figure 1. Division of NIRSIT channels and functional areas.

2.5 Experiment Paradigm and Data Collection

We used E-prime software package(Psychology Software Tools, Pittsburgh, PA) to write the experimental paradigm, with each map numbered.Figure 2,and figure 3, are examples of drug abuse-related maps and neutral maps used in the experimental paradigm.figure 4, The whole process of the experimental paradigm.The first stage of the experimental paradigm, 10 minutes in total, during which the subjects need to close their eyes for 5 minutes and then open their eyes for 5 minutes.The second stage, it lasts 6.28 minutes and is divided into drug maps and neutral maps.Among them, each block lasts 9.6 seconds.There are a total of 16 maps, and the display time of each map is 0.6 seconds.At the beginning,the first four maps are displayed randomly, during which there are two drug maps.After displaying the first four maps, the remaining 12 neutral maps are displayed randomly.After a block ends, there will be a 4-second interval map with a white background and a black cross.The third stage, it lasts a total of 4.69 minutes, during which the maps are all neutral, with each block lasting 9.6 seconds.There are 16 maps in total, with a display speed of 0.6 seconds.There will be a 4-second interval between each block.

Figure 2. Examples of drug abuse-related images.

Figure 3. Examples of neutral images.

Figure 4. The whole process of the experimental paradigm.

III.METHODOLOGY

3.1 Classification Model of Drug Addiction Degree

3.1.1 Bimodal Signal Processing Architecture Based on Decision Fusion

Decision fusion means that each signal’s data makes independent decisions first,and then the final decision was obtained with the help of weighted calculations according to the results of independent decisions of each signal.EEG and NIRS data were obtained separately, due to the difference in sampling frequency of EEG equipment and NIRS equipment.After the two types of data were preprocessed by their respective CNN networks, feature extraction, and decisionmaking,the two types of data were then processed by batch normalization at the decision-making level, so that the data becomes a distribution with a mean value of 0 and a standard deviation of 1.

3.1.2 CNN Classification Model Based on Decision Fusion

Figure 5,Module 3,normalization of EEG and NIRS output characteristics: EEG characteristics: batch normalization level.The EEG features were normalized to optimize the one-dimensional feature distribution.Near-infrared characteristics: batch normalization level.The near-infrared features were normalized to optimize the one-dimensional feature distribution.Weight distribution: after normalization, EEG and NIRS data of the same dimension and the same amount of data were obtained.At the same time,50%weight was given to each type of data.The data was processed and output to the full connection level.Fully connected layers: One-dimensional feature distribution was searched and finally classified.

Figure 5. EEG-NIRS addiction degree classification framework.

3.2 MI-BCI+VR System Design

3.2.1 MI-BCI Combined with VR Rehabilitation System

The rehabilitation training system based on MIBCI+VR technology includes a closed-loop system with feedback composed of three parts: signal acquisition, signal processing, and application system.(1)Signal acquisition:EEG acquisition equipment is used to collect EEG generated by the user’s motor imagination.(2) Signal processing: the collected MI-EEG is used to identify the subject’s “thinking intention,”which usually includes three parts:preprocessing,feature extraction, and feature classification.(3) VR rehabilitation training system: MI-BCI system is connected through the TCP/IP communication protocol to receive control commands from signal processing results transformation and make corresponding feedback.

3.2.2 Rehabilitation Principle of Motor Imagery Brain–Computer Interface(MI-BCI)

MI-BCI is a branch technology of BCI based on MI therapy.The rehabilitation principle of MI-BCI is similar to normal mental consciousness; we can imagine left-hand,right-hand,and feet in the brain through MI.An EEG equipment is used to collect real-time EEG of drug addicts during MI, while BCI technology is used to process the collected MI-EEG(preprocessing,feature extraction,and feature classification),identify drug addicts’ movement intentions, and convert them into virtual reality(VR).The control commands of the model in the scene control the actions of the scene model.

3.2.3 Design of BCI Feedback System Based on VR

The BCI system structure diagram based on VR feedback is shown in figure 6.The system consists of two parts: BCI signal acquisition and processing system and VR feedback system.Among them,the signal acquisition and processing system converts the obtained EEG signal into a real-time control signal between-1 and 1 after signal processing,and stores it in the access database.VR feedback system calls control signals in access database in real-time and guides virtual people to do corresponding actions.Meanwhile, subjects receive feedback from virtual people on the computer screen.The subjects can train many times and adjust the way of motor imagination in real-time, to master the strategy of producing the best EEG signal.

Figure 6. VR feedback system structure.

3.2.4 TCP/IP Communication Design

The rehabilitation system includes online and offline training stages, which need to communicate with the signal acquisition and processing system.The system uses the TCP/IP communication protocol.When using TCP/IP for communication,the drug rehabilitation training system designed in this paper is used as the server-side;the signal acquisition and processing system is used as the client-side;and the socket is used as the programming port between the two.This system uses TCP/IP blocking mode connection.

3.2.5 Overall Architecture of the MI-BCI+VR Rehabilitation System

There is no scientific research team to study the application of BCI in drug rehabilitation in terms of training for self-control, attention, and emotion of drug addicts.The development of a program that considers therapeutic effect and entertainment and enables drug addicts to actively participate in the training has become the pursuit of drug rehabilitation in the new era.Figure.7 presents MI-BCI+VR closed-loop motor imagery rehabilitation architecture.This includes the abuse evaluation module of different levels of drug addiction (First), EEG motor imagination modeling(Second), and a closed-loop feedback training module (Third).The evaluation module of different addiction degree includes EEG-NIRS data collection 1.The EEG-NIRS equipment is used to collect and analyze the EEG and NIRS data according to the experimental paradigm.Drug addicts are divided into three categories: mild, moderate, and severe, which are transmitted to the VR training scene model 3 (A.Horror scene; B.Warm scene; C.Hamster game; D.Parkour; E.Object grasping; F.Catch butterfly; G.Maze;H.Racing Car).EEG motor imagery modeling(Second)includes MI control command classification 2 and VR training scene model 3.MI control command classification 2; EEG data preprocessing; feature extraction;improved FBCSP algorithm classification; the recognition of left hand,right hand,and feet motor imagery tasks are entered into the corresponding control command through the TCP/IP communication protocol sent to the VR rehabilitation training system to control the VR scene model action.Closedloop feedback training module(Third), real-time collection of drug addicts in-game training,and the EEG and NIRS real-time change information are displayed on the computer.The data change real-time feedback shows whether the game control is correct and reminds the focus of motor imagination.In this way,the brain function of drug addicts can be trained,which can improve self-control and drug refusal.

The Princess was much agitated27 by this speech, and feared lest the Enchanter should have overheard it; but he had been loudly calling her attention to the flowers, and chuckling28 over his own smartness in getting them for her; and it was rather a blow to him when she said very coldly that they were not the sort she preferred, and she would be glad if he would send them all away

3.3 MI-BCI+VR Algorithm Design

3.3.1 MI Experimental Paradigm Design Before performing MI and precise control model actions,each drug addict needs to be trained according to the designed MI experimental paradigm.After training,they can output precise control commands.figure 8 is a complete flow chart of the trial,which includes 2s of lead time, 4s of motor imagination time, and 2s of rest time.During the 2s guide time of each trial,the guided movement of the left-handed fist or righthanded fist will appear randomly in the VR scene,and four complete fist movements will be completed at a certain frequency within 2s.After the guided action is over,the 4s motor imagination task stage is entered,and the subjects will perform motor imagination according to the guided action and finally enter the 2s rest time.

Figure 7. MI-BCI+VR closed-loop motor imagery rehabilitation architecture.

Figure 8. The flow chart of a complete trial in the experimental group(with VR guidance).

3.3.2 MI Control Command Decoding Algorithm

The current decoding algorithms is optimized filter bank common spatial pattern (FBCSP).This article uses an optimized FBCSP, an MI-EEG decoding algorithm, and its analysis process includes five stages:motor imagery time window optimization,filter bankbased frequency band filtering, CSP spatial filtering,mutual information-based feature selection,and Na¨ıve Bayesian Parzen Window Feature Classification [25–27].

3.3.3 Band Filtering Based on Filter Banks

According to the experimental paradigm of MI, first,the data is intercepted in the 0–4 second time window in the EEG.After time-domain filtering,frequency domain filtering is performed.In this paper, the maximum attenuationApassallowed by passband and the minimum attenuationAstopallowed by stopband of bandpass filter were optimized by grid search.We set the range ofApassto[1,4]and that ofAstopto[20,40].We optimized the grid search and finally output the highest classification accuracy and its corresponding values ofApass,Astop.

3.3.4 CSP Spatial Filtering

After the band filtering is completed, the correspondingEbis obtained, which is spatially filtered using the Common Spatial Pattern(CSP)algorithm[28,29].The MI-EEG obtained from theb(1 ≤b≤9)bandpass filter is usedEbIndicates that each type of MI task containsntrial data.The process based on CSP is as follows:

The trace(X) represents the sum of diagonal elements of matrix X, and T is the transpose operator.(2) Through the average covarianceCbc,i,andCbc,jof left-hand and right-hand motion imagination,obtain the mixed space covariance of all trialsCbc:

(3) Eigenvalue decomposition of the mixed space covariance matrix:

Where,Ubcis the eigenvector matrix and Λbcis the eigenvalue diagonal matrix.(4) Construct whitening matrixPband spatial coefficient matrixSb: (3)Eigenvalue decomposition of the mixed space covariance matrix:

(5)The same eigenvectors ofSblandSbrare used to construct the matrix B to decompose it:

(6) Calculate the projection matrixWbof CSP to obtain the EEG signal after spatial filteringZb:

After obtainingZb, extract the first m lines and the last m lines (2m

Among them,var(X) represents the variance of the calculated sample X.By using CSP feature extraction,the final feature vectorfb={fb1,fb,2,...,fb,2m},where m is the logarithm of the selected feature.

3.3.5 Feature Selection Method Based on Mutual Information

After CSP feature extraction,the feature vector of each sub-band isfb={fb1,fb,2,...,fb,2m}(1≤b ≤9).The feature selection method used in this paper is mutual information-based best individual feature(MIBIF)algorithm[30,31].The feature selection using MIBIF are as follows:

(1)Initialize the feature set withdfeatures asF=[f1,f2,...fd](form:d=9∗2m), and initialize the selected feature set with mutual informationS=ϕ.

(2)Suppose class label setω=1,2,and use Parzen window to estimate conditional probabilityp(fj,i|ω):

Where,yrepresents the real class label,σrepresents the standard deviation of y,ϕis a smoothing kernel function with a smoothing parameter ofh.nωis the number of tracks in the training data belongs to theω.

(3)classIωIs the index set of training data belonging toωclassfj,rAndfj,iare therth eigenvalue of theith andjth trail,respectively.

Where,p(ω|fj,i) is the conditional probability of the classfj,iof the givenω;p(fj,i|ω) is theωconditional probability of thefj,iof the given class;p(ω)is the prior probability of the givenωclass.

(4) CalculateH(ω) and conditional entropyH(ω|fj):

Wherefj,iis theith feature value of thejth trail;ntrepresents the total number of trails in the training data.

(5)Calculate the mutual information of each featuref ∈F:

Assuming that there areNlike motor imagery tasks in common, Ω ={1,...N}is a collection of all class labels.

(6) Select the firstkfeatures.Repeat the work of selecting featuresfjuntil|S|=k.

3.3.6 Na¨ıve Bayesian Parzen Window(NBPW)Classification

As we only needed to recognize EEG under two classification tasks(left hand and right hand,left hand and feet), we selected the NBPW classification algorithm based on the Parzen window[32,33].

The probability density was estimated based on the Parzen window,and then the posterior probability ofωclass in the sample was calculated based on Bayesian.We took the maximum value inp(ω|fj,i)to obtain the final classification resultω:

IV.RESULTS

4.1 Drug Addiction Degree,Classification Accuracy

In the CNN drug addiction degree classification model,EEG selected 32 electrodes and fNIRS selected 16 channels.According to the experimental paradigm,each subject has 56 trials.The algorithm model data was selected as a time window of 1.125 s after the drug map appeared.

Based on the EEG-NIRS bimodal signal,the data of 45 drug users were collected.The data of 27 of them were selected as the training set, 9 as the validation set,and 9 as the test set.The classification of the drug addiction degree was achieved, and finally, the average 3-class accuracy result is stable between 67%and 80%.

4.2 MI-BCI+VR System Offline Test

To verify the feasibility of the MI-BCI+VR addiction rehabilitation training system, 10 subjects (6 males and 4 females) with good health and an average age of 26 years were selected for offline tests and online tests.The subject IDs were recorded as S1–S10.

The experiment was divided into two stages: offline training and online training.Each subject was required to participate in both the trainings.The offline stage experiment only required participants to perform the left hand and right hand two class motion imaginations.The results of the offline stage of the rehabilitation training system are depicted in Table 3.

Table 3. Experimental results of the offline phase of the rehabilitation training system.

4.3 MI-BCI+VR System Online Test

Online training for each subject was completed under the VR guidance + feedback experiment.In the 2-s prompt time, a left or right arrow randomly appeared on the screen.After the prompt was over,the 4-s motor imagination task time was entered,and the subject performed left hand or right hand two class motor imagination according to the prompt.During the last 2-s rest time,the MI-EEG was analyzed and recognized in real-time and then converted into control commands,which were then transmitted to the VR scene.The corresponding feedback appeared on the VR model.The results of the online stage of the rehabilitation training system are depicted in Table 4.

Table 4. Experimental results of the online training of the rehabilitation training system.

V.DISCUSSION

This paper designs a dual-modal signal detection algorithm on the degree of drug addiction, optimizes the FBCSP algorithm,classifies the MI-EEG intention classification,designs 8 VR scenes popular among the subjects,and encourages active participation.The MIBCI+VR system proves feasible based on the online and offline test among subjects.

The degree of drug addiction is measured based on the oral statements of drug users,statistical scales,and advices from hospital experts;and then classified into mild,moderate and severe degrees.CNN classifies the addiction degree based on EEG-NIRS data,overcoming the shortcomings of subjective measurement.

The Filter Bank Common Spatial Pattern (FBCSP)is optimized for MI-EEG analysis[34,35].Parameterspecific optimization is carried out in filter design and feature selection, thus better classifying the subjects based on MI-EEG data.

BCI is a new type of neural rehabilitation training method, which plays an active role in patients with dysfunction[36,37].During neural rehabilitation training,effective feedback can adjust and improve the corresponding brain activities [38, 39].The impact of VR guidance and feedback on the offline and online MI-EEG recognition of rehabilitation is analyzed.Rehabilitation training based on the MI-BCI and VR technologies take place both offline and online, and subjects are required to generate EEG signals via active motor imagery.

The limitation with this research is that the rehabilitation system proves effective after being tested among healthy subjects, but has not been put in clinical applications among drug addicts.Next, the system will be applied among drug addicts, so as to evaluate its rehabilitation effects.This research is meaningful because it achieves faster and more accurate diagnosis of the degree of drug addiction based on CNN.The MIBCI+VR rehabilitation system is safe and fun, with profound practical values.

VI.CONCLUSION

This research concludes that it is possible to classify the drug addiction degree.The MI-BCI+VR rehabilitation system proves stable and reliable after online and offline test.This makes it a reality to customize rehabilitation schemes and cycles based on the addiction degree, thereby solving the issue of untargeted rehabilitation schemes and encouraging participation.The rehabilitation can take place both at the recovery centers and home.

Existing studies focus on the effect of psychological treatment schemes, drug therapies, and physical therapies.The rehabilitation system designed in the paper has been verified after online and offline testing, and proves productive in recovering self-control and cognition, getting rid of addiction, and boosting moods.The MI-BCI+VR rehabilitation system is innovative because it is not only safe, but attractive to the users,featuring readily applicable features and enjoying bright prospects.

ACKNOWLEDGEMENT

This project was supported by Key Research & Development Project of National Science and Technique Ministry of China (No.2018YFC0807405,No.2018YFC1312903), Defense Industrial Technology Development Program(JCKY2019413D002),National Natural Science Foundation of China (No.61976133).