Human Stress Recognition from Facial Thermal-Based Signature:A Literature Survey

Darshan Babu L.Arasu,Ahmad Sufril Azlan Mohamed,★,Nur Intan Raihana Ruhaiyem,Nagaletchimee Annamalai,Syaheerah Lebai Lutfi and Mustafa M.Al Qudah

1School of Computer Sciences,Universiti Sains Malaysia,Penang,11800,Malaysia

2School of Distance Education,Universiti Sains Malaysia,Penang,11800,Malaysia

ABSTRACT Stress is a normal reaction of the human organism which triggered in situations that require a certain level of activation.This reaction has both positive and negative effects on everyone’s life.Therefore,stress management is of vital importance in maintaining the psychological balance of a person.Thermal-based imaging technique is becoming popular among researchers due to its non-contact conductive nature.Moreover,thermal-based imaging has shown promising results in detecting stress in a non-contact and non-invasive manner.Compared to other non-contact stress detection methods such as pupil dilation,keystroke behavior,social media interaction and voice modulation,thermal-based imaging provides better features with clear boundaries and requires no heavy methodology.This paper presented a brief review of previous work on thermal imaging related stress detection in humans.This paper also presented the stages of stress detection based on thermal face signatures such as dataset type,thermal image face detection,feature descriptors and classification performance comparisons are presented.This paper can help future researchers to understand stress detection based on thermal imaging by presenting the popular methods previous researchers use for stress detection based on thermal images.

KEYWORDS Stress state; stress recognition; skin temperature; thermal signature; thermal imaging

1 Introduction

The word “stress” is described in many contexts [1].An inclusive definition of stress refers to the biological response to a physiological or psychological stimulus [2].Emotional and physical stressors can be detrimental to the human body.The effects of stress on human wellbeing and symptoms have been extensively researched in recent times [3-9].Kim [10] found that people in their 30 s experience the highest stress level due to mask-wearing.The authors reveal that the early stress detection techniques depend on psychological questionnaires [11] and consultations [12].

In recent years,researchers have been exploring the non-invasions method to detect stress.Skin temperature is one of the established stress markers based on physiological signals.The amount of heat dissipated by the body has the capacity as a tool to measure the temperature of the human skin.Body temperature is affected by blood flow,metabolic activities,subcutaneous tissue structure,sympathetic nervous (SNS)activities,and muscle contractions [13-15].The healthy people’s body temperature was recorded between 35.5°C and 37.7°C under normal conditions.The human body can regulate body temperature to keep it stable [16].A noticeable rise in core body temperature may indicate an illness such as fever or hypothermia and a change in the human affective state [17].Hypothalamus is a part of the brain located at the brain base responsible for regulating body temperature.Sometimes it may fail to function well under abnormal conditions [18,19].However,neglecting treatment of the symptom of continuously high body temperature may lead to harmful consequences; injures body organs.Another factor that affects human temperature is muscle contractions which generate heat through muscles movement [20].The internal body heat is transferred from the internal issue to the human skin via the blood supply in the vascular system.The control of blood flow in the skin by the processes of vasoconstriction and vasodilation processes is part of the thermoregulatory mechanism,i.e.,the thermal homeostasis of internal body temperature to external factors such as cold and heat [18,21].The skin surface is an essential body part in regulating core body temperature:the body heat is transferred to the skin via internal vessels and the skin loses the heat in several ways:evaporation,thermal radiation,conduction,and exhalation through a respiratory process [22-25].

Pavlidis firstly discovered the alarming thermal signature and announced an increment of blood demand in the periorbital region [26-30].Consequently,it contributes to effective feature extraction,thus provides extensive research on the physiological signals of the human body such as breathing [28],sweating [30],blood flow velocity [31,32],and heart rate [29].Table 1 summarized the commonly used model in literature that detects human stress.

Table 1:Literature work of human stress state recognition

(Continued)

Table 1(Continued)AuthorModalFeature[43]Skin temperatureSlope of skin temperature Mean of skin temperature Temperature deviation Temperature standard deviation[43]Galvanic Skin Resistance (GSR)Number of response Mean value of GSR Amplitude of response Rising time of response Energy of response[44]Heart rateMean Deviation Deviation squared?

Many studies have explored the possibility of stress detection with facial skin temperature[45-47].Researchers also investigated other modalities such as pupil dilation,breathing pattern,behavior pattern,keystroke pattern,and social media activity.In [25,41,42],the authors proposed a novel method to detect stress based on facial expressions.The results demonstrated that the proposed method has similar accuracy performance to other state-of-the-art methods.However,this method is proposed based on RGB images.Similar method can be proposed for thermal images as a new potential research direction.Compared to other modalities,thermal-based stress detection provides a reliable accuracy rate concerning user privacy.To investigate the facial skin temperature for human stress recognition,Al Qudah et al.[48] explored the recent use of thermal imaging in distinguishing human affective states and the problems that have surfaced.The authors also suggested a framework for solving the issues discussed and the mentioned challenges.Therefore,this paper discussing previous literature regarding facial detection in the thermal image and stress recognition based on the facial thermal signature.More importantly,this paper will reveal the future research potential and challenges faced by the previous work and the solution proposed to overcome these limitations.

The following is the structure of this paper:Section 2 will discuss the stages in thermal-based stress state detection such as dataset type,face detection method,facial temperature as stress signature,and stress classification performance comparison.Section 3 will detail the modalities listed above.Section 6 will propose future work to extend the knowledge of stress detection based on thermal imaging.

2 Thermal Based Stress State Recognition

Thermal imaging is one of the popular topics among researchers to detect stress in a non-invasion manner.Thermal camera technology was initially unfeasible due to its low resolution,high cost,and heavyweight,combined with the ability to regulate the surroundings for a steady ambient temperature [49].Thermal system inventions further paved the way for new varieties of accessible and adaptable thermal sensors that are lightweight,low-cost,and have high resolutions,such as handheld thermal sensors.As a result,sophisticated thermal sensors encourage researchers to investigate thermal imaging in laboratories and real-world settings in many applications,including human stress recognition [50].Moreover,the COVID-19 pandemic enhanced the ability of thermal sensors.It can detect human face temperatures in a non-invasive and contactless manner.ANS is responsible for coordinating human physiological signals such as heart rate,respiration rate,blood perfusion,and body temperature during human stress state from a psychophysiological standpoint.Thermal imaging can measure the temporal temperatures of the face [51].The usage of thermal imaging is a realistic solution to achieve stress detection in a contactless manner.

Several studies have also investigated thermal imaging to explore other psychological signals that correlates with human stress states like respiration rate,pulse rate,and skin temperature.It has the potential to transcend the limitations of contact-based and intrusive physiological sensors [52].When thermal imaging to visual (RGB)imaging,studies have shown that thermal images have many benefits over RGB images.The variation of human skin colour,facial structure,texture,ethical contexts,cultural distinctions,and eyes could affect the accuracy of the human emotion includes stress state applied visual-based methods.Visual-based systems are also sensitive to illumination change.In unregulated settings,visual-based imagination techniques have unreliable recognition precision [52].Thermal imaging,on the other hand,is light-resistant and can be used in low-light situations.The connection between the human stress state and the variety of skin temperature is confirmed with the thermo-muscular and hemodynamic-metabolic components [16,23].Researchers have focused on thermal imaging that encourages them to gauge the transient temperature esteems from the selected facial region to detect the stress state.In the literature,a temperature difference between the left and right sides of the face,and temperature change in the periorbital and nasal facial regions,has been linked to human stress.

However,the alteration in blood flow in the periorbital area allows measuring both instantaneous and prolonged stress conditions [51].This method is a procedural step where begins with a thermal dataset or thermal signature data collection activity.Thermal imaging is typically used in laboratory experiments to collect data.Stress stimulation is applied to cause participants to become stressed,and their facial thermal signatures are measured.Subsequently,the preprocessing and feature extraction approach is adopted to extract facial features and classify them with a classification procedure.However,the accuracy of the performance for each method is varied.The accuracy of the approach also depends on the number of chosen criteria,the feature descriptor and the classification method.Table 2 summarized the previous work related to human stress detection based on facial skin temperature based on the timeline.

Table 2:Literature work of thermal-based human stress state recognition

(Continued)

Table 2(Continued)AuthorObjectiveYear[56]Bashar attempted to study thermal variation in the periorbital region as a candidate for better deception detection.[57]Jenkins and Brown studied the three-way correlation between forehead temperatures,frontal EEG and self-report.The findings established that the right-domain group has a remarkable correlation while no correlation for the left-domain.This reveals the insight of the temperature variation dependent.[58]Sorostinean et al.studied the temperature variation of a subject when engaging with a robot.The study shows little correlation is proved with stress and claimed more studies are needed to support the findings.2014 2014 2015[59]Haji proposed a new approach detecting stress in thermal images based on SIFT feature to match RGB and thermal.The findings show that the forehead is highly correlated to stress.2015[60]Abouelenien et al.demonstrated that thermal variations exist in participants and concluded that thermal imaging can be applied for non-invasions stress detections.The authors have also proven that the fusion of thermal features with other physiological traits outperforms stress detection by single sensors.2016[61]Baltaci studied the stress classification-based fusion features of thermal and pupil.2016[62]Kan Hong proposed a method to assess stress in real-time.The study has proven the correlation between stress markers with temperature variation in periorbital.It is proven that thermal imaging has the ability as a stress marker.2017[50]Yho Choo proposed automatic stress recognition thru mobile thermal imaging.In this study,the author proposed Optimal Quantization and thermal gradient flow techniques to extract blood dispersions of the facial capillary.It contributed that have chances to discriminate multiple stress levels.2017[63]Abdelrahman et al.suggested that the possibility to estimate and quantifying the mental load.2017[64]Nilesh explored thermal imaging to classify challenged individuals from deception.2017[65]Vasavi studied the ability of thermal signature to detect psycho psychological of the subject.The authors proposed two methods of signal extraction on two regions.The result proves better accuracy than a similar experiment done by Pavlidis et al.[53].2018

(Continued)

Table 2(Continued)AuthorObjectiveYear[66]Vashni applied regression modelling to identify the most prominent thermal feature for better stress detection.This study achieves better accuracy than a similar study done by her.This study also proved that periorbital is a good candidate as a thermal signature for stress detection.2018[67]Kopaczka conducted an experiment to measure temperature changes over stress stimulation activity based on the GLCM feature.This study established that the temperature variation occurs in the upper lip region when stress invokes.2018[68]Stoynova studied the thermography changes due to student cognitive load during knowledge assessment tests.2018[69]Changjiang investigated the effectiveness of the face temperature as a marker for stress detection.2018[70]Derakhshan attempted to identify good performing machine learning techniques to increase accuracy.His studies add to evidence that thermal imaging holds high accuracy in detecting deception compared to golden physiological measurement.This study also shows that perinasal and chin are the most engaging ROIs during mental state assessment.2020[71]Jared studied the impact of psychophysical stimuli impact on facial thermal emissions.They compared numerical analysis and deep learning methods to classify stress states.2020 2020[72]Reshma proposed a hybrid system to detect stress.2021[73]Kumar proposed StressNet.2021[45]Hong proposed a method to detect physical stress by maximizing thermal signature.

Table 3:Literature work of human stress classification based on facial thermal signature

2.1 Type of Data

In previous literature,there are two types of data identified; static facial image and moving face.Initially,studies begin with static facial image detection and recognize a few regions of interest on the facial.In the study conducted by [45],participants have to restrict their head movement.In the real world,it is an impractical approach to instruct the subject to be static.This limitation emphasizes the importance to track the facial in motion.Several studies focused on automating facial detection in thermal imaging where the participant can move freely.The majority of studies conducted self-experiment to collect their dataset.A laboratory experiment was conducted in [62,68,69,74] to collect thermal images and other physiological signals.

2.2 Methods for Face Detection in Thermal-Based Image

Face detection is the first step in stress detection based on the facial thermal image.Thermal images are commonly used in many circumstances where ordinary perception is limited,hindered,or inadequate.For example,during night surveillance and fugitive searches.The facial detection algorithm in the thermal image was inefficient in the beginning and was also not sophisticated for visual RGB images.Many studies reported this as a limitation that affects the experiment methodology and findings.Several stress detection methods prohibit head movement during the data collection phase.This is because thermal imaging has limitations for many head movements.As in [62],the authors willingly crop regions of interest (ROIs)manually due to this limitation.The common practice in face detection in the scope of stress detection based on the thermal image is knowledge-based techniques feature invariant facial approaches,template matching method,appearance-based method,colour information,and fusion with visible spectrum imaging.In previous literature,the methodology involved in facial detection in thermal imaging is classified as appearance-based [75,76],feature extraction [77-79],fusion with RGB image [80-85]and multimodal analysis.

Zheng [86] proposed the Projection Project Analysis (PPA)algorithm for face detection algorithm.Studies related to facial thermal images adapted this algorithm to detect face regions.Zheng [75] continued the experiment that adapted PPA to detect faces that have eyeglasses.One of the main limitations of thermal imaging is facial occlusion,which may occur from glass opacity in individuals who are eye-glass wearers.Therefore,occlusion will prevent thermal sensors from reading the heat pattern produced in that region.Several models have been introduced to handle such limitations; different facial ROI is being explored to tackle this challenge.Basu et al.[87]proposed a thermal-based occluded images model by applied Kotani Thermal Facial Expression Database (KTFE)as input images and the Viola-Jones algorithm for facial detection.The study applied a median filter to remove noise and Contrast Limited Adaptive Histogram Equalization(CLAHE)for image enhancement.The study selected six facial patches:forehead,eyes,right and left cheeks,nose,and mouth.Hu’s [88] seven-moment invariant methods were chosen for feature extraction and the classification process.The study selected multi-scale SVM to classify four basic emotions and the average accuracy was 87.5%.

A number of studies [45,89,90] applied a state-of-the-art face detection algorithm,the Haarbased Viola-Jones face detection algorithm [91].Reese et al.[90] conclusively proven that learningbased methods; Viola-Jones and Gabor have a high tendency to detect a face,and the Viola-Jones algorithm can be used as mainstream for face detection in thermal images.Basbrain et al.[92] suggested strategies for improving the Viola-Jones algorithm’s face detection performance in thermal images.The findings suggest that the Viola-Jones method with LBP performs significantly better in thermal imagesvs.Haar-like features.The study also found that using the Otsu technique in the pre-processing stage improves the detection rate.In a recent study,Tran et al.[93] proposed to integrate cross-examples into a proposed scheme,which effectively improved the face detection accuracy.Kowalski et al.[94] evaluated three common face detection algorithms,the Viola-Jones,YOLO,and CNN.The deep learning network outperforms the Viola-Jones algorithm.Faster RCNN has better performance with a near-perfect detection rate and a low false detection rate.Researchers presented the bioheat model,a special identification method,in [95],by creating a thinned vascular network,similar to work done in [96].Cho et al.[97] integrated Modified Hausdorff Distance features to improve the precision performance implemented by [95].

Kopazka and his colleagues [98] have proven that a state-of-the-art machine learning algorithm designated for visual images performs better than dedicated algorithms in detecting facial in thermal imaging in terms of accuracy and false-positive rate.Five algorithms were selected;Haar Cascade Viola-Jones,Haar Cascade classifier with local binary patterns (LBPs),Histograms of Oriental Gradients (HOG),the Deformable Parts Model (DPM),and Pixel Intensity Comparisons Organized in Decision Trees (PICO)to compare with two algorithms; Eye Corner Detection (ED)[99],Projection Profile Analysis (PPA)[100] dedicated for facial detection in thermal imaging.These comparative studies exposed the algorithms have a sensitivity towards a change in pose and facial expression as it impacts the accuracy rate.VJ-LBP and HOG produce good similar detection rates.DPM performs best in the detection and false-positive rates at the cost of having the longest computation time.The authors recommend the PICO method performs fast and produce better results.Kopaczka et al.[100,101] proposed one of the earliest works on thermal facial landmark detection based on active appearance models [102].The authors applied PCA to landmark data and trained the AAM model based on dense HOG and SIFT features.Chu et al.[103] tested a theory by applying an image transfer model by transferring a thermal image to visible and used one of the landmark detectors designated for visual images.The result indicated that the dedicated facial landmark detector need for thermal imaging.This leads the authors to propose a thermal facial landmark detection based on deep multi-task learning.Kopaczka et al.[101] suggested the face tracker method by using thermal videos and images based on AAM.The study focused on strong landmarks to detect and track ROI within head pose and rotation.Furthermore,the study proposed several enhancement algorithms such as sharp masking,USM with bilateral filtering and USM with a Gaussian kernel.The study used several descriptors with AAM like scale-invariant feature transform (SIFT)and Histogram Oriented Gradient (HOG)for fitting the algorithm.The study also used Project-out inverse compositional,alternating inverse compositional (IC)and Simultaneous Inverse compositional.

Sonkusare et al.[104] proposed a novel deep-learning assisted facial landmark to detect method for the thermal image.This is to extract thermal signals from the facial regions.The authors applied the sudden auditory stimulus of a loud stimulus to invoke the physiological responses.The authors aimed to characterize the spatial changes in temperatures of different facial regions i.e.,nose-tip,right and left cheeks and forehead).The GSR and HR is selected as bench marker.In this work,the authors compared two methods; (1)a task-constrained deep convolutional network (TCDCN)and (2)an OpenPose detector.For method (1),the authors implemented TCDCN which trained on RGB images and then fine-tune by further learning on thermal images.Method (2)OpenPose detector is presented by [105] that employs a robust multiview bootstrapping architecture.Then this two methods are combined to improve the landmark localization accuracy.

Several studies [90] demonstrated that face detection in the thermal image is possible without being aided by a visual image.Researchers explored the artificial intelligence method.Mohd et al.[59] suggested a BoCNN architecture framework to overcome one’ limitation,detect occluded facial in a thermal image.A variety of CNN models proven to perform well in thermal imaging [106-110].Hong [45] proposed a multi-subject correlation [100] method to detect ROIs;forehead,nose,and mouth in a thermal image.

2.3 Type of Stress Stimulus

There are many stresses stimulus widely used in literature; Stroop test,Trier Social Stress Test (TSST)[55,62,74],arithmetical questionnaire [61,68,69,71],mock crime scenario setup [70]and physical activities [55].In [45],the participants were required to run on a treadmill to induce physical stress as this study aimed to differentiate physical stress and baseline status.This study achieves an accuracy rate of 90%.

2.4 Feature Extraction(Descriptors)

Thermal images,which have a distinct texture,appearance,and form than RGB images,play a crucial role to identify human stress.Various forms of thermal descriptors have been used in previous research.The effectiveness of facial features from the entire face or facial parts(ROI)is required in stress state identification.Numerous types of feature extraction are shown in Tab.3 of this survey.Based on Tab.3,the majority of studies have used statistical features.For example,He et al.[69] used maximum temperature,and mean.Derakhshan et al.[70] extracted the six temporal features including mean,minimum,maximum,standard deviation,means of the absolute values of the first and seconds’derivatives of the pre-processed signals.Vasavi et al.[65]calculated heart rate based on the mean value of the frame over time.The authors [64] extracted the mean of the top 10% thermally hot pixels,minimum,maximum,and standard deviation.The authors [71] calculated the mean value of emissivity and performed normalization by using min-max normalization.In [61] the authors used the mean of the top 10% of the pixels.Cross et al.[55] extracted average pixel value,maximum pixel value and the mean of the top 10% of the pixels.Berlovskaya et al.[111] also extract the average value of the pixel and standard deviations.

2.5 Facial Thermal Signature Correlation with Stress State

The researchers conducted several studies to establish the correlation between thermal features with stress state of a human [74].suggested that the individual stress state recognized by the facial temperature is measurable with thermal imaging.Kan Hong and his colleagues [62] found a Pearson correlation value less than 1 between facial thermal signatures and known stress indicator such as heart rate (HR)and cortisol level.In this study,the authors proposed new physiological signal extraction; the Eulerian magnification algorithm,to amplify the physiological signals.This study also shows the significant correlation between perinasal ROI and ground truth.A 96%accuracy is achieved by the proposed algorithm compared to ground truth features.

The authors [64] applied the thermal imaging to classify the challenged participants among the threatened participants.The findings show that features extracted from forehead and nose regions yield better accuracy of stress detection.The stress classification based on individual features such as forehead and the nose yields 80% accuracy.When the data from the forehead and nose are combined,the accuracy improves even further.This research reveals insight temperature variations in the facial region that can be used to identify different human emotional states.

The authors [71] studied the impact of psychophysical stimuli on facial thermal emissions.In this study,the authors attempt to distinguish the facial pattern produced by physical activity and mental stress.The findings show that thermal variation caused by psychological stimuli has more changes in pixel intensity compared to the caused by physical stimuli.Reference [68] investigated the impact of student cognitive load on the facial thermographic during knowledge assessment test.The output of the study provides strong evidence to support the correlations between student cognitive load and thermal changes.The authors agreed that thermal signature changes on facial as criteria for stress detection.

Vasavi et al.[66] attempted to identify the most engaging feature for better stress detection by applying regressing modelling.The findings support that the periorbital region is the most engaging thermal feature for stress detection.Authors [112] investigated the correlation between stress and topography of facial temperature changes over time.With Bonferroni-corrected pairwise comparisons,the correlation between induced stress state and the temperature changes in forehead,cheeks and perioral is acknowledged.He et al.[69] justify the usefulness of using facial temperature to evaluate mental stress.The authors also attempted to evaluate the effectiveness of employing face temperature as a mental stress biomarker in comparison to other established biomarkers; HRV,TLI,and PDM.The results established that the face temperature can provide an accountable indicator for human stress recognition in a non-contact approach.Derakhshan et al.[70] confirmed that perinasal and chin areas mostly correlated to stress state in studies.Hong et al.[54] found that number of hot pixels increases in the periorbital region when participants have induced physical stress.When participant experiences emotional stress,prefrontal region has its pixel increases temperatures.This exhibits characteristics of the thermal distribution.

2.6 Stress Classification Based on Facial Skin Temperature Model

Researchers investigated the stress classification based on the extracted thermal signature of the face.Table 2 provides the summary.

In [60],the authors experimented to detect human acute stress by integrating physiological features and thermal features.The authors evaluate the performance of the chosen feature individually and in combination.The performance of thermal features as individual features achieves the highest accuracy of 73% as similar to the fusion of four physiological features,stateof-the-art biomarkers of stress detection.These findings provide evidence to support that the thermal features hold high accountability to measure stress remotely.The authors have suggested a combination of the thermal and respiration rate features can improve the accuracy of stress detection.This combination contributes 26% to improve accuracy.The decision tree classifier is employed for stress classification and validated by the leave-one-subject-out-validation.

Cross et al.[55] demonstrated that the system yields the highest accuracy in classifying mental stressvs.physical stress.The features used are frequency analysis of the respiratory and cardiovascular pulse.The authors extracted statistical descriptors,such as pixel value,maximum value,and the mean of the 10% hottest pixel.The authors also compared four classifiers; AAN,Naiye Bayes classifier,linear discriminant analysis,and SVM.The accuracy rate of classifications is 96.4% (AAN),100% (LDA),92.9% (SVM),and 82.1% (NB).Each classifier is validated with four-fold cross-validation.Among the four classifiers,LDA was found to perform better that provides the high accuracy classification in a short time of computation.The authors claimed that isolated face regions have the potential to improve classification accuracy.

Baltaci et al.[61] proposed a method to separate the stress state of a computer user.The authors investigated the classification accuracy for each feature and fusion.The results show that individually thermal features perform better with an accuracy of 76%,while individual pupil features achieved 73% accuracy.The combined features of thermal and pupil achieved 83%accuracy.Two classifiers were compared which are Adaboost with Random Forest outperform Decision trees.10-fold cross-validation is applied.The stress simulation used in this study revealed limitation emotional that influences the result.The mean of the hottest pixel of the periorbital region is used as descriptors in this study.

Derakhshan et al.[70] conducted an experiment to discriminate deception and truth by comparing four machine learning techniques which SVM,KNN,LDA,and decision tree (DT).This experiment aimed to improve the accuracy of thermal imaging and also to identify the ROIs that can show significant results.From the perspective of physiology,deceptive anxiety leads to spontaneous physiological signs including perspiration,increased heart rate,blood flow changes and so on [70].According to Cannon,this physiological reaction to acute stress is called the“fight or flight” response [113].The raw measurement obtained from thermal data is maximum and minimum values.The six temporal features are extracted from these raw measurements:mean,minimum,maximum,standard deviation,and means of absolute values of the first and second derivatives of the signals.In this study,the authors carried two activities to trigger deceptive,mock crime,and best friend scenarios.For mock crime scenarios,the classification accuracies of thermal data,GSR,and PPG are 83.8%,67.7% and 64.5%,respectively.Thermal data performs better.While in best friend scenarios,accuracies for thermal data,GSR,and PPG are 62.9%,66.6%,and 79.6%.Physiological signals show its discrimination property is stronger than thermal data.After the feature reduction technique is applied,the accuracy of the thermal data jumps from 41% to 90%.The DT performs better than other models.LDA classifier achieves 905 accuracies after feature reduction.The authors also compared the four thermal reduction methods:t-test,relative entropy,ROC,and MWW.The result shows that ROC and MWW produce high accuracies and thet-test shows improvement in other classifiers.The authors used the leave one out validation method to get classifier accuracy.This study established that thermal features outperform gold standard measurement and the accuracy can be improved with the feature reduction method.The authors also found that perinasal and chin contributed to high accuracy classification with help of the feature reduction method.

The authors [65] presented a framework to measure thermal signatures to detect cardiovascular and stress.In this study,the authors extracted thermal signatures such as card pulse,stress responses (heart rate and heart rate variability),breath rate,and sudomotor responses.They categorized the stress state based on rules.For stress responses,the carried two methods,the first method uses FFT and the second method applied wavelet and FFT to calculate heart rate.The first method achieves 91% accuracy while the second method achieves 90.3%.The proposed method performs better than the similar method proposed in [114].

Hong [45] proposed a contact-free model to detect the physical stress of the human body by maximizing thermal signature in the facial region.After extract the ROIs by using the multi-object correlation method,the stress signal was extracted,converted into an independent component by the blind source separation (ICA)method and then amplified by Euclidean Magnification (EM)algorithm.They applied the deep learning algorithm model to classify the baseline and physical stress and achieve 90% accuracy.Before [45],the authors also conducted studies to detect stress by analyzing facial temperature.They achieve 96% accuracy between the proposed EM algorithm and ground truth that consists of the established stress markers.The authors magnified the stress signal after preprocessed with the FFT algorithm.

In the recent years,very few studies explored the deep learning technique to produce better high accuracy stress classification.Reshma [72] presented a hybrid deep learning network for stress detection in the thermal image.Z-normalization based on the mean and standard deviation is applied for better training.In this work,the authors combine two deep learning neural networks.Raw image provided as input to first network DNN-1,and generate frequency features based on wavelet transform technique.This frequency feature is given as input to the second network,DNN-2.This hybrid system performance is compared with the machine learning technique.The comparison shows the proposed system produces high accuracies,96.2%.This system is also compared with two transfer learning networks.The hybrid system outperforms these transfer learning techniques,Alexnet and Vgg-16.Alexnet accuracy is 92%,Vgg-166 accuracy is 94.5 and proposed system accuracy is 96.2%.

Kumar et al.[73] presented a novel deep learning-based methodology that explored the new feature ISTI to detect stress in the thermal videos.This study also proposed emission representation modules that can be used to model variations in emitted radiation due to the motion of blood and head movements.The authors explored the neural network based on facial skin temperature and established evidence to introduce a new feature that has similar performance to the state-ofthe-art features.Biomarkers of Stress State (BOSS)and Cold Pressor test used to invoke stress.Mean Squared Error (MSE)and Pearson’s correlation coefficient (R)were used to evaluate ISTI prediction.The findings show that ISTI extracted by the proposed model have a high prediction rate.Average precision (AP)as a validation metric is applied for stress detection.Predicted ISTI signal is better in detecting stress state than HR (12% higher AP)and HRV (4% higher AP).Also,higher AP with the ground truth ISTI signal confirms that ISTI is the most performing index of stress state in the experiment.Panasiuk et al.[71] compared the numeral analysis and deep learning method for stress state detection.Both techniques depend on heat emissions as a feature.The proposed deep learning methods achieved a high classification accuracy of 88.21%.The numerical analysis produces an accuracy of 76.40%and 78.10% for psychological and physical tests.

Bara et al.[115] presented a preliminary approached based on deep learning towards multi modal stress detection.The authors evaluated a different set of deep learning method.The multimodal used in this works are thermal video; RGB Closeup Video; RGB Wideangle Video; Audio;the QA and monologues; Physiological signals:(1)heart rate,(2)body temperature,(3)skin conductance,(4)breathing rate; and text:transcripts extracted from the QA.In this work,the proposed architecture is based on Convolutional-Autoencoders and Recurrent Neural Networks.The Gated Recurrent Unit (GRU)is used for implementation.The subject-based leave-one-out cross-validation is used for validation.Results demonstrated that the deep-learning methods can generate rich state representations related to stress,regardless of relatively limited amount of data.

Gupta [116] proposed a stress detection method based on a deep learning technique.The authors employed the deep learning model that consists of LSTM layer and a fully connected layer.The output from fully connected layer is channeled into a softmax function for stress prediction of a person.The authors used the 5-fold cross-validation to train the model.The results show the average accuracy of classification by this model is 87%.Proposed model performs better than similar accuracy produced in [117].

3 Discussion

Human stress detection is crucial in different disciplines.This paper discusses the approach used in stress detection by using thermal imaging on facial skin temperature.The reason for selecting the mentioned modality is to focus on contactless,physiological signals,and imagingbased modalities.This paper has discussed the stages involved in stress detection such as face detection in the thermal image,ROI localization,feature extraction,and stress classification.Detecting an image in a thermal image is more complicated than a visual image.Sufficient research has been done on detecting images in visual images more than thermal images.A solid state-ofthe-art face detection algorithm has been established compared to the thermal image,the simple algorithm has been tailored to fit the gap.This becomes a major limitation in this stress detection based on thermal.To overcome this limitation,the researchers have contributed in comparing performance state-of-the-art face detection algorithm based on a visual image in a thermal image.The outcome shows that the researchers can adapt a visual-based face detection algorithm for the thermal image.The researchers also attempted to adopt deep learning techniques to detect the face.These contributions ease the current limitation of stress detection based on thermal imaging.

This paper also highlights the thermal patterns from facial regions as an indicator to detect stress.Thermal-based modalities have focused on the binary relationship between facial temperature and human stress state.Variation of thermal distribution patterns can be used to distinguish different stress types such as emotional stress and physical stress.The paper reveals that many studies have been done to prove the correlation between temperature changes in the face and the stress state of a person.The stress classification accuracy performance of the thermal features has been compared to the physiological biomarker such as heart rate,GSR,and HRV.These physiological signals are considered as a state-of-the-art biomarker for stress detection through the traditional method.The findings established that the thermal imaging technique is a good candidate for detecting stress in a non-invasions manner.The study has the potential to be an ideal experimental evaluation if a person wears a facial mask or relevant protection mode to fight against the COVID-19 pandemic.Because protective gear covers a large portion of the face,the periorbital region only has the potential to be used for feature extraction.More studies are needed to investigate the ROI localization and feature extraction if a person wears a face shield where the shield may hide the actual temperatures on the face.

However,the measurement of facial skin temperature by using low-cost equipment thermal imaging to induce human stress state is not explored in previous literature.Major literature studies have investigated the frontal face for stress detection and correlate it to the stress state.There are insufficient studies to establish the relationship between side face and front face.Very few studies are investigated the side face especially the neck side and ear to study the stress impact.More studies are needed to cover this area.This investigation may lead to a discovery of ROI and an established 3D thermal pattern model to study the stress impact in more detail.

4 Future Work

Based on the discussion,several gaps are identified and this section is to propose the suggestion to fill those gaps.The proposed future work for human stress recognition is based on the thermal signature.Future studies should investigate the relationship between lateral faces and frontal faces.Also,the proposed work should correlate with the stress level induced by an individual.Future work should also consider novel methodologies to extract facial temperature when a person wears a protective face shield.This study can be useful to detect stress among the frontlines.A methodology that computes stress level and correlates it to the temperature changes in a face is needed.This methodology would be useful for detecting stress levels in an individual and can respond accordingly to the perceived stress level.The stress detection based on facial expression in thermal images also is a potential research direction

5 Conclusion

This paper aimed to review the studies on stress detection based on thermal imaging.Many studies that established the thermal feature can be useful for detecting stress remotely.The performance of thermal features has been compared to the state-of-the-art physiological stress marker.However,this methodology has its own limitations.Thus,more studies need to be conducted to overcome the limitations.Based on the quantitative result comparison,it can be concluded that the thermal features have full potential to detect stress.While face detection in thermal imaging limits the thermal imaging in stress detection,the face temperatures changes provide stress information.Further research is needed to determine the temperature changes on lateral sides of the face and also to understand the relationship with the frontal face when a person induced stress.

Funding Statement:This research was pursued under the Research University Grant by Universiti Sains Malaysia [1001/PKOMP/8014001].

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding the present study.