Indoor Environment and Human Sensing via Millimeter Wave Radio:A Review

LIU Haipeng,ZHANG Xingyue,ZHOU Anfu,LIU Liang,MA Huadong

(Beijing University of Posts and Telecommunications,Beijing 100876,China)

Abstract:With the rapid development of 5G technology,more and more attention has been attracted to mmWave sensing.As an emerging sensing medium,mmWave has the advantages of both high sensitivity and precision.Different from its networking applications,the core method of mmWave sensing is to analyze the reflected signal changes containing the relevant information of different surrounding environments.In this paper,we conduct a systemic review for mmWave sensing.We first summarize the prior works on environmental sensing with different signal analysis methods.Then,we classify and discuss the work of sensing humans,including their behavior and gestures.Finally,we discuss and put forward more possibilities of mmWave human perception.

Keywords:millimeter wave radio;review;environment construction;human sensing;5G

1 Introduction

M illimeter wave(mmWave)communications are considered an essential component of 5G-and-beyond ultra-dense wireless networks,and the 5G breakout brings development opportunities to the study of mmWave sensing.Compared with traditional sound waves,ultrasonic waves and Wi Fi signals,mmWave sensing has great advantages,such as fine-grained resolution and the ability to detect subtle movements.Compared with camera-based sensors(such as visible light and infrared 3D structured light),mmWave can penetrate a few non-conductive objects including plastic,paper,glass,cloth,rain,and fog.Therefore,the unique sensing feature of mmWave has attracted more and more attention.

We,according to the different application scenarios,divide the existing methods into two categories.

1)Static indoor environment.The indoor structure being sensed is often utilized in robot vision and environment mapping,reacting upon indoor mmWave Wi-Fi networking.Depending on the capability of the equipment,we regroup the related work from a technology implementation perspective.We believe that such a classification is beneficial for researchers to quickly grasp not merely the cutting-edge works but also the technical details.In the beginning,this task is utilized for robot vision,i.e.,help the robots discover obstacles in their routes[1].Inspired by this application,researchers manage to“see”and build the surrounding construction through mmWave signals[2].Utilizing the acquired surrounding spatial structure,researchers further facilitate the popularity of the mmWave Wi-Fi networking.

2)Dynamic human movements.Since mmWave signals have a short wavelength,it is sufficient to detect centimeterlevel distance and sensitive to capture the millimeter-level motions around.Based on this knowledge,many researchers utilize mmWave signals to identify people through their gait[3],recognize several predefined hand gestures[4–5],and track someone’s fingers[6].More subtlely,the vital signs(such as respiratory and heart rates)can also be extracted by changes in reflected mmWave signals.

The core method of mmWave sensing is to analyze the signal changes reflected on the surrounding environment,so as to obtain its state.Therefore,in a practical operation process,we first need to use the transmitter module to transmit the signal and then receive the signal reflected and“modulated”by the surrounding environment.

The information of the target is contained in these signals,so the sensing systems then use different methods to“demodulate”the information of the surroundings,which will be processed and counted to get an overall message of the surrounding environment.However,there are two methods to perform the“demodulation”:the traditional signal analysis and the AIbased learning.The former analyzes the angle and orientation of the reflection object by analyzing the changes in the communication information of the signal after reflection.The latter uses black-box machine learning methods to comprehend more complex human movements.

Interestingly,most studies on dynamic human movements rely on artificial intelligence(AI)technology because the features extracted from human movements are very complicated.These features include a great many reflection points with different distances,angles,speeds,and energy intensities.Therefore,the AI tool,originally designed for image learning,can discover the underlying relationship across such massive and complex feature information,so as to help the system sense human movements.Inversely,only a few studies on static indoor environments rely on AI.For the reader’s convenience,we categorize these existing applications based on whether using AI and sensing scenarios.Based on this analysis,we explore each of the sensing scenarios in terms of breadth and depth and also put forward the prospect of mmWave sensing in future applications.

2 Sensing Indoor Environment

When networking in an indoor scenario,the available mmWave links often include multiple paths reflecting off the surrounding wall,in addition to the LoS path,as shown in Fig.1.Different surrounding walls have different effects on the information of received signal strength(RSS)and phase of the mmWave link paths.Then by analyzing these effects,researchers can extract relevant information about the paths,such as the time of flight,transmission length,or reflection angle.Using such information of paths,researchers construct the surrounding environment.Otherwise,researchers have focused on how to perceive the indoor environment to improve the efficiency and stability of 5G mmWave indoor networking.

▲Figure 1.Indoor environment is inferred by concatenating the reflection points

The radar system,using frequency modulated continuous wave(FMCW)mmWave signals,is easy to identify the surrounding environment because it has the advantages of robustness,low computational complexity,strong penetration,etc.However,the current work is often done by reusing lowcost and ubiquitous 5G mmWave communication devices.Compared with radar devices,this device is difficult to identify the surrounding environment and thus is unable to handle complex environments.Therefore,how to reuse 5G signals to obtain the surrounding environment has become the key to the problem.According to the limitations of different devices and the way they use signals,we categorize the related works into threefold implemental technologies(categorized in Table 1).

2.1 Utilizing Both Phase and RSS

During the process of device localization for mmWave,we normally have the problem of knowing nothing about the initial surrounding environment.To solve this,PALACIOS et al.design JADE[2],which can estimate the location of a mobile user in indoor space without any prior knowledge.The JADE algorithm can be directly combined with a commercial device to extract information using the angle of arrival(AoA)of signals,so as to obtain the location of users.Then,they propose CLAM[7]to localize the mobile user,estimate the position of access points,and finally form a map of the environment.Using these two algorithms,they realize the localization and mapping of the mmWave network without knowledge of the initial environment.

▼Table 1.Comparison of the state-of-the-art works on sensing static indoor environment

CLAM combines experiments with simulation because JADE is entirely dependent on simulation results.Thus,the above two works are too dependent on the simulation experiments,which theoretically discuss the possibility of indoor environment perception.E-Mi[8]achieves the same function experimentally.Specifically,E-Mi proposes a multi-path analysis framework,using a customized 360°omnidirectional antenna to obtain the information of theof all reflection links through the RSS and phase information of the signal,so as to infer the surrounding environment according to the geometric relationship.Based on E-Mi’s equipment and method,Beam-Forcast[9]makes a reverse engineering,i.e.,how to accelerate the alignment of network link for a mobile client through the angle change of the client in a given environment.

2.2 Utilizing Only RSS

The above methods can acquire the AoA information directly,but commercial off-the-shelf(COTS)mmWave nodes in daily life cannot accurately extract phase information.This is because the daily mmWave nodes use ideal laser-like beams,which are generated by horn antennas.Due to its lack of ability to maintain the phase offset between the transmitter(Tx)and the receiver(Rx),a lot of small phase jitters may cause a large error in practical applications.To solve this question,researchers try to determine the AoAs of the link by adjusting the Rx’s orientation(i.e.,the receiving direction with the largest RSS).

ZHU et al.design RSA[1]which moves radio antennas while collecting reflection signals to create a synthetic aperture radar(SAR)to infer the object’s boundaries,curvature,and surface material.Specifically,RSA fixes the Tx,a commodity HXI Gigalink 6 541 board,and moves the Rx with a deliberately designed path collecting reflection signals.In this process,RSA can extract the detected object’s status including both its curve direction(by aligning its receiver orientation towards the strongest one)and its material according to the signal strength.Furthermore,Ulysses[10]binds Tx and Rx(the same as that used by RSA)together and moves them at the same time.According to this,the system can prevent collision in motion,which can be used in future scenarios,e.g.,the unmanned driving and robot movement.

Uniquely,utilizing RSS of the reflected signals,RadarCat[11]can identify different materials of plenty of static objects,such as glasses,water,metals,and mobile phones.Specifically,RadarCat is based on a principle that the signals reflected from different materials are highly characteristic because the thickness and geometry of the object will scatter,refract and reflect the radar signals differently.Therefore,RadarCat feeds the 8-channel radar signal RSS and their statistics into a welltrained classifier to distinguish the different objects.

2.3 Utilizing RSS with a Programmable Robot

▲Figure 2.A programmable sweeping robot that works precisely to position,steer,and track

All of the above indoor environment mapping works(except RadarCat)need to manually adjust the position and orientation of the devices,so researchers try to automate the task of collecting signals with the help of a programmable sweeping robot as shown in Fig.2,which is precise to position,steer and track.With the help of this robot,mmRanger[12]can use the commercial mmWave network cards to achieve the construction of and indoor environment by only utilizing the correlation between AoA and RSS.Specifically,mmRanger carries two mmWave cards on a commercial cleaning robot to make all of them as a whole,i.e.,a smart mobile environment sensor.The robot can sense the environment by exchanging mmWave signals between the two cards when it moves and rotates freely inside the target room.Upon this process,the geometry of the multiple reflection paths can be extracted by mmRanger from their RSS sequence,and then the environment layout can be reconstructed.

Actually,smart robots will eventually become a part of the home and enterprise environment to help automate our daily lives and improve productivity.Based on this vision,miDroid[13]binds the same network card node to the same robot,turning the robot into an indoor mmWave Wi-Fi relay and thereby achieving faster mobile client network optimization.More specifically,miDroid firstly analyzes the series of access point(AP)beacons to extract spatial factors and then finds out the AoA/angle of departure(AoD)of signal transmission paths,so as to map the environment.Then,miDroid proposes a real-time and adaptive path planning algorithm to instruct the navigation of the robot relay,thus improving the performance during the client’s blockage when he changes orientation rapidly.

3 Sensing Human

Different from the last section,human movements simultaneously generate multiple reflection points each with different position,velocity,reflection intensity,etc.Therefore,two research methods are adopted according to the complex degree of the sensing movements(categorized in Table 2).

3.1 Human Movement Tracking

Several works are aimed only at small,uniform movements on a certain part of the human body,such as vertical finger tracking,regular breathing,and heartbeat.Specifically,researchers utilize horn antennas to focus on the detected target,thereby eliminating the interference of unrelated motions around,as shown in Fig.3.In this way,any variation in the reflection signal represents the change of the target to sense changes in the target part of the human body.

To focus on a periodic movement,e.g.,human chest variation caused by the breath and heartbeats,YANG et al.[14]propose mmVital,a system using 60 GHz mmWave signals for vital sign monitoring.In this system,they utilize two horn antennas fixing their orientation to continuously capture the subtle variation of the skin on one’s chest by analyzing the signal modulation caused by the skin fluctuation.Specifically,they extract the periodic changes within the signal RSS to acquire the frequency of breath and heartbeat and filter the raw signals through the suited band-pass filter to achieve better accuracy.As a result,mmVital provides a mean estimation error of 0.43 breaths and 2.15 breaths per minute within 100 ms of dwell time on reflection.

On the other hand,for the non-periodic movement,e.g.,tracking a finger,the horn antenna needs to determine the direction of the finger movement by analyzing the signal changes to realize finger tracking.WEI et al.[6]trace a rectangle area as the tracking region in their proposed tracking system mTrack. Then, they place a quasi-omni-directional(180°beamwidth)transmitter on one of the rectangle vertexes and two horn-antenna receivers on the adjacent sides facing the region respectively.In this region,finger movements in any direction will be transformed into two relative movements:approaching or moving away from the two receivers.Fortunately,these two movements can be detected by the phase changes of the signal thank to the small wavelength of 60 GHz mmWave.As a result,mTrack can track a vertical finger with a 90th percentile error below 8 mm which is sufficient for a virtual trackpad.

▼Table 2.Comparison of the state-of-the-art works on sensing dynamic human movements

In addition to the above work on sensing a certain part of the human body,there are also some work on human detection.For example,GU et al.[15]propose mmSense,a devicefree multi-person detection framework.During this work,they use the properties of 60 GHz signal for human bodies and objects to fingerprint the environments including and excluding humans.Then based on the monitored 60 GHz signals and generated fingerprints of environments,mmSense can simultaneously detect the presence and locations of multiple persons.Furthermore,by correlating the 60 GHz RSS series with the measurement of different people’s outlines and vital signs,they propose a new method to identify multi-person.Finally,they demonstrate the effectiveness and low cost of this method through experiments.Besides,in the domain of user recognition,LIU et al.[16]propose mID,the first user identification approaches that utilize mmWave signals.

3.2 Complicated Behavior Sensing

As aforementioned,the movement of the human body creates a large number of reflection points with different spatial features,as shown in Fig.4.So we have to treat the human body as a soft body instead of a rigid body(e.g.,a wall).In order to obtain the position-scattering points,all of the work utilizes FMCW，which can separate the reflection points with their spatial features,i.e.,distance,velocity,and angle.With the different features of the points on a human body,his/her movements can be recorded in detail.However,it is too complex to recognize human movements by analyzing different changing features with simple geometrical relationships.Therefore,the AI tool acting as a black box is utilized to help the system sense human movements.

▲Figure 3.Detecting simple human movements

▲Figure 4.Detecting compound human movements

To recognize gestures,WANG et al.[4]directly utilize a mature method,Range Doppler(RD),in the FMCW signal processing field for each fleeting period slice of a gesture.Then,to classify these RD sequences,they input the RD image of each slice into a customized convolutional neural network(CNN)and put the discrete recognition results from each slice into a recurrent neural network(RNN).As a result,this work achieved 87%accuracy on 11 gestures.Crucially,the hardware in this work is the first small-size mmWave radar called“soli”[5],which transmits omni-directional FMCW signals to sense the gesture of the nearby environment.To verify its gesture sensing ability,the researchers,in addition to the above RD metrics,also introduce two series of features,i.e.,inphase/quadrature(I/Q)statistics,and some tracking information.In particular,the former is useful for detecting micro motions and the latter is beneficial for recording the moving trend of the gesturing hand.Then,these features are fed into a random forest(RF)classifier due to its computational speed,low memory consumption,and generalization ability.

Totally,this work is sufficient to recognize four gestures with 92.10%accuracy.Furthermore,using the same chip,SMITH et al.[17]implement the gesture recognition into a human-car interface with also an RF classifier.Based on the above solutions,PATRA et al.[18]present a low-cost mmwave radar-based system to save the computation resource.They detect gestures only by the AoA and the power profile extracted from the measured signal.Specifically,they use two low-complexity classification algorithms:unsupervised self-organization mapping(SOM)and supervised learning vector quantization(LVQ).With these methods,gesture recognition can reach 87%accuracy for some gestures.

Gesture recognition can be applied to home assistants.For example,the user can wave one’s hand at a distance to turn on a TV.To achieve gesture recognition in such home scenarios,there are several vital challenges:1)With the increase of the detection range,the details of gestures will become difficult to be captured due to the severe attenuation of the mmWave transmission in the air.In this case,the reflection points will be sparser when the user is standing at a distance.2)There are also plenty of non-target movements,i.e.,they are not predefined gestures but are performed in our daily life.In this case,the points,which are generated by reflection on the non-target movements or even by the multipath reflection on the user’s ambient appliances,will interfere with the original recognition.

To solve the problem,LIU et al.[19]propose mHomeGes,a real-time smart home gesture recognition system completely using mmWave.First,they obtain the position and dynamic variation of the gesture.Then they recognize fine-grained gestures by using a lightweight CNN.Next,they propose a user-discovery approach to focus on target human gestures,eliminating the adverse effects of surrounding interference.Finally,they realize the continuous gesture recognition in real time.In the end,mHomeGes achieved more than 95.30%of high-precision recognition in real-time smart home scenarios,successfully solving these problems.On the other hand,PALIPANA et al.[20]use a Pointnet++and LSTM combination to extract the spatiotemporal feature of point clouds.In this case,they build a 4D point cloud classification architecture that feeds on the point clouds directly to recognize the gestures.SANTHALINGAM et al.[21]also try to recognize the movements(e.g.,American sign language recognition for the deaf and hard-of-hearing people)at a distance,but they highly rely on cumbersome devices.Based on the home-scenario gesture recognition,LIU et al.further propose mTranSee[22]to largely reduce the adaptation effort in a smart home scenario via transfer learning,which promotes the practicability of mmWave gesture sensing.

Based on the similar inspiration of the point cloud,MENG et al.[3]utilize FMCW mmWave signals to realize the gait identification by three spatial features,i.e.,three-dimensional(3D)coordinates in space,speed relative to the radar,and the energy intensity of each reflection point on a human body,respectively.In order to adapt these three features,they also design a novel AI algorithm,i.e.,GaitNet,which concatenates five separated attribute networks(the first three are for the 3D coordinates)and feeds them into a fusion network.As a result,mmGaitNet achieves 90%and 88%accuracy for single person and five coexisting person scenarios,respectively.

Since point clouds reflect the movement of the human body,and millimeter waves can detect tiny movements,can we construct the human body through millimeter wave signals?For this,XUE et al.[23]propose mmMesh,a real-time 3D human mesh estimation system.This system can accurately align the 3D points with the corresponding body segments.With this approach,the lost part due to the sparsity of mmWave point cloud is introduced from the information of the previous frame so that a dynamic human mesh is completed.

4 Discussion

According to different sensing tasks(environment/human body)and core methods(AI/non-AI),we cross-classify the existing methods across application scenarios(categorized in Table 3).Next,we will discuss the deficiency of mmWave sensing-related work and look forward to the other perception applications.

▼Table 3.Existing work and prospect for the future

4.1 Deficiencies of State-of-the-Art Work

Deficiencies of the state-of-the-art work are introduced in the following scenes.

1)Sensing the indoor environment

For environment construction,the current work lacks the ability of complex environment construction,i.e.,it is limited to smaller indoor environments.Moreover,the current work can only identify simple construction,i.e.,it is difficult to distinguish the details of the surroundings(e.g.,a safe in the corner),so the challenge is to study how to accurately sense the details.None of the methods have ever tried to combine efficient AI tools with indoor environment construction.There is only one work[11]that uses the AI method to perceive the materials.For material reflected,though mmWave-based material recognition is currently used in security inspections,there are still problems like the bulky and inefficient machines.

2)Sensing human dynamics

Gestures are currently sensed only in an ideal environment,i.e.,the researchers only allow the user to perform the gestures in several fixed positions.For monitoring vital signs,current work only extracts the frequency of a person’s breath rate and heartbeat in calm status.Therefore,when the solution is applied at home,it is difficult to track a moving human body.Moreover,the current work only determines the breath and heartbeat rates approximately through the periodicity of the signal.However,compared with more accurate medical devices upon bioelectric signals,it greatly lacks accuracy and credibility.

4.2 Future Outlook on Novel Applications

We also look into the future of novel applications as follows.1)Sensing indoor environment

For environment sensing,since the current work has achieved the ability of simple indoor environment construction,in the future the same method will meet new challenges when applied to a wider space(e.g.,larger office space).Furthermore,to improve the accuracy of recognition of the surrounding environment,smaller FMCW radars can be used to achieve the same ability.However,FMCW radars are sensitive to minor moving variations which may affect the detection ability of small metal objects.Moreover,since mmWave cannot finely image the sensing objects,it may provide important assistance for the vision of some certain unmanned systems like domestic robots which need to protect user’s privacy security.

2)Sensing human dynamics

For human dynamics sensing,everyone making the same gesture will have slight differences in the motion habits,so we can also use the same gesture to authenticate different users.But it is important to extract the unique information differentiating host gestures from others.For monitoring vital signs,since the current work of measuring heartbeat or breath requires the person to be still,one can explore how to measure these vital signs in a dynamic process.In the future,once implementing the mobile tracking monitoring method,we can also track the user’s living habits and real-time health conditions,so as to better help people improve their living quality.Moreover,the data collected by medical devices can supervise the mmWave signal training on the AI algorithm,thereby judging the breath and heart rates more precisely,reliably and automatically.

As can be seen from Table 3,only Ref.[11]is based on the AI method and explores the indoor environment.Based on RadarCat’s inspiration,we can also learn about changes in signal strength to detect changes in the body’s blood sugar concentration“in air”,therefore reducing the pain of patients.Similarly,this approach could be applied to screening passengers for sensitive metal objects at security checkpoints,reducing the work of staff.Furthermore,researchers can further explore this direction,e.g.,AI algorithm can learn the changes in a signal state to judge the tendency of users to move in different positions in the room,so as to improve the performance of the mmWave network more quickly instead of the traditional iterative algorithms.

Besides,there are only two existing tasks based on non-AI and human dynamics,so researchers can design more targeted and simple algorithms,e.g.,tracking the angle of the swiping hand to determine its direction and speed.Furthermore,the hand location in space can be obtained by using the range and angle information of its reflection point cloud to realize hand tracking.Based on this,researchers can also realize a low-power virtual reality(VR)game sensor based on mmWave.

5 Conclusions

The state-of-the-art mmWave sensing solutions have performed basic functions:constructing the general indoor environment and recognizing the dynamics of the human body.We classify the existing work according to their sensing tasks,i.e.,static indoor environment and dynamic human movements,and then introduce the characteristics and advantages separately.Finally,we use a table to further classify the work and present a forward look at future work in this field.