Human-machine augmented intelligence:research and applications

Jianru XUE，Bin HU，Lingxi LI，Junping ZHANG

1Institute of Artificial Intelligence and Robotics，Xi’an Jiaotong University，Xi’an 710049，China

2School of Informati on Science and Engineering，Lanzhou University，Lanzhou 730099，China

3Department of Electrical and Computer Engineering，Purdue School of Engineering and Technology，Indiana University-Purdue University Indianapol is，Indianapolis，IN 46202-5195，USA

4 School of Computer Science，Fudan University，Shanghai 200433，China

E-mail:jrxue@mail.xjtu.edu.cn;bh@lzu.edu.cn;LL7@iupui.edu;jpzhang@fudan.edu.cn

Current research on artificial intelligence(AI)has been entering a new era，with AI technologies and AI-enabled applications emerging in almost every as‐pect of human life.Meanwhile，avoiding the risk caused by limitations of AI technologies has become a grand challenge.The main idea of human‒machine augmented intelligence(HAI)is to adopt the role of humans or to embed human-like cognitive abilities into intelligent machines.Increasing attention and ef‐forts from academia，industry，and governments are at‐tracted by the HAI idea，whose effects are far-reaching.

Two fundamental formulations of HAI include human-in-the-loop HAI(HITL-HAI)and cognitive computing based HAI(CC-HAI)，which have become hot and fundamental frontiers of AI，and an increasing amount of original research has emerged in recent years.

Recent existing research activities on HITL-HAI include theories for human‒machine collaboration，human‒brain interfaces，human‒machine coordination and teaming，and advanced perception and smart envi‐ronments for human‒machine collaboration.In partic‐ular，HITL-HAI has been widely used in interactive simulation models in aviation，driving，and robotics.In such simulations，humans play an important role because they influence the simulated environment with their own actions.Brain‒computer interfaces have become increasingly important among communi‐cation channels for human‒machine collaboration.

CC-HAI aims to develop computational models to mimic the mechanism or function of the human brain and improve a machine’s capabilities of percep‐tion，reasoning，and decision-making.We have wit‐nessed an increasing amount of research work on ca‐sual models，intuitive reasoning models，and associa‐tive memories that are proposed with the forms of deep neural networks.

With this background，we organize a special fea‐ture in the journalFrontiers of Information Tec hnology&Electronic Engineeringas“Human‒Machine Collaboration and Cognitive Computation.”This spe‐cial feature covers theories for human‒machine col‐laboration，brain-and neuroscience-inspired augmented intelligence，human‒machine coordination and team‐ing，and related applications.After a rigorous review process，three papers were selected.

The cognition，management，and control(CMC)of complex systems is a very challenging task.Thus，there is a critical need for developing innovative and comprehensive HAI approaches.Fei-Yue WANG and his collaborators proposed a novel concept called“mutually trustworthy human‒machine knowledge automation(HM-KA)，”which considers human in‐telligence，machine intelligence，and their interaction as a whole，to develop a general process for the CMC of complex systems.The technical process was explained in detail.A bulk power grid dispatch exam‐ple was also provided to illustrate the proposed ap‐proach.As the complexity of practical systems con‐tinues to grow，this novel concept and its related tech‐nical methods can offer a unique solution to the CMC of complex systems and be applied in a variety of practical science/engineering disciplines.

Affective brain computer interfaces(BCIs)have been used to realize emotional intelligence for more harmonious human‒machine collaboration.However，emotion is a complex concept，and stable personal‐ity traits usually influence the precision of an indi‐vidual’s emotion processing.To explore the influence of personality characteristics on emotion processing and implement more reliable affective BCIs，Bin HU and his collaborators proposed a novel personalityguided attention mechanism that can use Big-Five personality traits to guide the learning of spatial and temporal representations of electroencephalogram(EEG)signals.The experimental results showed that it can significantly improve the performance of subject-independent emotion recognition and outper‐form state-of-the-art methods.This method facilitates the development of affective BCIs and makes a fur‐ther step towards affective interaction between humans and machines.

Human‒multi-robot coordination systems play a key role in the development of human‒machine hy‐brid intelligence.However，it is a challenging task to achieve a good balance between human participation and repeated intervention.Yutao CHEN and his col‐laborators proposed a novel method，named behavioral control task supervision with memory，by combining reinforcement learning，long short-term memory，and null-space-based behavioral control.The proposed method has good novelty.Experiments on several benchmark datasets indicated that compared with other methods，the proposed method achieved the best human‒machine coordination performance.

The aforementioned three studies cover many in‐teresting HAI topics and complex tasks.They also pro‐vide a series of solutions to overcome the challenging problems of human‒machine collaboration.We hope that this collection of topics and applications will be beneficial to those with an interest in HAI or in related areas.

Finally，we would like to express our special grat‐itude to the authors and reviewers for their great ef‐forts and valuable contributions to this special feature.

Jianru XUE received his PhD degree from Xi’an Jiaotong University in 2003.From 2002 to 2003，he worked in FujiXerox，Tokyo，Japan.He visited the University of California，Los Ange‐les，from 2008 to 2009.He and his team won the National Natural Science Award(second class)in 2016，the Na‐tional Technology Invention Award in 2007，the IEEE ITSS Institute Lead Award in 2014，and the Best Application Paper Award in Asian Conference on Computer Vision 2012.He has published over 100 papers in top cited journals and conferences，includingIEEE TPAM I/TIP/T SMCB，CVPR，ICCV，ECCV，ACM MM，ICRA，and IROS.He had served as an organization chair or co-chair of international conferences，including VALSE2012，VLPR2011，VLPR2010，ACCV2010，and VSMM2006.His research interests include computer vision，pattern recognition and machine learning，and autonomous driving.