Application of Mixed Reality Technology in Visualization of Medical Operations

Yuan Gao, Ke Tan＊, Jian Sun, Tao Jiang, Xiaowen Zou

Educational Technology Center, The PLA General Hospital,100853 Beijing, China

Key words: mixed reality; medical surgery; visualization; simulation

Abstract With the continuous progress of virtual simulation technology, medical surgery visualization system has been developed from two-dimensional to three-dimensional, from digital to network and intelligentization.The visualization system with mixed reality technology will also be used in all stage of medical surgery, such as case discussion, surgical planning, intraoperative guidance, post-operative evaluation, rehabilitation, so as to further promote high intelligence, high precision of medical surgery, and consequently improve effectiveness of treatment and quality of medical service. This paper discusses the composition and technical characteristics of medical operation visualization system based on mixed reality technology, and introduces some typical applications of mixed reality technology in medical operation visualization, which provides a new perspective for the application of mixed technology in medical surgery.

INTRODUCTION

A new digi tal holographic imaging technology has emerged in the field of computer graphics after Virtual Reality (VR) technology and Augmented Reality (AR) technology.The core feature of this technology is that it breaks the boundary between the digital virtual world and the physical real world.

Initially this technology is an evolution of Mediated Reality， which is the father of intelligent hardware and proposed by professor Steve Mann at University of Toronto. In 1994， Milgram et al.1proposed the concept of “true-virtual continuum” (Figure 1)， and for the first time expounded the relationship between the mixed reality， the augmented reality and the augmented virtual. In this coordinate system， the left end of the horizontal axis represents pure real environment，the right end of the horizontal axis represents pure virtual environment， and mixed reality (MR) covers AR and augmented virtual (AV). In 2015， then-president Obama2proposed the “precision medicine plan” in his State of the Union Address， which marks arrival of the era of precision medicine. However， relying on traditional CT， MRI， PET and other two-dimensional medical images， doctors need to imagine the requirements of an accurate surgery. Therefore， mixed reality technology aims to combine data generated by two-dimensional tomographic imaging sequence with computer and make an intuitive three-dimensional visualization image， so as to achieve visualization of complex medical anatomical structure in three dimension， which is often the difficult part of medical surgery.3-5By means of the real-time interaction between the mixed reality equipment and the real individualized cases of patients， the complex human body structure and diseased parts can be realistically displayed in front of eyes. It is convenient for doctors to observe and analyze from multiple angles and levels， make operation plan preoperatively and guide operation intraoperatively. Besides， doctors can effectively participate in data processing procedure to achieve an accurate and safe surgery.6-10

CURRENT STATUS OF RESEARCH AND APPLICATION

In the past two years， with the development of miniaturization and wearable mixed reality equipment，research on visualization of medical surgery by mixed reality technology has made a great progress. Belmustakov S et al.11at Johns Hopkins University analyzed the application of mixed reality navigation in imaging-guided interventional radiotherapy of musculoskeletal system， and found this technology reduced the operation time and improved the safety and accuracy of surgery. Gregory TM et al.12from the Paris hospital in France presented a live webcast of shoulder prosthesis implantation through a mixed reality device. Javaux A et al.13from University of Leuven in Belgium developed a mixed realistic surgical trainer for minimally invasive fetal laser surgery， which had been used to train surgeons for the purpose of improving their surgical skills.

Wu P et al.14integrated the holographic anatomy teaching database into clinical training of plastic surgery， which improved teaching efficiency. Zhu J et al.15from the PLA general hospital in China combined the technology of minimally invasive surgery robot with remote guidance and developed a mixed reality platform， on which they had completed a tumor resection through the right retroperitoneal cavity with remote cooperative robot. Yan R et al.16at Xijing Hospital in Xi’an China applied real-time fusion of holographic virtual model and robotic surgical image to assist intraoperative navigation， and completed partial resections of endogenous renal tumor in 15 patients， which significantly reduced duration time of the operation.

Researches on mixed reality technology and its application in medicine have been almost synchronized around the world. In the field of surgical visualization research and application， there are many technical patents and software copyrights in mixed reality technology， which has been widely used in clinical practice，such as surgical planning， intraoperative guidance，evaluation of treatment outcomes， as well as in teaching and clinical training.

THE CHARACTERISTICS AND SYSTEM STRUCTURE OF MEDICAL OPERATION VISUALIZATION TECHNOLOGY BASED ON MIXED REALITY TECHNOLOGY

According to Steve Mann’s theory，1intelligent hardware will eventually transit from AR to MR. As a mixture of AR and VR， MR has higher technical requirements on system equipment than VR because it contains more types of technologies， including sensors，interactive devices， advanced optics and next-generation computing power. We believe that in the field of medical operation visualization， MR has following basic technical characteristics： 1) the combination of reality and virtual world; 2) accurate matching between virtual objects and the real world; 3) interaction of real world with virtual objects in real time; 4) multiple users cooperate and interact with the same virtual object.

Therefore， virtual simulation system using mixed reality technology has been increasingly applied to the whole process of medical surgery. Its architectures which constitute the application basis of mixed reality technology in the field of medical surgery visualization is illustrated in Figure 2.

Individualized patient image data， including MRI and CT data of the lesion site， were collected from Picture Archive and Communication System (PACS)of hospital. Through 3D model reconstruction system，image registration， segmentation， feature extraction，optimization， marking are carried out， and then 3D reconstruction can be performed for diseased parts and related tissues. Data are transmitted to the interactive system of mixed reality display for preoperative doctor-patient communication and preoperative planning discussion in real cases of patients， where the operation process can be formulated. The whole process of surgery is uploaded to the visual surgery file storage system， combined with the reconstruction data of the 3D model of the patient in operation by means of big data and artificial intelligent processing and analysing，the image recognition and the prompt feedback to the key steps of surgery can be realized intelligently during the operation. Connected by the remote intraoperative cloud forwarding data server， the intraoperative real-time mixed real images can interconnect with the remote surgery visualization system， thus the remote teaching interaction and remote intraoperative guidance can be realized， which breaks through the limitation of traditional virtual reality system.

Figure 2. Schematic overview of mixed reality system in visualization of medical operations.A. Individualized patient image data storage system; B. image processing and 3D reconstruction system; C. mixed reality display interactive system; D. intraoperative visual guidance positioning and interaction system; E. visual operation file storage system; F. remote visualization of operation cloud forwarding data server; G. remote surgery visualization system(divided into remote mixed reality display and instruction).

APPLICATION OF MIXED REALITY TECHNOLOGY IN MEDICAL OPERATION VISUALIZATION

Preoperative visual communication between doctors and patients

Due to the high specialization of medical knowledge and technology， the diversity of cultural background of patients and their families， the limitation of their knowledge and cognition levels， as well as the age and diseases status of patients， there is often a gap between expectation of therapeutic effects and their actual medical conditions. Patients usually have insufficient medical knowledge， and not fully understand their illness and the treatment. The information asymmetry become the major cause of medical disputes.According to medical complaint data， most of the doctor-patient disputes are caused by inadequate communication， which is partially related to medical negligence.17，18

To establish a three-dimensional model using real data of patients can be the fundamental part of a good communication between doctors and patients.By collecting medical imaging data， e.g. CT， MRI of the lesion site from the patient， we perform medical image analysis and processing， and conduct personalized reconstruction to acquire three-dimensional images.Then， we remove the noise and unimportant tissue structures in medical images and retain the concerned parts in 3D image reconstruction. The key objects and anatomical structures， such as tumors and blood vessels， are colored and contrast enhanced for subsequent observation and perception. For example， in hepatic tumor resection， we segment individualized medical image data of patients， and reconstruct the tumor and the eight hepatic segments in 3D， then mark them separately to facilitate interactive display (Figure 3).

Based on patients’ personalized three-dimensional model data， wearing mixed reality display glasses，patients can see the real state of the lesion， its location， size and the relationship with the adjacent organs/tissues， not just read the imaging reports that are often difficult to understand. In combination of doctor’s interactive explanation and virtual operation demonstration in 3D dynamic simulation processes，patients and their families usually can accurately and vividly understand the anatomical structure， treatment methods and surgical procedures， so that they have a clearer and more intuitive understanding of the treatment scheme. Their worries and unnecessary concerns can be greatly eliminated.

Figure 3. Reconstruction and interaction of liver tumor imaging data.

Visualized training of surgeons

Medical education for a qualified doctor is a process with high-cost and long-term accumulation. It requires enough anatomical specimens and continuous simulating exercises. In particular， medical surgery puts forward higher requirements on doctors’ understanding of anatomy and 3D spatial relationship. Currently，some new teaching methods， such as problem-based learning (PBL) and case-based learning (CBL)，19，20have been applied in medical surgery training and teaching.However， it is still unable to break through the traditional 2D interactive mode， that is， trainees recognize lesions by observing 2D medical images and reconstruct 3D anatomical relations in their brain; they also need to picture the surgical processes in 3D in their mind.

The visualization system using mixed reality technology can realize the fusion of image information and real objects， thus reduce their dependence on experiences and spatial imaginations. This is of great

significance for medical professional training， including continuous medical education (CME)， which can effectively increase quality of clinical training， accelerate growth of young doctors and shorten the training cycle for medical students.

We have developed a reconstructed brain image model using 3D printing technology combined with mixed reality glasses， which has been applied in clinical training of neurosurgery in our hospital. We built a visual training platform for 3D learning MR visualization， so that better visual simulation and navigation can be conducted in brain puncture surgeries， where beginners can determine the position and the angle of puncture needle based on virtual images to avoid important blood vessels. It provides a repeatable and risk-free training model for surgeons (Figure 4).

Intraoperative visualization guidance

Mixed reality technology combines the virtual model and real scene， and has created a new mode of visualized guidance for precision surgery. By using this technology， doctors can comprehensively observe details of the lesions， deeply excavate image information，and intraoperatively conduct navigation simulation， so as to achieve accurate localization of the lesion and reduce potential surgical risks. The reconstructed and rendered virtual 3D femoral tumor anatomical model has been loaded into MR glasses and applied in the orthopedic surgery in our hospital. The size， angle and contrast of the model can be manipulated by tablet computer， and particular tissues in the model can be hidden or transparent to facilitate observation of the tumor， important adjacent vascular and surgically related areas. When special marks are needed during the operation， direct manual instructions are given by hand without special operations. In addition， special marker can be set in advance at the desired position according to the need of operation， and superimposed with real human body during operation (Figure 5).This provides a reliable surgical basis to achieve accurate performance of surgical treatment scheme.

Figure 4. Application of mixed reality technology in neurosurgery navigation training (VIDEO 1 in supplementary material online).

Figure 5. Perfect fusion of mixed reality technology and femoral tumor surgery.

Teaching guidance of remote surgery visualization

With the help of mixed reality technology， an interactive information feedback bridge can be built between the virtual world and the real world for remote users.It can realize a high degree visual sharing of virtual model and real surgical scene， and enhance the sense of reality， substitution and experience of remote sur gery teaching. In June 2016， for the first time， six hospitals21in Shanghai live broadcasted a mixed reality surgery using remote video to more than 3，000 medical practitioners in 20 provinces nationwide， where doctors felt the innovation and convenience brought by mixed reality technology in remote surgery teaching.On May 14， 2017， our hospital integrated mixed reality technology into the broadcast of surgical teaching. A clinical case of Ⅶ segmental hepatectomy with methylene blue staining was broadcasted live at the hospital venue and on the internet at the same time. Real-time superimposition of the patient’s 3D images reconstruct model can be superimposed on the operating field environment， and be changed accordingly during the operative process (Figure 6). It is very helpful for visual operation guidance and teaching for some difficult surgical operations. With help of remote mixed reality technology， people in different regions can communicate across space interactively at the same time based on visualized presentation of a remote surgery， thus realize good collaboration and effective consultation.

Combination of mixed reality technology and artificial intelligence

Figure 6. Live broadcast of partial hepatectomy operation using mixed reality technology (VIDEO 2 in supplementary material online).

During processing DICOM data of original patient image to generate a 3D visualization model， deep learning and artificial intelligence technology can be used to conduct intelligent analysis and imaging data mining，realizing intelligent detection， recognition and segmentation of lesions from various surrounding tissues， and providing data support for later hybrid reality display model. When displaying the model， it can perform allround visual presentation to meet requirement of doctors， meanwhile， it can update information in real-time according to needs in process of surgery， thus realize intelligent automatic analysis and calculation of the operation path in surgical planning. For example， radiologists in our hospital use artificial intelligence technology to automatically recognize and segment 2D imaging data，extract 3D visualization models for normal arteries and dissecting vessels， and provide a model basis for hybrid reality display of vascular surgery (Figure 7).

Figure 7. In the operating room， wearing mixed reality glasses， surgeons can interactively observe a three-dimensional aortic dissection model of the patient， which was segmented and reconstructed automatically through artificial intelligence. The red part of the aorta represents the true luman and the blue part represents the false lumen. Readers can scan the 2D code at the right lower corner of the picture with mobile phones to view this aortic dissection 3D model （in supplementary material online） interactively at mobile terminals.

Visual data recorded during surgery provides artificial intelligence (AI) module for improvement of surgical procedures. The data can be used to further train machine learning algorithms for intelligent assessment and process analysis modules to improve their ability，and provide feedback immediately during or after surgery. Artificial intelligence， for example， can provide visual cue data on mixed-reality glasses by tracking where a surgeon’s gaze is focused during surgery， which can give young surgeons additional technical guidance.

PITFALLS AND CHANLENGES

The application of mixed reality technology in the field of medical surgery visualization is still in its infancy， and there are still some unsolved theoretical problems and technical difficulties：

·3D image reconstruction is easily affected by original image scanning thickness， motion artifacts，signal-to-noise ratio， etc. Particularly， the difference in slice thickness of different scan protocol as well as different equipment directly affect the accuracy of reconstruction model.

·Small blood vessels and some soft tissues are difficult to be reconstructed. Compared to the bone modeling， soft tissue’s signal intensity varies greatly，which requires a lot of manual intervention in the post processing stage， especially in reconstruction of blood vessels. When the concentration of contrast agent is low， small blood vessel branches are often difficult to be identified and reconstructed.

·Display hardware equipment is usually not powerful enough to present high-quality and high-fidelity medical images. For example， the most commonly used Microsoft HoloLens mixed reality glasses only have a view angle of about 35 degrees， and the resolution is not high enough to get a perfect user experience. The accumulation of delay in image processing can cause motion sickness. In a complex environment，the speed of image generation is relatively high. Due to the complexity of human organs and tissue， a 3D model usually consists of hundreds of thousands or even millions of faces， and image processing takes time， so the delay is inevitable.

·Most MR applications require users to wear special interactive tools， and the human-computer interaction experiences with handheld display equipment， such as Head-mounted Display(HMD)， and projection equipment are not natural and convenient enough by far.

SUMMARY

Virtual reality technology connects human beings with the virtual world created by computers， so that human beings can get the same perceptual feelings in virtual world as in real world. Mixed reality technology breaks the boundary between the virtual and the real world. It connects people in the virtual world to those in the real world at the same time. The application of mixed reality technology in the visualization of medical surgery can help to effectively avoid doctor-patient disputes by making patients more intuitively understand their own diseases and corresponding surgical plans.holographic display fusion of intraoperative real scene and 3-D virtual anatomical model can help doctors to make a safer and more reasonable surgical scheme.In surgical skill training， it helps to improve the learning efficiency with shortened training time and reduced training expenditure. It becomes possible that junior surgeon can complete difficult operations successfully through remote guidance of a senior surgeon by using mixed reality technology， so that patients can obtain advanced surgical treatment without geographic barrier.

It can be predicted that mixed reality technology will bring great changes to the visualization of medical operations in near future. The combination of intelligent medicine and big data with mixed reality technology will further improve the intelligence and precision of medical surgery， and thus promote the quality of medical treatment and service level. The research and application of mixed reality technology in the field of medicine have a broad and bright prospect.

Conflict of interests statement

All author declared no conflict of interests to disclose.

Supplementary materials

- Video 1. Application of mixed reality technology in neurosurgery navigation training.mp4. Available online at http://cmsj.cams.cn/fileup/1001-9294/SUPPL/20190612174826.mp4.

- Video 2. Live broadcast of partial hepatectomy using mixed reality technology.mp4. Available online at http://cmsj.cams.cn/fileup/1001-9294/SUPPL/20190612175701.mp4.

- A 3D model of aortic dissection for interactive observation. Available online at http://www.mol3d.cn/shareDeatil.html?publishId=8444.