Augmented Reality Visualization for Image-Guided Surgery: A Validation Study Using a Three-Dimensional Printed Phantom

Background: Oral and maxillofacial surgery currently relies on virtual surgery planning based on image data (CT, MRI). Three-dimensional (3D) visualizations are typically used to plan and predict the outcome of complex surgical procedures. To translate the virtual surgical plan to the operating room, it is either converted into physical 3D-printed guides or directly translated using real-time navigation systems. Purpose: This study aims to improve the translation of the virtual surgery plan to a surgical procedure, such as oncologic or trauma surgery, in terms of accuracy and speed. Here we report an augmented reality visualization technique for image-guided surgery. It describes how surgeons can visualize and interact with the virtual surgery plan and navigation data while in the operating room. The user friendliness and usability is objectified by a formal user study that compared our augmented reality assisted technique to the gold standard setup of a perioperative navigation system (Brainlab). Moreover, accuracy of typical navigation tasks as reaching landmarks and following trajectories is compared. Results: Overall completion time of navigation tasks was 1.71 times faster using augmented reality (P = .034). Accuracy improved significantly using augmented reality (P < .001), for reaching physical landmarks a less strong correlation was found (P = .087). Although the participants were relatively unfamiliar with VR/AR (rated 2.25/5) and gesture-based interaction (rated 2/5), they reported that navigation tasks become easier to perform using augmented reality (difficulty Brainlab rated 3.25/5, HoloLens 2.4/5). Conclusion: The proposed workflow can be used in a wide range of image-guided surgery procedures as an addition to existing verified image guidance systems. Results of this user study imply that our technique enables typical navigation tasks to be performed faster and more accurately compared to the ian, Department of Oral & Maxillofacial Surical Center Groningen, University of Gronin-

current gold standard. In addition, qualitative feedback on our augmented reality assisted technique was more positive compared to the standard setup. Medical imaging has evolved into a technology capable of multimodal, highly detailed, 4D imaging. 1 Understanding complex anatomy and pathology plays a key role in patient diagnostics and treatment planning. Today, oral and craniomaxillofacial surgery (OCS) routinely relies on the use of pre-operative image data, which is converted into a 3D virtual surgical planning (3D VSP). 3D VSP has become part of standard care to plan and predict the outcome of complicated surgery in disciplines like oral and maxillofacial surgery, trauma surgery and neurosurgery. In the current workflow of oncologic resection surgery, a VSP is made based on pre-operative imaging. Typically cone-beam computed tomography (CBCT) data 2 is used for segmenting bone structures, where MRI data 3 is used for soft tissue segmentation, such as a tumor. Based on that, resection margins and optionally a reconstruction is planned. 4 The VSP is routinely translated for use in the operating room by using computer-aided design and manufacturing (CAD/CAM) constructed surgical guides and patient-specific plates. However, incorrect positioning of guides leads to deviations from the planned surgery. 5 Moreover, surgical guides are not applicable for soft-tissues or deep and narrow surgical approaches. For these applications real-time navigation systems are used to translate the 3D VSP for use in the operating room. Such image-guided surgery (IGS) systems not only offer benefits for patient safety and surgical outcome, but also improve orientation in the surgical field. IGS has been shown to greatly help surgeons to identify anatomical structures, shorten the time needed for surgery and reduce the workload. 6 Surgical navigation is an application of IGS that links image data to the patient and instruments during surgery. The workflow of surgical navigation starts from a 3D virtual planning, based on the available imaging (CT, MRI) data. The plan includes aspects such as resection margins, screw locations and delineation of essential anatomical structures. The navigation system then registers the surgical plan with the patient. This registration is the basis of surgical navigation and it is done by mapping predefined landmarks on the image data to the actual positions on the patient. Since surgical navigation provides real-time information about the orientation of surgical tools with respect to the anatomy of the patient, surgeons can see the spatial relationship of the instrument and image data in real time. Virtual landmarks or targets in the surgical plan can be located on the patient without being physically present. Surgical navigation integrates the virtual surgical plan with the actual patient, enables minimally invasive surgical approaches and improves operating accuracy and time. 7 Most visualizations currently used in perioperative navigation systems are presented on regular 2D monitors. In most operating rooms, surgeons can only observe patient data on 2D monitors or wall-fixed screens from a distance, sometimes even in a different orientation than the surgical field. This requires a cognitive interpretation step from the surgeon in which the observed images are projected onto the surgical field. This can be challenging and causes surgeons to divert their visual focus and attention. It also leads to excess surgical time and is error-prone. [8][9][10][11] Furthermore, such a setup does not allow for careful data exploration by the surgeon during surgery, necessitating indirect data presentation by a second person. This indirect communication can compromise the results and may lead to a cumbersome sterilizing process, wastes time and endangers asepsis. To overcome some of these drawbacks, mobile screens have been used. Such instrument-mounted displays decrease operating time and make surgical navigation easier to learn. 12,13 However recent technological advancements have made augmented reality (AR) in the form of a head-mounted-device (HMD) an accessible technique as a solution to these drawbacks. The main benefit of AR in the operating room is that it provides surgeons with direct spatial perception of the real world, overlaying the virtual images onto or in close proximity of the anatomy of the patient. 14 Additionally, gesture-based interaction techniques enable surgeons to directly interact with image data while operating on the patient. Such a system may potentially enhance the surgeon's level of control and thus save precious time while maintaining a sterile environment.
Various surgical fields including OCS, orthopedics, spine surgery, neurosurgery, laparoscopy surgery and biopsy procedures are exploring the potential of AR. 15 Use of AR systems in OCS is described for indications as trauma reconstructive surgery, orthognathic procedures, temporomandibular joint motion analysis, sentinel node biopsy and tumor resection. 14-21 Badiali et al. 14 developed a localizer-free seethrough display using colored markers for visual tracking, used for treatment of LeFort orthognathic patients. Zinser et al. 17 superimposed virtual orthognathic planning onto patients using a portable display, which is tracked with surgical navigation. Kalavakonda et al. suggest the Microsoft HoloLens could be used for aiding tumor resection in skull-base surgery and used a manual point based registration method. 22 Meulstee et al. reported on an newly developed navigation system combined with an augmented reality interface using the Microsoft HoloLens. 23 Our study reports on a system, developed to visualize image data close to the patient in the operating room that is oriented from the surgeon's point of view. This workflow enables perioperative interaction in a sterile environment, with 3D patient-specific planning and navigation data during oncologic resection of OCS patients. The goal of the system was to aid the surgeon in translating the 3D VSP to the patient by increased visual feedback, improving eyehand coordination and ultimately improving speed and accuracy. The Microsoft HoloLens HMD (Microsoft, Redmond, Washington, United States) 24 was used for augmenting the surgeon's view during surgery with the surgical plan and perioperative navigation data. The system was validated by comparison with the current workflow.

SYSTEM DESIGN
This section describes the design of the augmented reality visualization system. Briefly summarized, we connected the surgical navigation system, in our case Brainlab (Brainlab AG, Munich), to the Microsoft Hol-oLens and then visualized the patient and navigation data. Information retrieved from the navigation system is visualized in 3D and positioned by the surgeons preference. Moreover, the surgeon was enabled to interact with the 3D virtual planning content through oral instructions or user-defined gestures.

SYSTEM REQUIREMENTS
For the system to be usable in a clinical setting, it should be compatible with and non-inferior to the current workflow. Therefore, we listed all design requirements based on the current surgical workflow. We designed our technique to: G1: not require a change in workflow from the current clinical practice G2: be able to recognize gestures and voice commands, and change the visualization accordingly G3: support the manipulation of the whole AR content as well as specific objects G4: allow for both showing and hiding individual objects G5: support focus and context visualization G6: improve eye-hand coordination, G7: be easily extensible, and G8: be intuitive and require little learning time.
The interface is designed to support the surgeon in the operating room. Therefore, integration of the system should not require a change in workflow from the current clinical practice (G1). The surgeons can then position the AR content wherever it is most convenient, overcoming the need for mental registration of image data, surgical plan, navigation information and the patient (G3). Gestures enable interaction with the 3D image data without the need of breaking the sterile environment. Voice commands are mainly used for functions that cannot be integrated in an intuitive way in the visualization. Moreover, voice commands prevent the surgeon's hands from having to leave the working area (G2). Surgical plans may consist of many objects, but may occlude some important objects or organs if everything is visualized in the HoloLens. Our system enables the user to hide/ show parts of the objects (G4). Sometimes users want to study and discuss an individual object. For instance, during the operation, the surgeons sometimes tend to discuss the shape or the structure of a tumor (the focus). However, at the same time they also want to keep the original context-in this case, keeping the tumor in the original position. For this reason, we enable the user to make a copy of an individual part by a simple air-tap gesture (G5). Then the copy of the object (the tumor) is visualized next to the original 3D data and the same interactions are applied to the new copy.

IMAGE-GUIDED SURGERY SYSTEM
An overview of the developed AR-IGS system is shown in Figure 1. The system includes 3 tasks: retrieving the surgical plan and navigation data from the navigation hardware, processing the images, and presenting them to the surgeon. Figure 2 provides an overview of the hardware components, including the commercially available navigation system Brainlab (Brainlab AG, Munich), Microsoft HoloLens and a PC. The Brainlab navigation system is validated and is routinely used in hospitals around the world. In the operating room, the PC receives the pre-defined VSP from Brainlab as well as the tracking data from surgical tools. The communication between Brainlab and the PC is based on OpenIGTLink technology. 25 It is an open-source network protocol for image-guided therapy (IGT) specifically developed for standardization of communication between medical equipment in the operating room. After downloading the VSP onto the PC, stereoscopic images are rendered and streamed wireless to the HoloLens. The stereoscopic translucent screens of the HoloLens enable the surgeon to view the 3D virtual content as well as the surgical work area. The weight and fitment of the HoloLens is comparable to wearing a surgical headlight, expecting it to be comfortable to wear during the procedure. Moreover, the HoloLens being untethered, it allows the surgeon freedom during surgery. It also allows voice and gesture input for data interaction while maintaining the sterile field.
In our system, these gestures and voice instructions from the surgeon are streamed from the HoloLens back to the PC. According to these user inputs, specific data exploration is processed and the result is shown through the HoloLens.

USER STUDY
To understand how people would perform with and rate our technique, we conducted a comparative study. Participants performed some typical navigation tasks on a purpose designed phantom by using a surgical instrument. The traditional navigation interface (Brainlab) was chosen as the baseline and compared to the augmented reality interface based on speed, accuracy and qualitative feedback for 2 exploration tasks.

PARTICIPANTS
Participants were asked to complete several navigation tasks using both the traditional navigation interface and the augmented reality interface. After completing all tasks, participants were asked to complete a questionnaire to rate the usability of the technique in terms of ease of use, precision, efficiency, and difficulty on a 5 step Likert scale. The complete questionnaire is included as supplementary data to this manuscript. Ethics approval for this study was granted by the Medical Ethics Review Board of the University Medical Center Groningen under number M19.225061. All methods were performed in accordance with the relevant guidelines and regulations.

ARTICLE IN PRESS
Participants where suitably informed and informed consent was obtained.

APPARATUS
A 3D printed phantom was used as a navigation test object. To exclude anatomical experience bias of senior surgeons, we designed the phantom so that it does not represent any anatomical structures. We thus ensured that all participants were equally familiar with the data and could not orient themselves on anatomical prescience. We made a CBCT scan of the 3D phantom and uploaded the scan to the perioperative navigation system (Brainlab). Based on that, we defined target landmarks and trajectories that had to be identified using the standard surgical navigation instrument (as shown in Fig 3).

TASKS
We tested 3 physical landmarks and 3 trajectory searching tasks. The 3 physical landmark tasks were characterized by low-level actions that find the predefined physical landmarks, which were located on the structure of the phantom. Hereby participants received physical feedback when they complete the task. Trajectory searching tasks required participants to follow planned trajectories and possibly avoid the barriers. The trajectories were defined as linear paths from different directions nonparallel to any axis of the CBCT. This task simulated real surgical planning, such as following a planned path to the tumor or placement of an implant. However, 6 extra floating landmarks (start and end points) in trajectory task were also considered to be additional tests, during which participants no longer received physical feedback when a landmark was reached. The trajectory task itself required participants to move the instrument from 1 floating landmark to another, following the planned trajectory. In both conditions, before the real testing, participants were allowed to practice with the instrument for 10 minutes. Participants were asked to perform the tests as precisely and quickly as possible. At the start of each trial, participants were asked to put the instrument back to the starting point and they were allowed to observe the target landmark. As long as the participant fully understood the position of target landmark, the trial could be started through the voice command "start." When participants believed that they had reached the goal, the task could be ended through the voice command "end." At all times the position of the instrument (the tip) was recorded. An example of both the Brainlab and AR interface can be seen in the supplementary video.

MEASUREMENTS
The spatial position of the instrument's tip was recorded during the tasks. The accuracy of reaching a landmark was measured by calculating the Euclidian distance between the final position of instrument's tip and the target landmark. The RMS of the orthogonal distance between the actual path and the planned trajectory was used as a measure of deviation from the trajectory. Student's t-test was performed on measurements of user performance between the 2 conditions. For every participant, 3 physical landmarks, 9 floating landmarks and 6 trajectory measurements were recorded per condition.

Results
Twelve male members from our local university hospital participated in the study and completed all the predefined navigation tasks in mutual randomized order on both systems, using Brainlab with or without the HoloLens. Out of the total of 12 participants, 6 have reported prior experience with the navigation system. Prior experience included a minimum usage of 10 times. Of these 6 experienced participants, 1 has reported a daily-use experience. The other 6 participants in the experiment reported no experience with the navigation system. Ages ranged from 30 to 57 (Median = 44) for the experienced users, and from 22 to 31 (Median = 26) for the inexperienced users. Six participants were surgeons and the rest were medical students.

TIME
Overall completion time of all tasks in the HoloLens condition was 1.71 times faster than Brainlab condition (P = .034). However, 3 participants performed The accuracy for the HoloLens condition was significantly better (P < .001), but not for the physical landmarks (P = .087). Deviation from the planned trajectories was smaller using the HoloLens (P < .001). There were no significant differences between the experienced and in experienced users for reaching the floating landmarks (P = .152) or following the trajectories (P = .631). However the inexperienced users had smaller errors reaching the fixed landmarks (M = 1.46 mm, SD = 0.47 mm) compared to the experienced users (M = 2.54 mm, SD = 3.06 mm, P = .042). There was no significant difference in the loss of tracking in both conditions (P = .21). Table 1 and Figure 4 show the accuracy and time results of the separate tasks. The total pathway length was not significantly different between both interfaces (P = .20), but following the trajectories, the pathway was shorter using the HoloLens (P = .035). Figures 5  and 6 show scatter plots of the accuracy of the physical and floating landmarks. Figures 7 and 8 show scatter plots of the accuracy and pathway length following trajectories.
In the post-session questionnaire, the participants were asked to choose the method they preferred. Although the participants were relatively unfamiliar with VR/AR (rated 2.25/5) and gesturebased interaction (rated 2/5), they reported that the tasks became easier to perform with the help of HoloLens (difficulty Brainlab rated 3.25/5, Holo-Lens 2.4/5). Intuitiveness of interaction commands and real-time experience was rated above average (voice 3.6/5, gestures 2.8/5, real-time experience 3.9/5; goals G2 and G8).

USE CASE
A first implementation of the method has been performed during a oncologic resection of a OCS patient, navigated surgery was used for landmark identification during a maxillectomy and subsequent reconstruction. Figure 9 and video 1 show an example of a surgeon interacting with the patient-specific data in an operating room. While operating, users could disable gesture inputs to prevent unwanted manipulations of the visualization. They could also save the spatial position and orientation of the AR content by voice commands. The user could show or hide a selection of objects when necessary due to the occlusion. If an air-tap was performed on an individual object, a copy was visualized next to the original one.

ARTICLE IN PRESS
GLAS ET AL.

Discussion
Our augmented reality visualization technique improves accuracy and completion time for navigating tasks, especially in the floating landmark and trajectory tasks. In the clinical setting, free-floating landmarks give a better representation of navigating soft tissues. Due to the smaller deviation from the trajectory tasks, augmented reality visualization improves eye-hand coordination, which potentially leads to less tissue damage.
Kalavakonda et al. suggest the Microsoft HoloLens could be used for aiding tumor resection in skull-base surgery. 22 However they developed a manual point based registration to superimpose the virtual content and did not compare accuracy with the gold standard. In our case, the gold standard is Brainlab. This is a validated and accurate system linked to the visualization on the HoloLens. Moreover our system does not rely on additional registration methods of the virtual content and physical world, but uses the data directly from the navigation hardware. It is our belief that this is a safe and predictable method, without introducing additional errors. Other studies show the effectiveness of the HoloLens in minimal invasive surgery, where the HoloLens is used as a potential alternative to conventional monitors. 26 Although it has shown its benefits, the HoloLens is used for 2D visualization    without navigation data. Meulstee et al. reported on a newly developed navigation system combined with an augmented reality interface 23 where accuracy of placing an object in a planned position was used to assess performance of the augmented reality interface. They showed an increased navigation error using an augmented reality interface due to the fact they relied on the accuracy of the alignment of virtual content with the real world. The difference between their and our study is that we do not try to align a 3D printed object guided with virtual projections. We studied the accuracy of validated pointer of the Brainlab system. We think this explains the difference in our findings, we were able to demonstrate an improved accuracy using the HoloLens. This is due to the fact that we project the navigation data obtained from the Brainlab system directly in the HoloLens, and do not rely on the superimposition of the virtual content. Pietruski et al. developed an AR navigation system used to perform osteotomies on a phantom mandible using a virtually projected saw. 27 Their 'simple AR' method is comparable to the method in this study; namely, based on the navigation data rather than exact alignment and visual guidance of the AR content. Although their accuracy includes the registration error of the navigation system, using 2 control points they obtained a mean accuracy of 1.77 mm ( §0.82mm) and 1.88 mm ( §1.05mm). This accuracy is similar to the accuracy we found using the fixed landmarks, namely 1.76 mm ( §0.88mm) for the experienced group and 1.36 mm ( §0.44mm) for the inexperienced group. However, users could be biased by the use of an anatomically shaped phantom and accuracy could be improved by physical guidance of the navigated saw during performing the osteotomy.
In the physical landmark tasks, the HoloLens was faster but not significantly faster. One possible reason for this is that the test phantom was rectangular, and was scanned in a way that the shape was aligned with the sagittal, transversal and coronal axis. This may help the eye-hand coordination in the Brainlab condition. Another obvious possibility is that the participants received physical feedback from the instrument when they reached the destination. However, this does not necessarily represent a clinical situation, especially where the navigation target is located in soft tissue. Moreover, the navigation system (Brainlab) was located straight ahead in front of the participants during the experiment. In a clinical setting this will not always be feasible. Independent of the used technique, inexperienced users were more accurate in reaching fixed landmarks.
The rate of lost tracking is important during surgery; when tracking of the instrument or the patient is lost by the navigation system, surgeons usually shift their focus from patients to the navigation system. Nevertheless, we did not find a significant difference in loss of tracking between the 2 conditions, although some participants showed a decrease of 8.5% with the HoloLens condition.
We developed an augmented reality system for image-guided surgery. Our system supports visualization and interaction with the virtual surgery plan and navigation data in a phantom and we expect it to perform comparable in the operating room as well. Our technique improves the surgeon's eye-hand coordination in the surgical working area (goal G6) and FIGURE 9. Peroperative use of the HoloLens viewing the patient specific operating plan during oncologic resection and reconstruction of a CMF patient. The surgeon is manually positioning the visualisation in the surgical field. The planned outcome is visualised for improved spatial orientation during tumour resection en subsequent reconstruction. Navigation was used for peroperative landmark identification. supports manipulation of the whole AR content and of specific objects (goals G3 and G4). With a simple air-tap gesture, users can create a copy of a focused object and manipulate the new object (goal G5). We conducted a formal user study to compare user performance between our new technique with the traditional setup of perioperative system. The results show that our new technique is more precise and faster in the navigation tasks. We also received positive feedback from the user questionnaire.
The workflow presented here can be used in a wide range of image-guided surgeries, as an addition to existing verified image guidance systems (goals G1 and G7). Our next step for this work will be to translate the surgical plan to the actual surgical site on the patient, making it a fully augmented reality driven navigation system. Such a system enables minimization and improvement of the visual feedback, possibly further reducing the associated workload of the surgeon and improvement of eye-hand coordination.
In conclusion, this study presented the added value of a validated AR visualization system used for image guided surgery. It is concluded that AR allows optimization of the 3D workflow by means of improved accuracy and reduction of time.