Table of Contents  
Year : 2016  |  Volume : 18  |  Issue : 5  |  Page : 732-734

Augmented reality assisted surgery: a urologic training tool

1 Scott Department of Urology, Baylor College of Medicine, Houston 77030, USA
2 University of South Florida, Department of Urology, Tampa 33606, USA

Date of Submission07-May-2015
Date of Decision30-Jul-2015
Date of Acceptance10-Sep-2015
Date of Web Publication27-Nov-2015

Correspondence Address:
Tariq S Hakky
Scott Department of Urology, Baylor College of Medicine, Houston 77030
Login to access the Email id

Source of Support: None, Conflict of Interest: None

DOI: 10.4103/1008-682X.166436

Rights and Permissions

Augmented reality is widely used in aeronautics and is a developing concept within surgery. In this pilot study, we developed an application for use on Google Glass ® optical head-mounted display to train urology residents in how to place an inflatable penile prosthesis. We use the phrase Augmented Reality Assisted Surgery to describe this novel application of augmented reality in the setting of surgery. The application demonstrates the steps of the surgical procedure of inflatable penile prosthesis placement. It also contains software that allows for detection of interest points using a camera feed from the optical head-mounted display to enable faculty to interact with residents during placement of the penile prosthesis. Urology trainees and faculty who volunteered to take part in the study were given time to experience the technology in the operative or perioperative setting and asked to complete a feedback survey. From 30 total participants using a 10-point scale, educational usefulness was rated 8.6, ease of navigation was rated 7.6, likelihood to use was rated 7.4, and distraction in operating room was rated 4.9. When stratified between trainees and faculty, trainees found the technology more educationally useful, and less distracting. Overall, 81% of the participants want this technology in their residency program, and 93% see this technology in the operating room in the future. Further development of this technology is warranted before full release, and further studies are necessary to better characterize the effectiveness of Augmented Reality Assisted Surgery in urologic surgical training.

Keywords: andrology; augmented reality assisted surgery; Google glass; inflatable penile prosthesis; optical head mounted display; surgical education

How to cite this article:
Dickey RM, Srikishen N, Lipshultz LI, Spiess PE, Carrion RE, Hakky TS. Augmented reality assisted surgery: a urologic training tool. Asian J Androl 2016;18:732-4

How to cite this URL:
Dickey RM, Srikishen N, Lipshultz LI, Spiess PE, Carrion RE, Hakky TS. Augmented reality assisted surgery: a urologic training tool. Asian J Androl [serial online] 2016 [cited 2022 Aug 19];18:732-4. Available from:

  Introduction Top

The earliest known surgical operation in humans was decompressive trepanation of the skull, performed in the Neolithic era. [1] Surgery continued to evolve with the advent of anesthesia, antiseptics, and X-ray in the 19 th century, allowing for more complex and precise procedures, as well as decreased mortality from sepsis. [2],[3] Novel use of technology has driven surgical advancement, from implantable technology such as the total artificial heart, cochlear implant, and inflatable penile prosthesis (IPP) to surgical techniques such as endoscopy and robotics. [4],[5],[6],[7],[8] The current state of endoscopic and robotic surgery has rendered many once open procedures to minimally invasive techniques. [4],[5] As the surgical evolution continues, new technologies will become more ubiquitous in the operating room, for both patient and surgeon.

Augmented reality (AR) is defined as a "technology that superimposes a computer-generated image on a user's view of the real world, thus providing a composite view." [9] AR is widely used in aeronautics to assist pilots in takeoff, maneuvers, and landing; some have suggested a similar embrace of this technology in the field of surgery. [10] Recent use of AR in minimally invasive surgery has resulted in the creation of hybrid image-guided surgery using endoscopic and robotic video feeds. A dedicated institution has been developed around image-guided hybrid therapies. [10] However, the delivery of AR in open surgery will require an alternative technology for the surgeon's interface.

Optical head-mounted displays (OHMDs) and specifically Google Glass's® wearable technology (Google Inc., Mountain View, California) have shown to be an effective tool in both the clinical and operative setting and can be used as an AR interface for surgery. [11] OHMDs have demonstrated to help anesthesiologists maintain awareness of the patient's status in a distracting operating theater. [12],[13] AR is an emerging open surgical tool in education and is effective in orthopedic surgery in the creation of a composite view of student and teaching surgeon. [14] Google Glass has been studied in other surgical fields including plastic surgery and Mohs surgery. [15],[16] In urology, a primitive form of AR has been implemented in robotic partial nephrectomy. [17],[18] To date, there is no study of the use of AR or Google Glass in open urologic surgery. In this pilot study, our aim is to determine the feasibility of implanting AR in the setting of an OHMD that could be built for open urologic surgery not only as a surgical assistant but also as a surgical training tool in the placement of IPP. This is a pilot study to look at the use of Google Glass as an Andrologic training tool. We use the phrase Augmented Reality Assisted Surgery (ARAS) to describe this novel application of AR in the setting of urologic surgery.

  Materials and Methods Top

After an IRB was obtained, Google Glass with software version XE 22.1 was purchased from Google during the beta testing phase. The device was loaded with a developed glass application, which contained a menu ([Figure 1]) with several options including a demonstration on how to place an IPP from start to finish in the penoscrotal approach, the most commonly used surgical approach for this procedure. [19] It also contained software that allowed for detection of interest points using an optical display camera from the OHMD output feeds to allow faculty to interact with residents during placement of the penile prosthesis ([Figure 2]). In the operating room, the OHMD is worn as a pair of glasses, with a monocular refractive lens over the right eye projecting input to the surgeon. The application is launched via voice commands or by tapping a menu display on the side of the Google glass. Once the application has loaded, the user is presented with a menu. Menu options include: how to use Google Glass, a step-by-step placement video of IPP placement in the penoscrotal approach, an interactive video display, and a section about the authors and build of the application.
Figure 1: ARAS menu screen shot from Google Glass application. ARAS: Augmented Reality Assisted Surgery.

Click here to view
Figure 2: ARAS application screen shot demonstrating detection of interest points using an optical display camera from the OHMD output feeds with faculty interaction during IPP placement. ARAS: Augmented Reality Assisted Surgery; OHMD: optical head-mounted displays.

Click here to view

In the first component of the application, video footage corresponding to the sequential steps of the IPP procedure was projected onto the live view of the patient in real time. This allowed trainees to visualize specific steps prior to completing them. The second component of the application included detection software that streamed live OR footage to a remote attending physician. The attending physician could then interact with the surgical trainee by moving a cursor highlighting areas of interest to explain details of the operation in depth. Consent was obtained to film sequences of the operations from patients undergoing IPP placement.

Study participants included urology residents and faculty at two institutions who volunteered to experience the technology and application and provide feedback. Participants were shown how to navigate Google Glass using either voice-command or tapping gestures and were given time to wear the Google Glass in either the operative or perioperative setting of IPP placement. Participants were then given a feedback survey to complete regarding the effectiveness and interest in the device and software. Survey questions included both quantitative and free text options ([Figure 3]). Quantitative questions asked participations to scale their perceptions to survey questions from 1 to 10, with 1 being "not at all" and 10 being "always". Other questions included a binary yes or no response with option to input free text. The surveys were then stratified into trainee and faculty categories and calculated to determine the mean rating for quantitative questions. Free text responses were analyzed for key words and grouped as closely as possible.
Figure 3: ARAS Experience Survey.

Click here to view

  Results Top

A total of 30 participants volunteered to take part in the study. Of these, 10 were urology faculty members in academic urology programs, and 20 were either urology residents or fellows. Out of a 10-point rating scale, total participants rated the educational usefulness of the application as 8.6. Ease of navigation of the device and application was rated 7.6. The likeliness to use the technology if made available was rated 7.4. The technology was also rated if perceived as too distracting, with overall rating of 4.9 ([Table 1]). For the recommendation to implement the technology into respective training programs, a total of 26 responses were received (4 nonresponses), with 21/26 (81%) participants responding in the affirmative. For the belief that Google Glass has a role in the operating room, a total of 28 responses were received (2 nonresponses) with 26/28 (93%) responding in the affirmative. For the consideration of using hands-free Google Glass in future practice, a total of 28 responses were received (2 nonresponses) with 20/28 (71%) indicating the affirmative. For the belief that the application bettered understanding of the surgical procedure and anatomy during IPP placement, 16 responses were received with 9/17 (53%) responding in the affirmative ([Table 2]).
Table 1: Mean results of ARAS experience survey from urologic faculty and trainees using 10-point rating scale

Click here to view
Table 2: Results of ARAS experience survey from urologic faculty and trainee respondents to yes-no questions

Click here to view

When stratified between trainees and faculty, educational usefulness was found to be 8.7 versus 8.4, respectively. Ease of navigation was found to be 7.5 versus 7.9, respectively. Likelihood to use was found to be 7.4 versus 7.5, respectively. Distraction in the OR was found to be 4.7 versus 5.4, respectively ([Table 1]).

Specific comments given by participants recommended the inclusion of voice activation, patient and staff educational components for the procedure, video recording in the OR instead of streaming only, the ability to pull up patient imaging from while operating, and magnification.

Our novel study has several limitations. It is limited because the survey has not been validated and that we have a limited number of responses to the survey. Additionally, our surgical training tool was only useful in training one type of open urologic surgery. AR itself has limitations, as much of the surgical programming has yet to be done, and this technology is continuing to evolve. Another concern may be surgeon over reliance on AR, creating an intraoperative conflict as the surgeon's decision may not always be in line with the AR software. Limitations of the current Google Glass technology include limited battery life, overheating of the battery, and cumbersome software integration.

  Discussion Top

As technology continues to permeate into every aspect of our lives, we look to integrate technology into urologic surgery and surgical training. Surgeons currently perform procedures after years of education and human surgical training to safely navigate through anatomical planes of the human body. The concept of AR, where a live view of the environment, in this case, a surgical operation, is supplemented by computer-generated sensory input using an OHMD, is the next evolutionary step in open urologic surgery and is an important training tool. Our goal is to introduce the concept of AR to enhance the surgeon's perception of reality during a procedure, and to enrich the learning process during a surgical procedure.

Here, we successfully demonstrate the first integration of Google Glass in the surgical training of urology residents and faculty along with the implementation of ARAS in feature detection during penile prosthesis surgery. The overwhelming response from faculty is to have a proctoring platform, while trainees slightly favored the implementation of ARAS over their training counterparts. As expected, we noticed that trainees, who are younger, rated the application more educationally useful and less distracting in the operating room than faculty, but were surprised to find faculty rating ease of navigation and likelihood to use in the future higher than trainees. This may suggest that though there is more acceptance of newer technology from trainees, there is also a higher expectation of available features and intuitive design than faculty. Suggested future capabilities include remote proctoring and a less cumbersome design. As ARAS evolves, it will continue to innovate surgical procedures by adding graphics, sounds, and haptic feedback. Advantages of ARAS include real-time feedback of residents during surgery, superior visibility, and interaction between faculty and residents. ARAS is part of the confluence of change that will be vital in the future armamentarium of surgical education.

  Conclusion Top

ARAS is a novel and effective urologic surgical training tool in this pilot study. Both faculty and trainees view this new technology in a positive light. Augmented reality in surgery represents a paradigm shift within surgery, and urologic training may benefit from further exploration of this technology within residency programs and practices alike. As future platforms of OHMDs become available, this technology will continue to change how medicine is both practiced and taught.

  Author Contributions Top

RMD wrote the manuscript, TSH developed the application and edited the manuscript, NS administered ARAS survey and collected data, LIL provided consultation, PES assisted with writing of the manuscript, REC developed the concept. All authors read and approved the final manuscript.

  Competing Interest Top

All authors declared no competing financial interests.

  Acknowledgments Top

The authors would like to acknowledge American Medical Systems for providing the funding for the development of the software application, and Guil Zriel for creating the Google Glass application.

  References Top

Bishop WJ. The Early History of Surgery. London: R. Hale; 1960. p. 192 p.  Back to cited text no. 1
Gawande A. Two hundred years of surgery. N Engl J Med 2012; 366: 1716-23.  Back to cited text no. 2
Rontgen WC. On a new kind of rays. Science 1896; 3: 227-31.  Back to cited text no. 3
Spaner SJ, Warnock GL. A brief history of endoscopy, laparoscopy, and laparoscopic surgery. J Laparoendosc Adv Surg Tech A 1997; 7: 369-73.  Back to cited text no. 4
Pugin F, Bucher P, Morel P. History of robotic surgery: from AESOP(R) and ZEUS(R) to da Vinci(R). J Visc Surg 2011; 148: e3-8.  Back to cited text no. 5
Mobley DF. Early history of inflatable penile prosthesis surgery: a view from someone who was there. Asian J Androl 2015; 17: 225-9.  Back to cited text no. 6
Macherey O, Carlyon RP. Cochlear implants. Curr Biol 2014; 24: R878-84.  Back to cited text no. 7
Cooley DA, Liotta D, Hallman GL, Bloodwell RD, Leachman RD, et al. Orthotopic cardiac prosthesis for two-staged cardiac replacement. Am J Cardiol 1969; 24: 723-30.  Back to cited text no. 8
Stevenson A. Oxford Dictionary of English. New York: Oxford University Press; 2010. pxxii. 2069 p.  Back to cited text no. 9
Marescaux J, Diana M. Next step in minimally invasive surgery: hybrid image-guided surgery. J Pediatr Surg 2015; 50: 30-6.  Back to cited text no. 10
Muensterer OJ, Lacher M, Zoeller C, Bronstein M, Kubler J. Google Glass in pediatric surgery: an exploratory study. Int J Surg 2014; 12: 281-9.  Back to cited text no. 11
Sanderson PM, Watson MO, Russell WJ, Jenkins S, Liu D, et al. Advanced auditory displays and head-mounted displays: advantages and disadvantages for monitoring by the distracted anesthesiologist. Anesth Analg 2008; 106: 1787-97.  Back to cited text no. 12
Liu D, Jenkins SA, Sanderson PM, Watson MO, Leane T, et al. Monitoring with head-mounted displays: performance and safety in a full-scale simulator and part-task trainer. Anesth Analg 2009; 109: 1135-46.  Back to cited text no. 13
Lindeque BG, Ponce BA, Menendez ME, Oladeji LO, Fryberger CT, et al. Emerging technology in surgical education: combining real-time augmented reality and wearable computing devices. Orthopedics 2014; 37: 751-7.  Back to cited text no. 14
Davis CR, Rosenfield LK. Looking at plastic surgery through google glass: part 1. Systematic review of google glass evidence and the first plastic surgical procedures. Plast Reconst Surg 2015; 135: 918-28.  Back to cited text no. 15
Kantor J. Application of google glass to mohs micrographic surgery: a pilot study in 120 patients. Dermatol Surg 2015; 41: 288-9.  Back to cited text no. 16
Hughes-Hallett A, Mayer EK, Marcus HJ, Cundy TP, Pratt PJ, et al. Augmented reality partial nephrectomy: examining the current status and future perspectives. Urology 2014; 83: 266-73.  Back to cited text no. 17
Hung AJ, Shah SH, Dalag L, Shin D, Gill IS. Development and validation of a novel robotic procedure specific simulation platform: partial nephrectomy. J Urol 2015; 194: 520-6.  Back to cited text no. 18
Simon R, Hakky TS, Henry G, Perito P, Martinez D, et al. Tips and tricks of inflatable penile prosthesis reservoir placement: a case presentation and discussion. J Sex Med 2014; 11: 1325-33.  Back to cited text no. 19


  [Figure 1], [Figure 2], [Figure 3]

  [Table 1], [Table 2]

This article has been cited by
1 Augmented Reality, Augmented Epistemology, and the Real-World Web
Cody Turner
Philosophy & Technology. 2022; 35(1)
[Pubmed] | [DOI]
2 Utility of Optical See-Through Head Mounted Displays in Augmented Reality-Assisted Surgery: A systematic review
Manuel Birlo, P.J. Eddie Edwards, Matthew Clarkson, Danail Stoyanov
Medical Image Analysis. 2022; : 102361
[Pubmed] | [DOI]
3 Auswirkungen der COVID-19 Pandemie auf die urologische Weiterbildung in Deutschland
Cem Aksoy, Philipp Reimold, Hendrik Borgmann, Mara Kölker, Angelika Cebulla, Julian Peter Struck, Viktor Zehe, Tim Nestler, Nicolas von Landenberg, Annemarie Uhlig, Katharina Boehm, Marianne Leitsmann
Aktuelle Urologie. 2022;
[Pubmed] | [DOI]
4 Practical Application of Augmented/Mixed Reality Technologies in Surgery of Abdominal Cancer Patients
Vladimir M. Ivanov, Anton M. Krivtsov, Sergey V. Strelkov, Anton Yu. Smirnov, Roman Yu. Shipov, Vladimir G. Grebenkov, Valery N. Rumyantsev, Igor S. Gheleznyak, Dmitry A. Surov, Michail S. Korzhuk, Valery S. Koskin
Journal of Imaging. 2022; 8(7): 183
[Pubmed] | [DOI]
5 A workflow to generate patient-specific three-dimensional augmented reality models from medical imaging data and example applications in urologic oncology
Nicole Wake, Andrew B. Rosenkrantz, William C. Huang, James S. Wysock, Samir S. Taneja, Daniel K. Sodickson, Hersh Chandarana
3D Printing in Medicine. 2021; 7(1)
[Pubmed] | [DOI]
6 The Effectiveness of Using Augmented Reality for Training in the Medical Professions: A Meta Analysis (Preprint)
Yahia Baashar, Gamal Alkawsi, Wan Nooraishya Wan Ahmad, Hitham Alhussian, Ayed Alwadain, Luiz Fernando Capretz, Areej Babiker, Adnan Alghail
JMIR Serious Games. 2021;
[Pubmed] | [DOI]
7 Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review
Zahraa Bassyouni,Imad H. Elhajj
Frontiers in Robotics and AI. 2021; 8
[Pubmed] | [DOI]
8 Lessons Learned: Medical Library Pilot Testing of a Virtual Reality Anatomy Lab
Emily C. Weyant,Matthew Carroll,Rachel R. Walden
Journal of Electronic Resources in Medical Libraries. 2021; : 1
[Pubmed] | [DOI]
9 Validity and effectiveness of augmented reality in surgical education: A systematic review
Joshua G. Kovoor,Aashray K. Gupta,Marc A. Gladman
Surgery. 2021;
[Pubmed] | [DOI]
10 Augmenting prehospital care
Yuval Glick,B Avital,J Oppenheimer,D Nahman,L Wagnert-Avraham,A Eisenkraft,L Dym,D Levi,A Agur,B Gustus,A Furer
BMJ Military Health. 2021; 167(3): 158
[Pubmed] | [DOI]
11 Augmented reality in surgical training: a systematic review
Matthew Adam Williams,James McVeigh,Ashok Inderraj Handa,Regent Lee
Postgraduate Medical Journal. 2020; 96(1139): 537
[Pubmed] | [DOI]
12 Head-Mounted Display Use in Surgery: A Systematic Review
Rafa Rahman,Matthew E. Wood,Long Qian,Carrie L. Price,Alex A. Johnson,Greg M. Osgood
Surgical Innovation. 2020; 27(1): 88
[Pubmed] | [DOI]
13 Google glass adoption in the educational environment: A case study in the Gulf area
Rana Saeed Al-Maroof,Aseel M. Alfaisal,Said A. Salloum
Education and Information Technologies. 2020;
[Pubmed] | [DOI]
14 Integrating Eye-Tracking to Augmented Reality System for Surgical Training
Shang Lu,Yerly Paola Sanchez Perdomo,Xianta Jiang,Bin Zheng
Journal of Medical Systems. 2020; 44(11)
[Pubmed] | [DOI]
15 Wave of Wearables
Jeffrey Tully,Christian Dameff,Christopher A. Longhurst
Clinics in Laboratory Medicine. 2020;
[Pubmed] | [DOI]
16 Augmented Reality for Robotics: A Review
Zhanat Makhataeva,Huseyin Varol
Robotics. 2020; 9(2): 21
[Pubmed] | [DOI]
17 A Systematic Review of the Use of Google Glass in Graduate Medical Education
Joseph F. Carrera,Connor C. Wang,William Clark,Andrew M. Southerland
Journal of Graduate Medical Education. 2019; 11(6): 637
[Pubmed] | [DOI]
18 Interface of mixed reality: from the past to the future
Steven Szu-Chi Chen,Henry Duh
CCF Transactions on Pervasive Computing and Interaction. 2019;
[Pubmed] | [DOI]
19 Video Technologies for Recording Open Surgery: A Systematic Review
Tomas J. Saun,Kevin J. Zuo,Teodor P. Grantcharov
Surgical Innovation. 2019; 26(5): 599
[Pubmed] | [DOI]
20 Surgical Telementoring Without Encumbrance
Edgar Rojas-Muñoz,Maria Eugenia Cabrera,Daniel Andersen,Voicu Popescu,Sherri Marley,Brian Mullis,Ben Zarzaur,Juan Wachs
Annals of Surgery. 2019; 270(2): 384
[Pubmed] | [DOI]
21 Touch interaction with Google Glass – Is it suitable for older adults?
Marten Haesner,Sebastian Wolf,Anika Steinert,Elisabeth Steinhagen-Thiessen
International Journal of Human-Computer Studies. 2018; 110: 12
[Pubmed] | [DOI]
22 Testing Smart Glasses in urology: Clinical and surgical potential applications
E. García-Cruz,A. Bretonnet,A. Alcaraz
Actas Urológicas Españolas (English Edition). 2018;
[Pubmed] | [DOI]
23 Use of Augmented Reality and Virtual Reality Technologies in Endoscopic Training
Sheena Bhushan,Sharmila Anandasabapathy,Richa Shukla
Clinical Gastroenterology and Hepatology. 2018;
[Pubmed] | [DOI]
24 Using Google Glass in Surgical Settings: Systematic Review
Nancy J Wei,Bryn Dougherty,Aundria Myers,Sherif M Badawy
JMIR mHealth and uHealth. 2018; 6(3): e54
[Pubmed] | [DOI]
25 Augmented Reality in Emergency Medicine: A Review (Preprint)
Brendan William Munzer,Mairaj Khan,Prashant Mahajan
Journal of Medical Internet Research. 2018;
[Pubmed] | [DOI]
26 Augmented reality: The use of the PicoLinker smart glasses improves wire insertion under fluoroscopy
Takafumi Hiranaka,Takaaki Fujishiro,Yuichi Hida,Yosaku Shibata,Masanori Tsubosaka,Yuta Nakanishi,Kenjiro Okimura,Harunobu Uemoto
World Journal of Orthopedics. 2017; 8(12): 891
[Pubmed] | [DOI]
27 Evaluación del uso de smart glasses en urología: potenciales aplicaciones en los ámbitos hospitalario y quirúrgico
E. García-Cruz,A. Bretonnet,A. Alcaraz
Actas Urológicas Españolas. 2017;
[Pubmed] | [DOI]
28 The use of head-mounted display eyeglasses for teaching surgical skills: A prospective randomised study
Robert G. Peden,Rachel Mercer,Andrew J. Tatham
International Journal of Surgery. 2016; 34: 169
[Pubmed] | [DOI]
29 Feasibility and safety of augmented reality-assisted urological surgery using smartglass
H. Borgmann,M. Rodríguez Socarrás,J. Salem,I. Tsaur,J. Gomez Rivas,E. Barret,L. Tortolero
World Journal of Urology. 2016;
[Pubmed] | [DOI]


Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

  In this article
Materials and Me...
Author Contributions
Competing Interest
Article Figures
Article Tables

 Article Access Statistics
    PDF Downloaded774    
    Comments [Add]    
    Cited by others 29    

Recommend this journal