OVERVIEW OF THE HOLODOCTOR SOFTWARE PACKAGE ‎‎

T.A. Suleymanov¹, A. M. Kosinkova², A. E. Mishvelov^3*, S.N. Povetkin⁴, A. N. Simonov⁵, I.V. Ziruk⁶

526B group of Pediatric Faculty, Stavropol State Medical University, Russian ‎Federation. 525A group of Pediatric Faculty, Stavropol State Medical University, Russian ‎Federation. Laboratory of 3D technologies, Stavropol State Medical University, Russian ‎Federation, Institute of Life Systems, North Caucasus Federal University, Russian Federation. Department of Epizootiology and Microbiology, Stavropol State Agrarian University, Russian Federation. Department of Morphology and a Pathology of Animals, Saratov State Agrarian University named after N.I.Vavilov, Russian Federation.

---

ABSTRACT

The article provides an overview of a novel program software package, HoloDoctor, which has been developed for surgical operations planning using 3D-models of patients' organs or systems of organs. The software package consists of three main modules: an Anatomical Atlas module, a Data analysis module, and a Surgical planning simulation module, so the overview shows results of the analysis of each module.

Keywords: HoloDoctor, surgical operation, medical practitioner, CT, MRI, MSCT, DICOM

---

Introduction

 

Medical devices demonstrate the advancement of the medical field [1-3]. In the planning of surgery, a better understanding of spatial relationships can be obtained by studying the pathological structure in an organ or tissue in interactive 3D format, since this approach helps to more accurately assess the conditions under which the operation can be performed, as well as to predict and prevent possible complications [4-10]. To date, there are numerous publications on various sections of Oncology, in which virtual reconstruction before surgery is already a routine event (removal of the tumor, placement of endovascular and endobronchial prostheses, etc.) [11-18]. In addition to virtual 3D reconstructions, there are more and more opportunities to create 3D models of organs and tissues. This opens a new field for future surgery, in which you can create a 3D model of the organ and surrounding structures so realistic that it is possible to work out an operational reception for a personal patient, taking into account all the anatomical nuances that can be implemented in a printed model [19-21]. This reduces the operation time and significantly improves its safety for the patient. An additional value of 3D modeling is the ability to visually represent the course of surgery for the patient and his family, thereby improving the relationship between the doctor and the patient [22, 23].

As part of the START-1 R & D program "Development of the HoloDoctor software package for real-time planning of surgical interventions", specialists of Stavropol State Medical University and North Caucasus Federal University created the HoloDoctor software package. The software package consists of three main modules: an Anatomical Atlas module, a Data analysis module, and a Surgical planning simulation module. The Anatomical Atlas module allows users to solve the problem of high-quality training of new medical personnel (students of medical universities). The module is an interactive anatomical atlas of organs, which includes three-dimensional models of structures of organs and systems of human organs in normal and pathological conditions. The data analysis module allows the user to view DICOM images of various studies (CT, MRI, MSCT, etc.). Like the anatomical Atlas module, it can be used for teaching students (as viewing information on the example of clinical cases), and for the medical practitioner. The Surgical planning simulation module allows users to perform realistic virtual surgical manipulations. Analytical studies have been conducted to assess the program's potential and effectiveness, and the results are included in this review.

1. Anatomical Atlas module

The Anatomical Atlas module allows students to learn more about the structure of tissues, organs, and organ systems. The interactive anatomical Atlas contains signatures as in Russian as in English (unlike many analogs), which reduce the time spent on translating from English to Russian and inverse. The Atlas contains models in pathology that represent real clinical cases, which ensures the uniqueness of this product (Fig. 1, 2).

                Figure 1 – A model of the human heart with a description of its anatomical structure (aortic arch).

Figure 2 – Model of blood vessels; coronary heart vessels (on the left), the human cardiovascular system (on the right).

The main task testing the Anatomical Atlas module was to check the usability of the interface, which can implement intuitive actions with three-dimensional models of organs, body parts, micro-objects, and their description. This module also supports the function of comparing models under normal conditions (without pathologies) with models obtained during the consideration of individual situational clinical tasks. In Fig. 3, we can see the result of the analysis of loading of a personal computer with the following technical characteristics:

Operating System:

Windows 10 Pro 64-bit 10.0.18362, Windows 7, 8 (emulator)

Motherboard ASUSTeK COMPUTER INC. H170M-PLUS

Memory 24576 MB

BIOS American Megatrends Inc. 0301

System Drive Model ST2000DM001-1ER164

Resolution 1920 x 1080 px

Power plan (balanced)

Processor

CPU Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz

Clock Frequency 3400 MHz

Max Frequency 3992 MHz

Processors 1

Cores 4 (8)

Package Socket 1151 LGA

Graphic adapters

GPU NVIDIA GeForce GTX 1080 Ti

Manufacturer NVIDIA

Dedicated Video Memory 11264 MB GDDR5X

Total Available Graphics Memory 23552 MB

Core Clock 1886 MHz

Starting the Anatomical Atlas module, the loading of the computer did not occur. As can be seen in fig. 3 and 4, the load on the CPU and RAM was reached from 60 to 99%. The anatomical module was also tested on Windows 7.8 with the same results.

 

Figure 3 – Loading the Central processing unit (CPU) when enabling the anatomical Atlas.

 

Figure 4- Graph of RAM consumption when running the anatomical Atlas module.

2. Data analysis module

This module is a result of the analysis of common structures. The patterns of different types of DICOM files obtained in research using different equipment was carried out. We searched for common patterns in different DICOM files to correctly interpret these files and output the necessary information. The module is based on the Unity 3D platform and allows you to perform the following functions: opening DICOM files; primary analysis of the file structure; loading graphical information and necessary parameters; loading the medical history (figure 5). A DICOM file is a file with an object-oriented tag structure. The General DICOM file model has a four-level hierarchical structure with the following levels of data representation: patient; study; study series; image/frame series. The file-level of the DICOM standard version 2020 describes:

1. the model and manufacturer of the equipment used for medical research;
2. the institutions where the study was conducted, information about the patient, information on medical staff examining a patient;
3. type of research and time of its conduct;
4. conditions and parameters of the patient's examination; parameters of an image/frame or a series of images/frames saved in a DICOM file;
5. unique keys for identifying groups of data described in DICOM files-image/frame, series or set of series obtained during the examination of the patient;
6. DICOM-Protocol for transmitting and receiving over TCP/IP.

An array of DICOM files created during various studies (ultrasound, CT, MRI, etc.) was studied. An additional module was created for batch processing of the DICOM array.

Modules for building 3D reconstructions of organs based on the analysis of source DICOM files were developed. For the program to create three-dimensional reconstructions of organs and individual structures of the organ, the skeleton, blood vessels, and the surface of the organ were used in the planning of surgery in real-time.

The study materials were images of patients’ organs (DICOM files) obtained using MRI on a PHILIPS "Ingenia 1.5 T" device and multispiral computed tomography on a Toshiba "Aguilion 64" device during 2018-2020. The software package and its modules can be installed on various devices running different platforms (Windows, Android). The software module also allows viewing DICOM images in augmented reality glasses (HoloLens). In addition to graphic information, the DICOM file contains many technical parameters of the survey, data about the equipment used for the study, and data about the patient and doctor. Therefore, the basic principles of this module when processing DICOM files are:

1. opening a DICOM-file;
2. Primary analysis of the file structure;
3. Loading the graphic information and the required parameters;
4. Loading the medical history (figure 5).

The processing time of DICOM files and their subsequent opening depends on several parameters: the type of study (CT, MRI, etc.); the volume of DICOM files; and the number of DICOM files. The maximum processing time for DICOM files is 30 seconds and the input file size is 2100 units (MRI examination). In comparison with the vast majority of solutions available on the market such as DICOM-viewer, Vitrea 2; for viewing DICOM files, our development stands out for its cross-platform nature. When developing modules for pre-processing medical images, a software development environment using Unity3D was prepared.

Figure 5 – Loading and viewing the patient's medical history (in Russian).

To compare the results of studies (CT and MRI reconstructions), the following programs were used: Vitrea2, DoctorCT-Slicer. From more than 300 patients, mostly with kidney diseases, who were examined and processed using these programs in the Stavropol regional clinical Advisory and diagnostic center, a group of 120 people was randomly selected and three comparison groups were formed. The results of 60 clinical cases were processed using HoloDoctor. The majority of patients were treated for the diseases of the parenchymatous organs.

The examination also revealed 10 patients with tumor processes (up to 6 mm in size) in the kidneys; in 15 clinical cases, metastases of tumors to the kidneys from distant organs were detected; one clinical case was with two tumors and metastases to one kidney; 10 clinical cases were studied with kidney hydronephrosis; one patient was examined with myxoma of the heart.

We conducted a study related to the integration of three-dimensional image reconstructions into the HoloDoctor software package for modeling surgery in HoloLens augmented reality glasses.

3. Surgical planning simulation module

For the development of the surgical intervention simulator module, a selection of diagnostic studies of patients using CT, MRI, and ultrasound in DICOM file format was made for processing using the DoctorCT-Slicer software and a simulation surgical system. The sample of patients was based on the principle of selecting complex clinical cases in Stavropol Diagnostics (40 cases in 2018). The developed application is designed to simulate incisions on the surface of a 3D model in real-time. 3D models obtained by computer tomography are used as objects. The application allows you to create a visual cut on the surface, eliminate it, and cut a closed contour of a part of the object of the user's choice. The cutting system is based on a special Shader model that allows you to convert polygons of an object to a transparent state.

Stitching works as a reverse operation, returning the normal Shader of the model and putting parts of it in the visible mode. This algorithm was chosen to reduce the load on the system, since most of the currently used computers do not have enough resources to emulate a physical section based on the material, which causes a long processing time for each click.

In addition, to simulate the work of the tumors, and modeling the surgeon prepared us HoloSurgery system, which allows making a surgical incision, and stitching, choice of surgical tools (scalpel, retractor, scalpel, laser), select the thickness and depth of cut on the 3D model of the body.

To optimize the process, a multithreaded data processing algorithm is used – the system determines the number of free processor threads, and, if necessary, allocates additional ones for calculations. Thus, we get a system that allows the user to perform operations on the object model in real-time. This system allows you to load the 3D model of the heart after reconstruction with CT into the simulator module and perform various manipulations: puncture, incision, stitching, scrolling the 3D model, zooming in and out of the organ model, and tool configuration mode.

It is possible to load the 3D model of the heart into the simulator module after reconstruction with CT and perform various manipulations: puncture, incision, stitching, scrolling of the 3D model, zooming in and out of the organ model, tool configuration mode. The system is based on Unity 3D using some elements prepared in Houdini. The presented models of organs are created based on computer tomography and 100% correspond to real objects (Fig. 6). The study of the multi-layer model allows the surgeon to more accurately locate the tumor in the kidney, assess the relative location of organs, and work out a specific surgical intervention scenario for the selected area on the body.

The next main stage of the work is the direct preoperative adaptation of the virtual model with the patient's body. In this regard, reference points on the 3D model are compared with the corresponding points on the real patient's body. This operation is performed using thermal sensors and a 3D marker or pointer by sequentially entering the coordinates of reference points. As a result, the coordinate system transformation matrix is calculated, and in the future, the pointer tip and its symmetry axis movements in space can be displayed on a virtual 3D model.

Figure 6 – The access path is traced through the tomogram slices, and the resulting trace result is displayed in the program interface on the screen. The area of surgical interest (4 points highlighted from the back on the 3D model), where a 3D model of the human body is built in the area

 

Access points specified by the computer program are evaluated by the surgeon performing minimally invasive operations. As a result, the developed system helps to choose the optimal volume of resection, improve the accuracy of surgery, shorten the operation time, reduce the volume of blood loss and tissue damage to the patient, and reduce the likely risks and complications.

To plan the course of surgery, you need to work out the scenario of the operation, the patient is given a standard CT or MRI procedure. Next, the radiologist builds a 3D model of internal organs and tissues of the operating field area from the DICOM images obtained. The developed software module for viewing DICOM images allows detecting pathological neoplasms to assess the severity and dynamics of pathological processes (diseases). The next step is to plan the surgery scenario based on the 3D model obtained. Before starting the operation, the doctor puts on augmented reality glasses, and the visualization system projects a 3D model on the skin. The surgeon begins the operation (Fig. 7). Throughout the operation, the navigation system monitors the position of the tools and the access path following the intended scenario.

Functional:

1. Alarm System about the danger of approaching the most important blood vessels;
2. Surgical navigation based on optical infrared tracking;
3. Measuring the distance from the tip of the pointer to anatomical structures;
4. Specialized loader-convector 3D models in the simulator, i.e. loading multi-layer 3D models of the human body, organ, organ systems (kidneys, heart, the human body with CT, MRI);
5. Export the results of building 3D models of organs for 3D printing;
6. The Ability to make an incision in any place of the organ, body;
7. Action Function: "making an incision" and animation of tissue opening in a pre-selected place on the model by the user's hand;
8. Intraoperative navigation of surgical intervention without ultrasound devices;
9. The possibility of angioscopy (virtual view of the cavity of organs, vessels);
10. The Possibility of performing a virtual colonoscopy (viewing the inside of blood vessels);
11. Navigation of the course of surgery (HoloLens);
12. Cutting and stitching with a laser scalpel;
13. Stitching of wound tissue with a laser;
14. The ability to specify reference points;
15. Fabric Highlighting;
16. Full access to the field of operation: scaling the model of the organ or organ systems, image detail, allowing you to study every inch of the body and every inner layer-files obtained on CT and MRI;
17. Selection of modes (additional modules) for dentistry, traumatology on the computer, and in augmented reality glasses.

Figure 7 - Using augmented reality glasses with a system that allows visualization projects a 3D model onto the skin

Conclusion  

It was found that the software package and its modules (simulator of surgical intervention, viewing DICOM images) proved their efficiency and effectiveness on the example of clinical cases with diseases of the kidneys, maxillofacial region, and cardiovascular system. In 2018-2020, 60 patients with kidney neoplasms, metastases, and stones were successfully operated on at the Stavropol diagnostic center. The developed software package allows surgeons to reduce the time of operations by 1-2 hours, depending on the complexity of the operation and also allowed to expand the field of view of surgical intervention, using simulation, to simulate the most successful course of the operation in the program. In General, we can conclude that the main advantages of the HoloDoctor software package are a large database of models of human organs and systems and the ability to perform operations online using HoloLens augmented reality glasses.

References

1. Kubanov S I, Savina S V, Nuzhnaya C V, Mishvelov A E, Tsoroeva M B, Litvinov M S, Irugova E Z, Midov A Z, Sizhazhev A A, Mukhadieva L B, Savitskaya M V, Povetkin S N. Development of 3d bioprinting technology using modified natural and synthetic hydrogels for engineering construction of organs. Int. J. Pharm. Phytopharm. Res. 2019; 9(5): 37-42.
2. Zabihinpour M, Mowlavi A A, Azadegan B. The Necessity of the Study of the Distribution of Radon Concentration in Residential Buildings. World J. Environ. Biosci. 2018; 7(2): 117-121.
3. Zeyaeyan S, Tohidi A, Ranjbar A, Dabaghsaz E. The Effect of Locating High-Rise Buildings On the Wind Flow Using CFD Simulation, Case Study Chitgar Towers in Tehran. World J. Environ. Biosci. 2017; 6(SI): 12-17.
4. Andrén O, Fall K, Franzén L, Andersson S O, Johansson J E, Rubin M A. How well does the Gleason score predict prostate cancer death? A 20yearfollowup of a population based cohort in Sweden. J Urol. 2006;175: 1337–1340. doi: 10.1016/S00225347(05)007342
5. Areshidze DA, Mischenko DV, Makartseva LA, Kucher SA, Kozlova MA, Timchenko LD, Rzhepakovsky IV, Nagdalian AA, Pushkin SV. Some Functional Measures of the Organism of Rats at Modeling of Ischemic Heart Disease in Two Different Ways. Entomology and Applied Science Letters. 2018 Jan 1;5(4):19-29.
6. Association of Oncologists of Russia. Clinical recommendations for the diagnosis and treatment of kidney cancer/Moscow, 2014.
7. Ljungberg B, Albiges L, Abu-Ghanem Y, Bensalah K, Dabestani S, Fernández-Pello S, Giles RH, Hofmann F, Hora M, Kuczyk MA, Kuusk T. European Association of Urology Guidelines on Renal Cell Carcinoma: The 2019 Update. (2019). EURURO-8276– 12 p
8. Bledzhyants GA, Mishvelov AE, Nuzhnaya KV, Anfinogenova OI, Isakova JA, Melkonyan RS, Hite GY, Suprunchuk VE, Makova AV, Popov AN, Ovechkin VS., The Effectiveness of the Medical Decision-Making Support System "Electronic Clinical Pharmacologist’, in the Management of Patients Therapeutic Profile”, Pharmacophore, 2019; 10(2), 76-81
9. Mishvelov EG, Melchenko EA, Vlasov AA, Anfinogenova OI, Nuzhnaya CV. Holodoctor Planning Software Real-Time Surgical Intervention, Pharmacophore, 2019; 10(2), 1-12.
10. Osipchuk GV, Povetkin SN, Ashotovich A, Nagdalian IA, Rodin MI, Vladimirovna I, Ziruk AN, Svetlakova EV, Basova NJ, Rzhepakovsky IV, Areshidze DA. The Issue of Therapy Postpartum Endometritis in Sows Using Environmentally Friendly Remedies”, Pharmacophore, 2019; 10(2), 82-84.
11. Russian Society of Urology Russian Society of Oncourology. Association of Oncologists of Russia Russian Society of Clinical Oncology. Clinical recommendations. Kidney parenchyma cancer/Moscow, 2018.
12. Hight GY, Mishvelov AE, Melchenko EA, Nuzhnaya CV, Epanov VA, Dolgalev AA, Kukharuk MY, Portnov MY, Suprunchuk VE. New Image Modeling Features For Planning Surgical Interventions. Research Journal of Pharmaceutical, Biological and Chemical Sciences. 2019, 10 (1):140-143.
13. Nuzhnaya KV, Mishvelov AE, Osadchiy SS, Tsoma MV, AM RS, Kurbanova KA, Rodin IA, Nagdalian AA, Rzhepakovskiy IV, Piskov SI, Povetkin SN. Computer Simulation and Navigation in Surgical Operations, Pharmacophore, 2019; 10(4), 43-48
14. Sizonenko MN, Timchenko LD, Rzhepakovskiy IV, DA SP, Areshidze VV, Nagdalian AA, Simonov AN, Svetlakova EV, Verevkina MN, Povetkin SN. The New Efficiency of the «Srmp» – Listerias Growth-Promoting Factor during Factory Cultivation, Pharmacophore, 2019; 10(2), 85-88.
15. Medical Mistakes Kill 100,000+ Americans A Year. // Journal of the American Medical Association. Electronical source:  http://www.yourmedicaldetective.com/drgrisanti/dangerous_medicine.htm
16. Nagdalian A.A., Selimov M.A., Topchii M.V., Oboturova N.P., Gatina Yu.S., Demchenkov E.L. Ways to reduce the oxidative activity of raw meat after a treatment by pulsed discliarge technology. Research Journal of Pharmaceutical, Biological and Chemical Sciences. 20 Т. 7. № 3. С. 1927-1932.
17. Pushkin SV, Nagdalian AA, Rzhepakovsky IV, Povetkin SN. AFM and CT Study of Zophoba Smorio Morphology and Microstructure, Entomol Appl Sci Lett, 2018, 5 (3): 35-40.
18. Iyer RK, Chiu LL, Vunjak-Novakovic G, Radisic M. Biofabrication enables efficient interrogation and optimization of sequential culture of endothelial cells, fibroblasts and cardiomyocytes for formation of vascular cords in cardiac tissue engineering. Biofabrication. 2012 Jul 31;4(3):035002.
19. Sayf Al-Katib, MD, Monisha Shetty, MD, Syed Mohammad A. Jafri, MD, Syed Zafar H. Jafri, MD, Radiologic Assessment of Native Renal Vasculature: A Multimodality Review, RG • 37(1), 136-156.
20. Selimov MA, Nagdalian AA, Povetkin SN, Statsenko EN, Litvinov MS, Mishvelov AE. Investigation of CdCl2 Influence on red blood cell morphology, Int. j. pharm. phytopharm. Res. 2019; 9(5): 8-13
21. I. O. Sokolova, O. V. Ivanov, D. A. Belyakov, T. B. Belyakova Creation and use of the program for studying the anatomy of the human cardiovascular system with the function of augmented reality, Scientific forum: Innovative science: collection of articles on materials VIII international conference. Moscow 2017, 7 (8): 26-30
22. Stalling, D.; Westerhoff, M.; Hege, H.-C.. C.D. Hansen and C.R. Johnson, ed. Amira: A Highly Interactive System for Visual Data Analysis. The Visualization Handbook. Elsevier: 2005: 749–767 doi:10.1016/b978-012387582-2/50040-x
23. Sovilj, S., Magjarevic, R., Lovell, N., Dokos, S. Realistic 3D bidomain model of whole heart electrical activity and ECG generation , Computing in Cardiology. 2013; 40: 377–380.