September 22, 2021

I, Science

The science magazine of Imperial College

A trial of contactless image manipulation is giving surgeons more control

In an operating room at St Thomas’ Hospital in London, a surgeon is inserting a stent into a patient’s aorta. He glances up to a screen showing a real-time image of the artery. With a wave of his hand, the surgeon rotates and zooms in on the image to give the exact view he needs. He has not had to waste time rescrubbing and does not have to break concentration to ask another member of the team to manipulate the image for him: the surgeon is in complete control.

On trial at the hospital is a piece of motion sensing technology called Kinect, more often used by video game players to control onscreen action than in a sterile operating room. The new system, specifically designed for use during vascular surgery, was developed by a team from Microsoft Research Cambridge, Kings College London and Lancaster University. It takes 3D images of a patient’s anatomy and produces several 2D images from different points of view. Crucially, it also allows surgeons to manipulate these images using voice control and hand gestures.

Today’s operating room relies on medical imaging technology. Vascular surgeons can see real-time images similar to x-rays, which are overlaid with a 3D rendering of a patient’s aorta, to help them place stents. But the ways a surgeon can interact with and manipulate these images has struggled to catch up with the imaging technology itself.

An unsterile computer mouse and keyboard are of no use to a fully scrubbed up surgeon. He must decide between rescrubbing after manipulating the image himself or instructing someone else to manipulate it for him. With the second option, an assistant operates the mouse and keyboard while the surgeon utters instructions – “zoom in… in… pan down… ok, that will do” –while trying to maintain concentration.

Kinect technology will hand control back to the surgeon. The technology is currently only at the trial stage. Initial tests took place in March. Field tests on patients at St Thomas’ got underway at the start of May and will continue until the end of June. Then the surgeons will decide if they wish to carry on using it.

The outlook is good. Tom Carrell, a surgeon at St Thomas’ involved with the trial says that the technology is intuitive and makes everything much quicker. He no longer has to break concentration or risk compromising the sterile environment and patient safety. Time savings mean that the patient is brought in and out of anesthesia more quickly.

Because it was originally designed for use with video games, Kinect required some tweaking before it was suitable for the operating room. Instead of flailing arms, a hazard to sterility, the sensor now responds to gestures in the space in front of a surgeon from chest to waist and shoulder to shoulder. By requiring an initial voice command, the sensor can also tell between people, despite the fact that everyone in masked and dressed the same.

The technology and gestures used to control it are tailored to vascular surgery. But a prototype for use in neurosurgery is also under development and will eventually be trialed in Addenbroke’s Hospital, Cambridge. Neurosurgeons use 3D models of the brain to find the position and structure of tumors and aneurysms. At the moment there is no way to manipulate these images in real-time, so researchers are developing the Kinect technology to be able to do this too.

Also in development is a toolkit that could be used in any hospital that wants to apply contactless interaction to an imaging system. These advances are the first step towards a world in which an operating room where surgeons have full control is the norm.

Image from