How to Measure Effectiveness of a Novel Device

The Challenge

During laparoscopic (or keyhole) surgery the surgeon’s view can become obscured by smoke from the cutting instruments. Various methods for clearing the smoke exist but they generally involve interrupting the operation to empty and then re-inflate the cavity. A new device was designed to eliminate smoke in the surgical field that, in theory, removed the risk of obscuring the surgeon’s field of vision and the need for time-consuming methods of smoke clearance.  In order to test the effectiveness of this new device it was necessary to develop a way of measuring the amount of smoke in the surgeon’s visual field during the procedure. Previous trials of other smoke clearance techniques measured the level of smoke particles within the operating theatre as it’s not possible to take measurements within the surgical field itself. An alternative approach was needed to assess the direct effect of the new device on the ability of surgeons to have a clear view of the surgical fields.

The Approach

Rather than trying to physically measure the amount of smoke, it was decided that the subjective opinions of the clarity of view would be recorded on an appropriate scale. Since the new device could be placed in situ but switched to either on or off without the surgeons knowledge the surgeons were blinded to the patient’s treatment group, and their subjective assessment of the surgical field recorded. This was measured using a so-called visual analogue scale (VAS) and also on a categorical scale adapted from one devised for assessing the quality of video-recordings. In order to also obtain a more objective view of the visual field, the procedures were recorded and a panel of independent experts were asked to give their assessment of the visual field on the same scales. They were asked to review 1-minute segments of the procedure and to rate each on the quality scale as well as recording the proportion of that segment that was free from smoke. These measurements were augmented by recording the number of times that alternative smoke clearance methods were required, the number of times the camera was removed for cleaning and the duration of the procedure, for example. From these it was possible not only to detect the positive effect of the new device but also to assess its effectiveness relative to alternative methods.

The Value

By including a number of measures to try and assess the level of smoke in the surgeon’s visual field it was possible to maximise the chance of capturing any effect of the device at this early stage of clinical testing. On the basis of this analysis, the device progressed to a larger-scale phase III study, the results of which will determine whether or not this new device is approved for general use. Without the innovative study design and subsequent analysis it would not have been possible to gather sufficient evidence to justify progressing to phase III trials.